Re: regression test regressed
On 14.07.10 11:31, Knut Anders Hatlen wrote: On 07/14/10 10:57 AM, Tiago Espinha wrote: [ snip ] I think the best and simplest way to solve this is to just bump up the limit to what's required right now. Even if we require 11 or 12 ports that still allows us, in theory, to have ~5000 parallel runs in the same machine, which would be madness even in a top-of-the-line server. Agreed. I don't think the issue here is that we use too many ports (yet). It's just that it would be good if we didn't have to maintain a max limit that developers may or may not discover that they need to increase depending on how they ran the tests. This limit also has to be maintained in the scripts the developers use to run the tests. So I think it would be nice if we could somehow make the port allocation more scalable, but I would be fine with just increasing the limit to, say, 20 for now. I have committed a fix with increases the limit to 20 for now (see DERBY-4700). There seems to be more issues to address when it comes to ports in the test framework, so we should follow up on that. Regards, -- Kristian
Re: regression test regressed
On 14.07.10 11:32, Tiago Espinha wrote: Oh I see, then yes I guess we could store it as a static variable. Might it not be an inconvenient though? What port would we set it to? What if someone is doing test runs and has something bound to the port we came up with? Maybe the ideal solution would be to use a system property as well... we could allocate the bogus port normally through the getAlternativePort() if no property is defined, or simply use the static variable if one is defined as a property. I'm a bit skeptical about too many system properties, as they are inconvenient to maintain and use. They're fine for special cases, but the test suite should run "out-of-the-box" if possible. Your concern about something running on the bogus port already is valid, but we generally have the same issue with all ports we use - especially when the user specifies a base port. We could try to connect a socket to the bogus port when creating the TestConfiguration, or maybe even when getBogusPort() is invoked, but I'm not sure it adds significant value. On the other hand, maybe the extra time spent doing so is negligible. In my view, it is better to have a well-defined mechanism to get the bogus port, instead of doing getPort() -1 or something like that. I also think that the failure messages for connect tests should include the port number that was used, this makes it easier to debug. Does this make sense, or is there a better solution? Regards, -- Kristian Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 10:09:23 Subject: Re: regression test regressed On 14.07.10 10:59, Tiago Espinha wrote: A-ha I see what you mean. This limit is indeed for suites.All, so indeed it wouldn't come up when individual suites are ran. I'm not sure what you meant by bogus port though? The bogus port is a port where no Derby network server (or anything else) is running. It is used for negative connect tests and such, for instance trying to ping a network server on a port where there is no network server.
Re: regression test regressed
Oh I see, what you need is access to the port so that you can do your own calculations. I think that's fine really. I'd go with the static method for encapsulation reasons (even if the attribute is set as 'final'). Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 10:42:18 Subject: Re: regression test regressed On 14.07.10 11:28, Tiago Espinha wrote: > Would you define MAX_PORTS_USED through a system property? I suppose we could > keep a default limit that is a tight fit on the suites.All and if a property, > say derby.tests.maxPortsUsed, is defined then we override it with that value. > No, MAX_PORTS_USED would be defined in TestConfiguration as today. My point was that parallel runners should be able to calculate port ranges without the user having to specify how many ports are required per run. Something like: int basePort = getUserSpecifiedValue("baseport"); String[] suiteNames = AllPackages.getTopLevelSuiteNames(); // A smarter runner would use a port range pool, but this is just for illustration :) int[] basePorts = new int[suiteNames.length]; for (int i=0; i < basePorts.length; i++) { basePorts[i] = basePort + (i * TestConfiguration.getMaxPortsUsed()) } This way, I don't have to update my parallel run-script when someone adds a new test that causes MAX_PORTS_USED to be incremented. -- Kristian > Would this be ok? > > Tiago > > > - Original Message > From: Kristian Waagan > To: [email protected] > Sent: Wed, 14 July, 2010 10:06:54 > Subject: Re: regression test regressed > > On 14.07.10 10:57, Tiago Espinha wrote: > > [ snip ] > > >> By having this constant and enforcing this limit, we can safely have a test >> > run > >> with basePort=1527 and another one with basePort=1538. If we were doing runs >> like this and someone creates a new test that requires a new alternative port >> (and if we didn't have the limit) then at some point the first run would be >> using 1538 and our second run would also be using that at some point. We can't >> say for sure that they will collide but the possibility exists. >> >> > I wonder if we should make MAX_PORTS_USED public (maybe a public static > method would be better), then parallel runners can configure the port > ranges without any more help from the user than a base port. > > >
Re: regression test regressed
On 14.07.10 11:28, Tiago Espinha wrote:
Would you define MAX_PORTS_USED through a system property? I suppose we could
keep a default limit that is a tight fit on the suites.All and if a property,
say derby.tests.maxPortsUsed, is defined then we override it with that value.
No, MAX_PORTS_USED would be defined in TestConfiguration as today.
My point was that parallel runners should be able to calculate port
ranges without the user having to specify how many ports are required
per run.
Something like:
int basePort = getUserSpecifiedValue("baseport");
String[] suiteNames = AllPackages.getTopLevelSuiteNames();
// A smarter runner would use a port range pool, but this is just
for illustration :)
int[] basePorts = new int[suiteNames.length];
for (int i=0; i < basePorts.length; i++) {
basePorts[i] = basePort + (i * TestConfiguration.getMaxPortsUsed())
}
This way, I don't have to update my parallel run-script when someone
adds a new test that causes MAX_PORTS_USED to be incremented.
--
Kristian
Would this be ok?
Tiago
- Original Message
From: Kristian Waagan
To: [email protected]
Sent: Wed, 14 July, 2010 10:06:54
Subject: Re: regression test regressed
On 14.07.10 10:57, Tiago Espinha wrote:
[ snip ]
By having this constant and enforcing this limit, we can safely have a test
run
with basePort=1527 and another one with basePort=1538. If we were doing runs
like this and someone creates a new test that requires a new alternative port
(and if we didn't have the limit) then at some point the first run would be
using 1538 and our second run would also be using that at some point. We can't
say for sure that they will collide but the possibility exists.
I wonder if we should make MAX_PORTS_USED public (maybe a public static
method would be better), then parallel runners can configure the port
ranges without any more help from the user than a base port.
Re: regression test regressed
Oh I see, then yes I guess we could store it as a static variable. Might it not be an inconvenient though? What port would we set it to? What if someone is doing test runs and has something bound to the port we came up with? Maybe the ideal solution would be to use a system property as well... we could allocate the bogus port normally through the getAlternativePort() if no property is defined, or simply use the static variable if one is defined as a property. Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 10:09:23 Subject: Re: regression test regressed On 14.07.10 10:59, Tiago Espinha wrote: > A-ha I see what you mean. This limit is indeed for suites.All, so indeed it > wouldn't come up when individual suites are ran. > > I'm not sure what you meant by bogus port though? > The bogus port is a port where no Derby network server (or anything else) is running. It is used for negative connect tests and such, for instance trying to ping a network server on a port where there is no network server. -- Kristian > Tiago > > > - Original Message > From: Kristian Waagan > To: [email protected] > Sent: Wed, 14 July, 2010 8:52:32 > Subject: Re: regression test regressed > > On 13.07.10 22:33, Tiago Espinha wrote: > >> I think ideally we'd keep the max number of ports on a tight fit to what is >> actually needed, that's why I left them at 10. This way if new ports are >> required along the way, whoever makes the changes gets alerted that they need >> to >> increase this constant. >> >> Of course we can also overshoot and give it a large margin and possibly never >> get alerted by it again but I tend to prefer the tight fit approach. If we are >> introducing tests that require more ports, it's a good idea to alert the >> developer that it might have some repercussions in terms of parallel runs and >> of >> how far apart the port ranges should be. >> >> > I agree with this reasoning. After all, only a few tests are actually > requesting additional ports. > > >> It's interesting though, even if you define a base port through a property, >> this >> issue should still come up because you'll still be using more ports than the >> max >> allows... >> >> > That's true if you run all the suites in suites.All in serial, but when > you run tests in parallel each suite gets it's own TestConfiguration > (with its own base port). > Depending on which tests require an additional port (calling > getNextAvailablePort(), directly or indirectly), a parallel run may or > may not trigger the assert. > > Returning to keeping the number of ports down, would it make sense to > store the bogus port in a static variable? This way, all the > TestConfiguration instances in a JVM would share the same port, which is > possible since it is reserved as a port where nothing is supposed to be > listening. > > > Regards, >
Re: regression test regressed
On 07/14/10 10:57 AM, Tiago Espinha wrote: > At this point port reclaiming is not possible but it's just a matter of > creating > another method in TestConfiguration that decrements the number of ports used > so > far. If you are 100% positive that the latest port requested isn't going to > be > used again by the test, then I guess you could "free" it. Say you've started > on > port 2 and you've just got 20001 for the current test. If at the end of > the > test you are sure that 20001 won't be used again by that test, you could > decrement the last assigned port to 2 and as such, the next port that > will > be assigned to other tests will be 20001 again. > > This might be dangerous when it comes to replication (I think it is > replication > that uses this) because at the same point in time we will have multiple ports > allocated to different instances of Derby and then it is essential that they > don't overlap. > > In response to what you said, yes, it does mean that we can't have more than > 10 > tests decorated using that method but at a more lower level, what it actually > means is that we cannot invoke getAlternativePort() more than 10 times. But > this > constant isn't an actual limit, it's just the number of ports I've found to > be > needed when I implemented this mechanism. It was a quick way to deal with the > fact that we have tests that require alternative ports that can't overlap. > Since > the ports are sequenced based on a basePort, it also means we can safely have > parallel runs without the risk of port collision. > Thanks for explaining, Tiago. I suppose it would be safe to free a port when NetworkServerTestSetup is torn down, but it seems like most of the calls to getNextAvailablePort() come from the suite() methods, so freeing the ports when the tests have completed won't be of much help. (ServerPropertiesTest is the exception. It allocates four alternative ports at runtime.) > By having this constant and enforcing this limit, we can safely have a test > run > with basePort=1527 and another one with basePort=1538. If we were doing runs > like this and someone creates a new test that requires a new alternative port > (and if we didn't have the limit) then at some point the first run would be > using 1538 and our second run would also be using that at some point. We > can't > say for sure that they will collide but the possibility exists. > > I think the best and simplest way to solve this is to just bump up the limit > to > what's required right now. Even if we require 11 or 12 ports that still > allows > us, in theory, to have ~5000 parallel runs in the same machine, which would > be > madness even in a top-of-the-line server. > Agreed. I don't think the issue here is that we use too many ports (yet). It's just that it would be good if we didn't have to maintain a max limit that developers may or may not discover that they need to increase depending on how they ran the tests. This limit also has to be maintained in the scripts the developers use to run the tests. So I think it would be nice if we could somehow make the port allocation more scalable, but I would be fine with just increasing the limit to, say, 20 for now. -- Knut Anders
Re: regression test regressed
Would you define MAX_PORTS_USED through a system property? I suppose we could keep a default limit that is a tight fit on the suites.All and if a property, say derby.tests.maxPortsUsed, is defined then we override it with that value. Would this be ok? Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 10:06:54 Subject: Re: regression test regressed On 14.07.10 10:57, Tiago Espinha wrote: [ snip ] > By having this constant and enforcing this limit, we can safely have a test run > with basePort=1527 and another one with basePort=1538. If we were doing runs > like this and someone creates a new test that requires a new alternative port > (and if we didn't have the limit) then at some point the first run would be > using 1538 and our second run would also be using that at some point. We can't > say for sure that they will collide but the possibility exists. > I wonder if we should make MAX_PORTS_USED public (maybe a public static method would be better), then parallel runners can configure the port ranges without any more help from the user than a base port. -- Kristian > I think the best and simplest way to solve this is to just bump up the limit to > what's required right now. Even if we require 11 or 12 ports that still allows > us, in theory, to have ~5000 parallel runs in the same machine, which would be > madness even in a top-of-the-line server. > > Tiago > > > - Original Message > From: Knut Anders Hatlen > To: [email protected] > Sent: Wed, 14 July, 2010 9:34:48 > Subject: Re: regression test regressed > > On 07/14/10 09:24 AM, Tiago Espinha wrote: > >> It means that at some point in time during a suites.All run, each of the ports >> > >> in the range [basePort, basePort+10] will be in use. They won't all be in use >> at >> >> the same time but we assume that any combination of them might be and as such, >> > >> we know to keep away from that range for other parallel test runs. >> >> > Thanks. Let me see if I understand... This means that we can't have more > than 10 tests decorated with > TestConfiguration.clientServerDecoratorWithAlternativePort() before we > run out of ports? Could we somehow reclaim ports after a test has > finished and make them available to subsequent tests so that we only run > out of ports if a single test attempts to allocate more than 10 ports? > >
Re: regression test regressed
On 14.07.10 10:59, Tiago Espinha wrote: A-ha I see what you mean. This limit is indeed for suites.All, so indeed it wouldn't come up when individual suites are ran. I'm not sure what you meant by bogus port though? The bogus port is a port where no Derby network server (or anything else) is running. It is used for negative connect tests and such, for instance trying to ping a network server on a port where there is no network server. -- Kristian Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 8:52:32 Subject: Re: regression test regressed On 13.07.10 22:33, Tiago Espinha wrote: I think ideally we'd keep the max number of ports on a tight fit to what is actually needed, that's why I left them at 10. This way if new ports are required along the way, whoever makes the changes gets alerted that they need to increase this constant. Of course we can also overshoot and give it a large margin and possibly never get alerted by it again but I tend to prefer the tight fit approach. If we are introducing tests that require more ports, it's a good idea to alert the developer that it might have some repercussions in terms of parallel runs and of how far apart the port ranges should be. I agree with this reasoning. After all, only a few tests are actually requesting additional ports. It's interesting though, even if you define a base port through a property, this issue should still come up because you'll still be using more ports than the max allows... That's true if you run all the suites in suites.All in serial, but when you run tests in parallel each suite gets it's own TestConfiguration (with its own base port). Depending on which tests require an additional port (calling getNextAvailablePort(), directly or indirectly), a parallel run may or may not trigger the assert. Returning to keeping the number of ports down, would it make sense to store the bogus port in a static variable? This way, all the TestConfiguration instances in a JVM would share the same port, which is possible since it is reserved as a port where nothing is supposed to be listening. Regards,
Re: regression test regressed
On 14.07.10 10:57, Tiago Espinha wrote: [ snip ] By having this constant and enforcing this limit, we can safely have a test run with basePort=1527 and another one with basePort=1538. If we were doing runs like this and someone creates a new test that requires a new alternative port (and if we didn't have the limit) then at some point the first run would be using 1538 and our second run would also be using that at some point. We can't say for sure that they will collide but the possibility exists. I wonder if we should make MAX_PORTS_USED public (maybe a public static method would be better), then parallel runners can configure the port ranges without any more help from the user than a base port. -- Kristian I think the best and simplest way to solve this is to just bump up the limit to what's required right now. Even if we require 11 or 12 ports that still allows us, in theory, to have ~5000 parallel runs in the same machine, which would be madness even in a top-of-the-line server. Tiago - Original Message From: Knut Anders Hatlen To: [email protected] Sent: Wed, 14 July, 2010 9:34:48 Subject: Re: regression test regressed On 07/14/10 09:24 AM, Tiago Espinha wrote: It means that at some point in time during a suites.All run, each of the ports in the range [basePort, basePort+10] will be in use. They won't all be in use at the same time but we assume that any combination of them might be and as such, we know to keep away from that range for other parallel test runs. Thanks. Let me see if I understand... This means that we can't have more than 10 tests decorated with TestConfiguration.clientServerDecoratorWithAlternativePort() before we run out of ports? Could we somehow reclaim ports after a test has finished and make them available to subsequent tests so that we only run out of ports if a single test attempts to allocate more than 10 ports?
Re: regression test regressed
A-ha I see what you mean. This limit is indeed for suites.All, so indeed it wouldn't come up when individual suites are ran. I'm not sure what you meant by bogus port though? Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Wed, 14 July, 2010 8:52:32 Subject: Re: regression test regressed On 13.07.10 22:33, Tiago Espinha wrote: > I think ideally we'd keep the max number of ports on a tight fit to what is > actually needed, that's why I left them at 10. This way if new ports are > required along the way, whoever makes the changes gets alerted that they need >to > increase this constant. > > Of course we can also overshoot and give it a large margin and possibly never > get alerted by it again but I tend to prefer the tight fit approach. If we are > introducing tests that require more ports, it's a good idea to alert the > developer that it might have some repercussions in terms of parallel runs and >of > how far apart the port ranges should be. > I agree with this reasoning. After all, only a few tests are actually requesting additional ports. > It's interesting though, even if you define a base port through a property, >this > issue should still come up because you'll still be using more ports than the >max > allows... > That's true if you run all the suites in suites.All in serial, but when you run tests in parallel each suite gets it's own TestConfiguration (with its own base port). Depending on which tests require an additional port (calling getNextAvailablePort(), directly or indirectly), a parallel run may or may not trigger the assert. Returning to keeping the number of ports down, would it make sense to store the bogus port in a static variable? This way, all the TestConfiguration instances in a JVM would share the same port, which is possible since it is reserved as a port where nothing is supposed to be listening. Regards, -- Kristian > Tiago > > > - Original Message > From: Kristian Waagan > To: [email protected] > Sent: Tue, 13 July, 2010 20:25:47 > Subject: Re: regression test regressed > > On 13.07.10 20:59, Tiago Espinha wrote: > >> Hello Rick, >> >> Do we know which patch it was that introduced this? The second failure has to >> do >> with my test conversions from last year and it should come up if we're trying >> to >> request one more port from TestConfiguration.getNextAvailablePort() without >> adjusting the constant for the maximum number of ports needed. >> >> > This was surely introduced by my commit for DERBY-4700. > > >> This is as intended, to make sure all the ports used are sequential to allow >> for >> parallel test runs. >> >> If this is just the result of a new test invoking the getNextAvailablePort() >> then all there needs to be done is increment the >> TestConfiguration.MAX_PORTS_USED constant to 11 and the following wiki page >> also >> should be updated for >> coherence: http://wiki.apache.org/db-derby/DerbyJUnitTesting#Running_Tests >> >> > I guess we just have to bump the number of ports we are allowed to use. > The patched caused two more ports to be allocated (I think, it's not > quite clear to me how many TestConfiguration instances are created). > Shall we increase it to 12, 15 or 20? > > Now, the reason why it didn't fail when I ran the test, is of course > that I used the new parallel run capability. In that case, each suite > gets 10 ports for it self. > > > Sorry for the noise, I'll fix this tomorrow. > If someone wants to quickly bump the constant, feel free to do so under > DERBY-4700 (and update the wiki as Tiago mentioned). As a last step, the > example script provided by Knut for parallel runs should also be updated. > > >
Re: regression test regressed
At this point port reclaiming is not possible but it's just a matter of creating another method in TestConfiguration that decrements the number of ports used so far. If you are 100% positive that the latest port requested isn't going to be used again by the test, then I guess you could "free" it. Say you've started on port 2 and you've just got 20001 for the current test. If at the end of the test you are sure that 20001 won't be used again by that test, you could decrement the last assigned port to 2 and as such, the next port that will be assigned to other tests will be 20001 again. This might be dangerous when it comes to replication (I think it is replication that uses this) because at the same point in time we will have multiple ports allocated to different instances of Derby and then it is essential that they don't overlap. In response to what you said, yes, it does mean that we can't have more than 10 tests decorated using that method but at a more lower level, what it actually means is that we cannot invoke getAlternativePort() more than 10 times. But this constant isn't an actual limit, it's just the number of ports I've found to be needed when I implemented this mechanism. It was a quick way to deal with the fact that we have tests that require alternative ports that can't overlap. Since the ports are sequenced based on a basePort, it also means we can safely have parallel runs without the risk of port collision. By having this constant and enforcing this limit, we can safely have a test run with basePort=1527 and another one with basePort=1538. If we were doing runs like this and someone creates a new test that requires a new alternative port (and if we didn't have the limit) then at some point the first run would be using 1538 and our second run would also be using that at some point. We can't say for sure that they will collide but the possibility exists. I think the best and simplest way to solve this is to just bump up the limit to what's required right now. Even if we require 11 or 12 ports that still allows us, in theory, to have ~5000 parallel runs in the same machine, which would be madness even in a top-of-the-line server. Tiago - Original Message From: Knut Anders Hatlen To: [email protected] Sent: Wed, 14 July, 2010 9:34:48 Subject: Re: regression test regressed On 07/14/10 09:24 AM, Tiago Espinha wrote: > It means that at some point in time during a suites.All run, each of the > ports > in the range [basePort, basePort+10] will be in use. They won't all be in use >at > > the same time but we assume that any combination of them might be and as > such, > we know to keep away from that range for other parallel test runs. > Thanks. Let me see if I understand... This means that we can't have more than 10 tests decorated with TestConfiguration.clientServerDecoratorWithAlternativePort() before we run out of ports? Could we somehow reclaim ports after a test has finished and make them available to subsequent tests so that we only run out of ports if a single test attempts to allocate more than 10 ports? -- Knut Anders
Re: regression test regressed
On 07/14/10 09:24 AM, Tiago Espinha wrote: > It means that at some point in time during a suites.All run, each of the > ports > in the range [basePort, basePort+10] will be in use. They won't all be in use > at > the same time but we assume that any combination of them might be and as > such, > we know to keep away from that range for other parallel test runs. > Thanks. Let me see if I understand... This means that we can't have more than 10 tests decorated with TestConfiguration.clientServerDecoratorWithAlternativePort() before we run out of ports? Could we somehow reclaim ports after a test has finished and make them available to subsequent tests so that we only run out of ports if a single test attempts to allocate more than 10 ports? -- Knut Anders
Re: regression test regressed
On 13.07.10 22:33, Tiago Espinha wrote: I think ideally we'd keep the max number of ports on a tight fit to what is actually needed, that's why I left them at 10. This way if new ports are required along the way, whoever makes the changes gets alerted that they need to increase this constant. Of course we can also overshoot and give it a large margin and possibly never get alerted by it again but I tend to prefer the tight fit approach. If we are introducing tests that require more ports, it's a good idea to alert the developer that it might have some repercussions in terms of parallel runs and of how far apart the port ranges should be. I agree with this reasoning. After all, only a few tests are actually requesting additional ports. It's interesting though, even if you define a base port through a property, this issue should still come up because you'll still be using more ports than the max allows... That's true if you run all the suites in suites.All in serial, but when you run tests in parallel each suite gets it's own TestConfiguration (with its own base port). Depending on which tests require an additional port (calling getNextAvailablePort(), directly or indirectly), a parallel run may or may not trigger the assert. Returning to keeping the number of ports down, would it make sense to store the bogus port in a static variable? This way, all the TestConfiguration instances in a JVM would share the same port, which is possible since it is reserved as a port where nothing is supposed to be listening. Regards, -- Kristian Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Tue, 13 July, 2010 20:25:47 Subject: Re: regression test regressed On 13.07.10 20:59, Tiago Espinha wrote: Hello Rick, Do we know which patch it was that introduced this? The second failure has to do with my test conversions from last year and it should come up if we're trying to request one more port from TestConfiguration.getNextAvailablePort() without adjusting the constant for the maximum number of ports needed. This was surely introduced by my commit for DERBY-4700. This is as intended, to make sure all the ports used are sequential to allow for parallel test runs. If this is just the result of a new test invoking the getNextAvailablePort() then all there needs to be done is increment the TestConfiguration.MAX_PORTS_USED constant to 11 and the following wiki page also should be updated for coherence: http://wiki.apache.org/db-derby/DerbyJUnitTesting#Running_Tests I guess we just have to bump the number of ports we are allowed to use. The patched caused two more ports to be allocated (I think, it's not quite clear to me how many TestConfiguration instances are created). Shall we increase it to 12, 15 or 20? Now, the reason why it didn't fail when I ran the test, is of course that I used the new parallel run capability. In that case, each suite gets 10 ports for it self. Sorry for the noise, I'll fix this tomorrow. If someone wants to quickly bump the constant, feel free to do so under DERBY-4700 (and update the wiki as Tiago mentioned). As a last step, the example script provided by Knut for parallel runs should also be updated.
Re: regression test regressed
On 14.07.10 09:24, Tiago Espinha wrote: It means that at some point in time during a suites.All run, each of the ports in the range [basePort, basePort+10] will be in use. Just a minor detail, but I think it means each of the ports *may* be in use at some point. For instance, not all runs will use the JMX port. If you run an embedded test only, no ports will be actually used. -- Kristian They won't all be in use at the same time but we assume that any combination of them might be and as such, we know to keep away from that range for other parallel test runs. We also know that it'd be a mistake to set a base port of, for example, 70 if you have a web server running on that machine, as it eventually will go up to port 80 and it will fail because it can't bind on it. Tiago - Original Message From: Knut Anders Hatlen To: [email protected] Sent: Tue, 13 July, 2010 23:11:42 Subject: Re: regression test regressed On 07/13/10 10:33 PM, Tiago Espinha wrote: I think ideally we'd keep the max number of ports on a tight fit to what is actually needed, that's why I left them at 10. This way if new ports are required along the way, whoever makes the changes gets alerted that they need to increase this constant. What does the limit of 10 ports mean, exactly? Does it mean that there could be a maximum of 10 ports in use at the same time? Or does it mean that a maximum of 10 tests can use an alternative port? Thanks,
Re: regression test regressed
It means that at some point in time during a suites.All run, each of the ports in the range [basePort, basePort+10] will be in use. They won't all be in use at the same time but we assume that any combination of them might be and as such, we know to keep away from that range for other parallel test runs. We also know that it'd be a mistake to set a base port of, for example, 70 if you have a web server running on that machine, as it eventually will go up to port 80 and it will fail because it can't bind on it. Tiago - Original Message From: Knut Anders Hatlen To: [email protected] Sent: Tue, 13 July, 2010 23:11:42 Subject: Re: regression test regressed On 07/13/10 10:33 PM, Tiago Espinha wrote: > I think ideally we'd keep the max number of ports on a tight fit to what is > actually needed, that's why I left them at 10. This way if new ports are > required along the way, whoever makes the changes gets alerted that they need >to > > increase this constant. > What does the limit of 10 ports mean, exactly? Does it mean that there could be a maximum of 10 ports in use at the same time? Or does it mean that a maximum of 10 tests can use an alternative port? Thanks, -- Knut Anders
Re: regression test regressed
On 07/13/10 10:33 PM, Tiago Espinha wrote: > I think ideally we'd keep the max number of ports on a tight fit to what is > actually needed, that's why I left them at 10. This way if new ports are > required along the way, whoever makes the changes gets alerted that they need > to > increase this constant. > What does the limit of 10 ports mean, exactly? Does it mean that there could be a maximum of 10 ports in use at the same time? Or does it mean that a maximum of 10 tests can use an alternative port? Thanks, -- Knut Anders
Re: regression test regressed
I think ideally we'd keep the max number of ports on a tight fit to what is actually needed, that's why I left them at 10. This way if new ports are required along the way, whoever makes the changes gets alerted that they need to increase this constant. Of course we can also overshoot and give it a large margin and possibly never get alerted by it again but I tend to prefer the tight fit approach. If we are introducing tests that require more ports, it's a good idea to alert the developer that it might have some repercussions in terms of parallel runs and of how far apart the port ranges should be. It's interesting though, even if you define a base port through a property, this issue should still come up because you'll still be using more ports than the max allows... Tiago - Original Message From: Kristian Waagan To: [email protected] Sent: Tue, 13 July, 2010 20:25:47 Subject: Re: regression test regressed On 13.07.10 20:59, Tiago Espinha wrote: > Hello Rick, > > Do we know which patch it was that introduced this? The second failure has to >do > with my test conversions from last year and it should come up if we're trying >to > request one more port from TestConfiguration.getNextAvailablePort() without > adjusting the constant for the maximum number of ports needed. > This was surely introduced by my commit for DERBY-4700. > This is as intended, to make sure all the ports used are sequential to allow >for > parallel test runs. > > If this is just the result of a new test invoking the getNextAvailablePort() > then all there needs to be done is increment the > TestConfiguration.MAX_PORTS_USED constant to 11 and the following wiki page >also > should be updated for > coherence: http://wiki.apache.org/db-derby/DerbyJUnitTesting#Running_Tests > I guess we just have to bump the number of ports we are allowed to use. The patched caused two more ports to be allocated (I think, it's not quite clear to me how many TestConfiguration instances are created). Shall we increase it to 12, 15 or 20? Now, the reason why it didn't fail when I ran the test, is of course that I used the new parallel run capability. In that case, each suite gets 10 ports for it self. Sorry for the noise, I'll fix this tomorrow. If someone wants to quickly bump the constant, feel free to do so under DERBY-4700 (and update the wiki as Tiago mentioned). As a last step, the example script provided by Knut for parallel runs should also be updated. -- Kristian > The first failure might be (i.e. probably is) related. > > Tiago > > > - Original Message > From: Rick Hillegas > To: "[email protected]" > Sent: Tue, 13 July, 2010 19:44:29 > Subject: regression test regressed > > I'm seeing these now: > > 1) >testAttributeAccumulatedConnectionCount(org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest)java.security.PrivilegedActionException: >: > javax.management.InstanceNotFoundException: > org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e > at java.security.AccessController.doPrivileged(Native Method) > at >org.apache.derbyTesting.functionTests.tests.management.MBeanTest.getAttribute(MBeanTest.java:379) >) > > at >org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest.testAttributeAccumulatedConnectionCount(NetworkServerMBeanTest.java:93) >) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >) > > at >org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109) > at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) > at junit.extensions.TestSetup$1.protect(TestSetup.java:21) > at junit.extensions.TestSetup.run(TestSetup.java:25) > at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) > at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) > at junit.extensions.TestSetup$1.protect(TestSetup.java:21) > at junit.extensions.TestSetup.run(TestSetup.java:25) > at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) > at junit.extensions.TestSetup$1.protect(TestSetup.java:21) > at junit.extensions.TestSetup.run(TestSetup.java:25) > at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) > Caused by: javax.management.InstanceNotFoundException: > org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e > at >com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.
Re: regression test regressed
On 13.07.10 20:59, Tiago Espinha wrote: Hello Rick, Do we know which patch it was that introduced this? The second failure has to do with my test conversions from last year and it should come up if we're trying to request one more port from TestConfiguration.getNextAvailablePort() without adjusting the constant for the maximum number of ports needed. This was surely introduced by my commit for DERBY-4700. This is as intended, to make sure all the ports used are sequential to allow for parallel test runs. If this is just the result of a new test invoking the getNextAvailablePort() then all there needs to be done is increment the TestConfiguration.MAX_PORTS_USED constant to 11 and the following wiki page also should be updated for coherence: http://wiki.apache.org/db-derby/DerbyJUnitTesting#Running_Tests I guess we just have to bump the number of ports we are allowed to use. The patched caused two more ports to be allocated (I think, it's not quite clear to me how many TestConfiguration instances are created). Shall we increase it to 12, 15 or 20? Now, the reason why it didn't fail when I ran the test, is of course that I used the new parallel run capability. In that case, each suite gets 10 ports for it self. Sorry for the noise, I'll fix this tomorrow. If someone wants to quickly bump the constant, feel free to do so under DERBY-4700 (and update the wiki as Tiago mentioned). As a last step, the example script provided by Knut for parallel runs should also be updated. -- Kristian The first failure might be (i.e. probably is) related. Tiago - Original Message From: Rick Hillegas To: "[email protected]" Sent: Tue, 13 July, 2010 19:44:29 Subject: regression test regressed I'm seeing these now: 1) testAttributeAccumulatedConnectionCount(org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest)java.security.PrivilegedActionException: javax.management.InstanceNotFoundException: org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e at java.security.AccessController.doPrivileged(Native Method) at org.apache.derbyTesting.functionTests.tests.management.MBeanTest.getAttribute(MBeanTest.java:379) at org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest.testAttributeAccumulatedConnectionCount(NetworkServerMBeanTest.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) Caused by: javax.management.InstanceNotFoundException: org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1010) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:627) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659) at org.apache.derbyTesting.functionTests.tests.management.MBeanTest$4.run(MBeanTest.java:382) ... 41 more There was 1 failure: 1) ttestSetPortPriority(org.apache.derbyTesting.functionTests.tests.derbynet.ServerPropertiesTest)junit.framework.AssertionFailedError: Port 1537 exceeeds expected maximum. You may need to update TestConfiguration.MAX_PORTS_USED and the Wiki page at http://wiki.apache.org/db-derby/DerbyJUnitTesting if test runs now require more available ports at org.apache.derbyTesting.junit.TestConfiguration.getNextAvailablePort(TestConfiguration.java:1413) at org.apache.derbyTesting.functionTests.tests.derbynet.ServerPropertiesTest.ttestSetPortPriority(ServerPropertiesTest.java:445) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109) at junit.extensions.TestDecorator.basicRun(TestDecorat
Re: regression test regressed
Hello Rick, Do we know which patch it was that introduced this? The second failure has to do with my test conversions from last year and it should come up if we're trying to request one more port from TestConfiguration.getNextAvailablePort() without adjusting the constant for the maximum number of ports needed. This is as intended, to make sure all the ports used are sequential to allow for parallel test runs. If this is just the result of a new test invoking the getNextAvailablePort() then all there needs to be done is increment the TestConfiguration.MAX_PORTS_USED constant to 11 and the following wiki page also should be updated for coherence: http://wiki.apache.org/db-derby/DerbyJUnitTesting#Running_Tests The first failure might be (i.e. probably is) related. Tiago - Original Message From: Rick Hillegas To: "[email protected]" Sent: Tue, 13 July, 2010 19:44:29 Subject: regression test regressed I'm seeing these now: 1) testAttributeAccumulatedConnectionCount(org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest)java.security.PrivilegedActionException: javax.management.InstanceNotFoundException: org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e at java.security.AccessController.doPrivileged(Native Method) at org.apache.derbyTesting.functionTests.tests.management.MBeanTest.getAttribute(MBeanTest.java:379) at org.apache.derbyTesting.functionTests.tests.management.NetworkServerMBeanTest.testAttributeAccumulatedConnectionCount(NetworkServerMBeanTest.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) Caused by: javax.management.InstanceNotFoundException: org.apache.derby:type=NetworkServer,system=c013800d-0129-cd0d-100f-e1d7aa3e at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1010) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:627) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659) at org.apache.derbyTesting.functionTests.tests.management.MBeanTest$4.run(MBeanTest.java:382) ... 41 more There was 1 failure: 1) ttestSetPortPriority(org.apache.derbyTesting.functionTests.tests.derbynet.ServerPropertiesTest)junit.framework.AssertionFailedError: Port 1537 exceeeds expected maximum. You may need to update TestConfiguration.MAX_PORTS_USED and the Wiki page at http://wiki.apache.org/db-derby/DerbyJUnitTesting if test runs now require more available ports at org.apache.derbyTesting.junit.TestConfiguration.getNextAvailablePort(TestConfiguration.java:1413) at org.apache.derbyTesting.functionTests.tests.derbynet.ServerPropertiesTest.ttestSetPortPriority(ServerPropertiesTest.java:445) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) FAILURES!!! Tests run: 8845, Failures: 1, Er
