Re: squeezing some multi-processing out of the nightly test runs?
Kristian Waagan <[EMAIL PROTECTED]> writes: > Knut Anders Hatlen wrote: >> Rick Hillegas <[EMAIL PROTECTED]> writes: >> >>> Mike Matrigali wrote: I would like to get the nightly tests to run faster on machines that have more than one cpu - or even hyperthreaded on a single cpu. Since most of the tests are single threaded I think this means somehow running more than one test at a time. For me it might even help if I was just able to run tests on 2 different codelines on the same machine at the same time. I think going forward this will be more and more common for developers. You can find reasonably priced laptops nowadays that come dual core already. I know what outstanding issue is that the network tests default to the same port. I believe at least for the junit tests one can set a different port number. Does anyone know if this fixes all the issues for the junit tests - I think I have seen some tests try different ports rather than the default one. >>> Hi Mike, >>> >>> I don't know how much work is needed to make the junit tests >>> independent of the port number. I can't find a JIRA which lists all of >>> the problem cases. However, the following JIRAs may be helpful: >>> DERBY-2440 and DERBY-2404. >> >> It is possible to change the port for all tests in a JUnit suite if we >> run the suite like this: >> >> java -Dport=XXX -DjmxPort=YYY ... > > Hi Knut Anders, > > Have you tested this? > The last time I looked at it, the relevant code was only invoked for > test runs in the DERBY_HARNESS_CONFIG configuration, which I believe > is some kind of backward compatibility mode with respect to the old > harness. > It seems this mode can be triggered by various things, for instance if > derby.system.home is specified, so people might see different behavior > depending on how they start the tests. > > I also see that that some methods are using the constant DEFAULT_PORT > directly, which suggests we are not quite there yet... Yes, I tested it, but only after I sent the mail. :) I saw the same behaviour as you describe. Both -Dport=XXX and -DjmxPort=YYY only work if the JUnit test are run from the old harness. It also looks like the replication tests have port 1527 hard-coded. > I find this code, or at least the naming and the comments, a tad > confusing. I would like to remove the notion of the old harness, and > can do another pass and see if we can remove it now. I don't think we have any JUnit tests in the old harness any more (except the compatibility test, but that one doesn't use TestConfiguration), so removing the notion of the old harness sounds like a good idea to me. -- Knut Anders
Re: squeezing some multi-processing out of the nightly test runs?
Knut Anders Hatlen wrote: Rick Hillegas <[EMAIL PROTECTED]> writes: Mike Matrigali wrote: I would like to get the nightly tests to run faster on machines that have more than one cpu - or even hyperthreaded on a single cpu. Since most of the tests are single threaded I think this means somehow running more than one test at a time. For me it might even help if I was just able to run tests on 2 different codelines on the same machine at the same time. I think going forward this will be more and more common for developers. You can find reasonably priced laptops nowadays that come dual core already. I know what outstanding issue is that the network tests default to the same port. I believe at least for the junit tests one can set a different port number. Does anyone know if this fixes all the issues for the junit tests - I think I have seen some tests try different ports rather than the default one. Hi Mike, I don't know how much work is needed to make the junit tests independent of the port number. I can't find a JIRA which lists all of the problem cases. However, the following JIRAs may be helpful: DERBY-2440 and DERBY-2404. It is possible to change the port for all tests in a JUnit suite if we run the suite like this: java -Dport=XXX -DjmxPort=YYY ... Hi Knut Anders, Have you tested this? The last time I looked at it, the relevant code was only invoked for test runs in the DERBY_HARNESS_CONFIG configuration, which I believe is some kind of backward compatibility mode with respect to the old harness. It seems this mode can be triggered by various things, for instance if derby.system.home is specified, so people might see different behavior depending on how they start the tests. I also see that that some methods are using the constant DEFAULT_PORT directly, which suggests we are not quite there yet... I find this code, or at least the naming and the comments, a tad confusing. I would like to remove the notion of the old harness, and can do another pass and see if we can remove it now. -- Kristian [ snip - parallelization ideas ]
Re: squeezing some multi-processing out of the nightly test runs?
Rick Hillegas <[EMAIL PROTECTED]> writes: > Mike Matrigali wrote: >> I would like to get the nightly tests to run faster on machines that >> have more than one cpu - or even hyperthreaded on a single cpu. >> Since most of the tests are single threaded I think this means >> somehow running more than one test at a time. For me >> it might even help if I was just able to run tests on 2 different >> codelines on the same machine at the same time. >> I think going forward this will be more and >> more common for developers. You can find reasonably priced laptops >> nowadays that come dual core already. >> >> I know what outstanding issue is that the network tests default to the >> same port. I believe at least for the junit tests one can set a >> different port number. Does anyone know if this fixes all the >> issues >> for the junit tests - I think I have seen some tests try different ports >> rather than the default one. > Hi Mike, > > I don't know how much work is needed to make the junit tests > independent of the port number. I can't find a JIRA which lists all of > the problem cases. However, the following JIRAs may be helpful: > DERBY-2440 and DERBY-2404. It is possible to change the port for all tests in a JUnit suite if we run the suite like this: java -Dport=XXX -DjmxPort=YYY ... Note that some tests also use port XXX+1, so you shouldn't allocate ports to close to each other if you run many JUnit processes in parallel. One thing I've considered, is to have a shell script that runs each top-level test suite in a separate process using different ports. To make such a script more maintainable (for instance, I don't like the thought of having to update the script each time a new suite is added) I was thinking that we could change the AllPackages class so that it prints the list of test suites if we execute the class from the command line. Then we could have a shell script which did something like this: port=1527 for i in $(java org.apache.derbyTesting.functionTests.suites.AllPackages) do java junit.textui.TestRunner -Dport=$port -DjmxPort=$((port+5)) $i & ((port += 10)) done >> Also what is the situation with the derbyall tests. How much work >> to get the rest of the network server tests converted to junit. >> >> On a separate note how hard would it be to get a test runner to execute >> more than one test (fixture? not sure of the junit terminology), >> at a time. I don't think we could execute them in the same db as most >> seem to assume they can use any name space and then clean up the whole >> db when they are done. From those knowledgeable about junit, should I >> look for a test runner with these features or is this just a programming >> job for derby specific tests? Maybe just a different set of suites? > I haven't experimented with this myself, but the following link might > be useful: https://parallel-junit.dev.java.net/ That looks interesting! We could perhaps have a helper method that creates a ParallelTestSuite, if that class is available, or an ordinary TestSuite if the class isn't available. As a first step to increase the parallelism, we could make TestConfiguration.defaultSuite() run the embedded test suite in parallel with the client/server test suite. Then we don't need to worry about port collisions, we only need to make sure the two threads use different databases. Many of the existing tests can probably be parallelized even further with that library without too much effort, but there are also many that need to be executed sequentially because they work on the same set of tables and require that the tearDown() method of the preceding test case has done the proper clean-up. Some of the tests also reboot the database, which could affect tests running in parallel against the same database. -- Knut Anders
Re: squeezing some multi-processing out of the nightly test runs?
Rick Hillegas <[EMAIL PROTECTED]> writes: > I don't know how much work is needed to make the junit tests > independent of the port number. I can't find a JIRA which lists all of > the problem cases. However, the following JIRAs may be helpful: > DERBY-2440 and DERBY-2404. When I last looked at this (DERBY-2440), what was missing for suites.All was just a way to specify at runtime which default port to use (instead of using the default in TestConfiguration#DEFAULT_PORT. But new tests have been added since then, and those may or may not heed TestConfiguration#DEFAULT_PORT.
Re: squeezing some multi-processing out of the nightly test runs?
Mike Matrigali wrote: I would like to get the nightly tests to run faster on machines that have more than one cpu - or even hyperthreaded on a single cpu. Since most of the tests are single threaded I think this means somehow running more than one test at a time. For me it might even help if I was just able to run tests on 2 different codelines on the same machine at the same time. I think going forward this will be more and more common for developers. You can find reasonably priced laptops nowadays that come dual core already. I know what outstanding issue is that the network tests default to the same port. I believe at least for the junit tests one can set a different port number. Does anyone know if this fixes all the issues for the junit tests - I think I have seen some tests try different ports rather than the default one. Hi Mike, I don't know how much work is needed to make the junit tests independent of the port number. I can't find a JIRA which lists all of the problem cases. However, the following JIRAs may be helpful: DERBY-2440 and DERBY-2404. Also what is the situation with the derbyall tests. How much work to get the rest of the network server tests converted to junit. On a separate note how hard would it be to get a test runner to execute more than one test (fixture? not sure of the junit terminology), at a time. I don't think we could execute them in the same db as most seem to assume they can use any name space and then clean up the whole db when they are done. From those knowledgeable about junit, should I look for a test runner with these features or is this just a programming job for derby specific tests? Maybe just a different set of suites? I haven't experimented with this myself, but the following link might be useful: https://parallel-junit.dev.java.net/ Thanks for looking into this, -Rick
