Re: Distributed tests take a long time, and that's ok.
Thanks for adding the instructions for Bluemix in the PR today - I will check them out and try to run everything on Bluemix. Yes, we should print 'SUCESS' or 'FAILURE' messages at the end of the test runs... On Tue, Oct 11, 2016 at 8:06 AM, Tim Ellison wrote: > On 06/10/16 14:32, Walter Ray-Dulany wrote: > > Tim Ellison noted in a recent email to the list (subject: [CANCEL] > [VOTE]: > > Apache Pirk 0.2.0-incubating Realse) the following: > > > >> The distributed tests took a long time (~35mins IIRC), is that normal? > > > > I feel like noting this in a separate thread, and discussing it here, is > > worthwhile. > > > > The distributed tests *do* take a long time. The reason isn't because any > > one test is slow; indeed, most of the individual tests take only a > handful > > of seconds. The slowness is because of the very large number of tests. > This > > vast array of tests, upon inspection, can be seen to be caused by a very > > thorough testing of the large number of different actions that can be > > performed by Pirk and platforms it supports. > > There was certainly lots of logging going by - not that I was watching > it constantly for half an hour - so I appreciate there is plenty of code > being run and I trust that it is doing something useful ;-) > > I was also trusting that if there was a failure, it would be obvious, > and there was nothing that said "FAILURE" at the end of the run, so I > took that as a good sign. > > > I think this is ok. As an example of why this thorough testing is > > necessary, just yesterday, while finishing up PIRK-45, I ran into an > error > > 15 minutes into the distributed tests. Without this thorough testing, > that > > code would have made it to a PR yesterday night, and (without the > > distributed tests) it is likely no one would have caught it (it was a > > subtle "Spark has an ancient package on its classpath pre-empting a newer > > version I'd included, and the old package has a bug" error). > > Testing is good. Automatic background testing for this type of test > suite may be better. I'm not likely to run the dist tests on every > pause in development if they take a long time, and cost real money (a > single run on AWS reported that it would cost me $2.50-ish*) > > It does run faster on the IBM JVM, and I can run it faster and for free > on Bluemix, so I'll go that route for the moment -- but don't want to > loose sight of trying to make the testing readily available for anyone > who drops by in the community. As mooted earlier, maybe the best way to > do that is investigate a shared resource test environment hosted at the > ASF. > > * I know, I'm cheap :-) > > Regards, > Tim >
Re: Distributed tests take a long time, and that's ok.
On 06/10/16 14:32, Walter Ray-Dulany wrote: > Tim Ellison noted in a recent email to the list (subject: [CANCEL] [VOTE]: > Apache Pirk 0.2.0-incubating Realse) the following: > >> The distributed tests took a long time (~35mins IIRC), is that normal? > > I feel like noting this in a separate thread, and discussing it here, is > worthwhile. > > The distributed tests *do* take a long time. The reason isn't because any > one test is slow; indeed, most of the individual tests take only a handful > of seconds. The slowness is because of the very large number of tests. This > vast array of tests, upon inspection, can be seen to be caused by a very > thorough testing of the large number of different actions that can be > performed by Pirk and platforms it supports. There was certainly lots of logging going by - not that I was watching it constantly for half an hour - so I appreciate there is plenty of code being run and I trust that it is doing something useful ;-) I was also trusting that if there was a failure, it would be obvious, and there was nothing that said "FAILURE" at the end of the run, so I took that as a good sign. > I think this is ok. As an example of why this thorough testing is > necessary, just yesterday, while finishing up PIRK-45, I ran into an error > 15 minutes into the distributed tests. Without this thorough testing, that > code would have made it to a PR yesterday night, and (without the > distributed tests) it is likely no one would have caught it (it was a > subtle "Spark has an ancient package on its classpath pre-empting a newer > version I'd included, and the old package has a bug" error). Testing is good. Automatic background testing for this type of test suite may be better. I'm not likely to run the dist tests on every pause in development if they take a long time, and cost real money (a single run on AWS reported that it would cost me $2.50-ish*) It does run faster on the IBM JVM, and I can run it faster and for free on Bluemix, so I'll go that route for the moment -- but don't want to loose sight of trying to make the testing readily available for anyone who drops by in the community. As mooted earlier, maybe the best way to do that is investigate a shared resource test environment hosted at the ASF. * I know, I'm cheap :-) Regards, Tim
Re: Distributed tests take a long time, and that's ok.
The tests were designed to be Responder platform centric. So, if one want to test Pirk on Spark, then they can run the corresponding test suite. The individual tests within each Responder platform scope test test the various cases (and corner cases) of Pirk. One correction to what you wrote about test input - the Inputs for the tests (see the DistributedTestDriver initialize(fs) method), are created once per run of the driver - if multiple platforms are being tested, the inputs are only created one time. I am not at all opposed to refactoring the tests (they absolutely need a bit of cleanup) On Thu, Oct 6, 2016 at 6:39 PM, Darin Johnson wrote: > Being pretty familiar with that section of the code base right now. I > think a few things could be done. Regardless I'm refactoring it for > submodules do it'll be easier to iterate these. The main place I'd start is > currently each platform test creates a new input, which means the querier > logic is repeated for the number of platforms tested - setting up the test > inputs once, running all platforms for that test and then tearing down the > inputs would be a start. > > > Maybe have all tests implement an interface like: > > Public interface BaseTest { > > > public void createInputs(...); > > > public boolean checkOutputs(List results); > > } > > > The a runner could do: > > BaseTest myTest=new myTest() > > myTest.createInputs(...) > > For(responder:responders) > > { > > results = responder.runDistributedTest(); > > If(!myTest.checkResults(results) fail(); > > } > > > If nothing else it'd make adding tests easier. > > > > On Oct 6, 2016 9:33 AM, "Walter Ray-Dulany" wrote: > > > Tim Ellison noted in a recent email to the list (subject: [CANCEL] > [VOTE]: > > Apache Pirk 0.2.0-incubating Realse) the following: > > > > > The distributed tests took a long time (~35mins IIRC), is that normal? > > > > I feel like noting this in a separate thread, and discussing it here, is > > worthwhile. > > > > The distributed tests *do* take a long time. The reason isn't because any > > one test is slow; indeed, most of the individual tests take only a > handful > > of seconds. The slowness is because of the very large number of tests. > This > > vast array of tests, upon inspection, can be seen to be caused by a very > > thorough testing of the large number of different actions that can be > > performed by Pirk and platforms it supports. > > > > I think this is ok. As an example of why this thorough testing is > > necessary, just yesterday, while finishing up PIRK-45, I ran into an > error > > 15 minutes into the distributed tests. Without this thorough testing, > that > > code would have made it to a PR yesterday night, and (without the > > distributed tests) it is likely no one would have caught it (it was a > > subtle "Spark has an ancient package on its classpath pre-empting a newer > > version I'd included, and the old package has a bug" error). > > >
Re: Distributed tests take a long time, and that's ok.
Being pretty familiar with that section of the code base right now. I think a few things could be done. Regardless I'm refactoring it for submodules do it'll be easier to iterate these. The main place I'd start is currently each platform test creates a new input, which means the querier logic is repeated for the number of platforms tested - setting up the test inputs once, running all platforms for that test and then tearing down the inputs would be a start. Maybe have all tests implement an interface like: Public interface BaseTest { public void createInputs(...); public boolean checkOutputs(List results); } The a runner could do: BaseTest myTest=new myTest() myTest.createInputs(...) For(responder:responders) { results = responder.runDistributedTest(); If(!myTest.checkResults(results) fail(); } If nothing else it'd make adding tests easier. On Oct 6, 2016 9:33 AM, "Walter Ray-Dulany" wrote: > Tim Ellison noted in a recent email to the list (subject: [CANCEL] [VOTE]: > Apache Pirk 0.2.0-incubating Realse) the following: > > > The distributed tests took a long time (~35mins IIRC), is that normal? > > I feel like noting this in a separate thread, and discussing it here, is > worthwhile. > > The distributed tests *do* take a long time. The reason isn't because any > one test is slow; indeed, most of the individual tests take only a handful > of seconds. The slowness is because of the very large number of tests. This > vast array of tests, upon inspection, can be seen to be caused by a very > thorough testing of the large number of different actions that can be > performed by Pirk and platforms it supports. > > I think this is ok. As an example of why this thorough testing is > necessary, just yesterday, while finishing up PIRK-45, I ran into an error > 15 minutes into the distributed tests. Without this thorough testing, that > code would have made it to a PR yesterday night, and (without the > distributed tests) it is likely no one would have caught it (it was a > subtle "Spark has an ancient package on its classpath pre-empting a newer > version I'd included, and the old package has a bug" error). >