Oops, different context: I meant about automated testing of the mahout multi-stage jobs. If one of the stages gets unruly in its memory use, that is a regression in software quality that is hard to spot.
On Wed, Aug 10, 2011 at 11:54 PM, Sean Owen <[email protected]> wrote: > That would reduce memory requirements unilaterally? no, don't think so. I > have not run into memory problems. > The issue here is execution time and it's a big problem indeed. > > Would anyone object to disabling this test for now? it's getting costly in > the dev cycle. > > On Thu, Aug 11, 2011 at 5:52 AM, Lance Norskog <[email protected]> wrote: > >> Another aspect of testing Map/Reduce jobs is memory bounds. An M/R job >> should be able to run in a fairly constant amount of ram per JVM from >> beginning to end, to avoid blowing up late in the game. Is there any >> harness around that would do this? >> >> Lance >> >> On Sun, Aug 7, 2011 at 9:04 PM, Ted Dunning <[email protected]> wrote: >> > We have used testNg at work for just this segregation of fast and slow >> tests and it works pretty well. It is also useful for stripping out >> cascading failures. This means that integration tests can be executed only >> if the underlying unit tests succeed. Another nice thing is that testNg says >> how many tests were skipped. >> > >> > Junit has recently been adding similar capabilities but I haven't kept >> up. >> > >> > Sent from my iPhone >> > >> > On Aug 7, 2011, at 19:18, Grant Ingersoll <[email protected]> wrote: >> > >> >> It would likely help if we could get them to run in parallel, perhaps. >> Also, seems like TestNG might have some better features on paper for this >> kind of stuff (I think you can annotate some things as "slow" or "fast" and >> then choose to run them separately). I haven't explored much yet in this >> way. Has anyone else used TestNG? >> >> >> >> -Grant >> >> >> >> On Aug 7, 2011, at 9:12 PM, Ted Dunning wrote: >> >> >> >>> I really don't think that this distinction needs to be made. The >> >>> distinction between unit and integration test is important from a >> technical >> >>> point of view, but as an organizing principle the topic and target of >> the >> >>> test is probably a better idea than whether the test is a functional or >> unit >> >>> test or whether it has randomized initial conditions or whether it has >> for >> >>> loops in it. Tests should be organized by what they test. >> >>> >> >>> On Sun, Aug 7, 2011 at 5:08 PM, Lance Norskog <[email protected]> >> wrote: >> >>> >> >>>> Sure! Perhaps the long-running ones can move to a new 'regression' >> >>>> area? examples/ is partly what these are, so examples/regression makes >> >>>> sense. >> >>>> >> >>>> On Sun, Aug 7, 2011 at 11:11 AM, Sean Owen <[email protected]> wrote: >> >>>>> This test is indeed by far the culprit. I already reduced its test >> input >> >>>>> size to hurry it up, but it's gone slow again. >> >>>>> >> >>>>> Lance, indeed, these are not all unit tests -- nobody said they were. >> The >> >>>>> test is useful. >> >>>>> >> >>>>> I do suggest, however, we comment it out. Jake suggested it coudl be >> made >> >>>>> faster but I don't think he followed up. >> >>>>> >> >>>>> Sean >> >>>>> >> >>>>> On Sun, Aug 7, 2011 at 12:13 AM, Lance Norskog <[email protected]> >> >>>> wrote: >> >>>>> >> >>>>>> Comment out DistributedLanczosWhatsit. Zing! >> >>>>>> >> >>>>>> A unit test takes a bit of code X and checks that code path A goes >> >>>>>> "tick" and code path B goes "tock" and bogus input C throws an >> >>>>>> exception. There's no such thing as a "unit test" that runs twelve >> M/R >> >>>>>> jobs in a row. >> >>>>>> >> >>>>>> There's MRUnit, which seems trapped in the Hadoop >> 0.20/0.21/0.22/0.23 >> >>>>>> morass. This is a squib about how to do unit testing of mappers and >> >>>>>> reducers with Mockito: >> >>>>>> >> >>>>>> http://nubetech.co/testing-hadoop-map-reduce-jobs >> >>>>>> >> >>>>>> What the Mahout jobs want is more of a regression test, which would >> >>>>>> have two purposes: >> >>>>>> 1) does the whole orchestration still work, and >> >>>>>> 2) does it still acquire the information it is supposed to acquire? >> >>>>>> 2a) this requires some amount of real data and a "gold standard" >> >>>>>> output to match against. >> >>>>>> >> >>>>>> On Sat, Aug 6, 2011 at 12:34 PM, Grant Ingersoll < >> [email protected]> >> >>>>>> wrote: >> >>>>>>> Granted, I'm on a slow machine, but our tests take forever to run. >> On >> >>>> an >> >>>>>> 2 core MBP, it takes well over an hour to run all the tests (I did >> just >> >>>>>> order a new MBP, so it will be faster, but it doesn't lend itself to >> a >> >>>> good >> >>>>>> OOTB experience for people) >> >>>>>>> >> >>>>>>> One idea would be to add in parallel test execution in Maven. I >> think >> >>>>>> this requires Mvn 3, but I am not sure. Another is to take a look >> at >> >>>> our >> >>>>>> tests, especially the slow ones and see if we can speed them up. >> >>>>>>> >> >>>>>>> When I try adding in parallel tests to Maven, I get a bunch of >> >>>> failures >> >>>>>> in the tests. >> >>>>>>> >> >>>>>>> I was using: >> >>>>>>> <plugin> >> >>>>>>> <groupId>org.apache.maven.plugins</groupId> >> >>>>>>> <artifactId>maven-surefire-plugin</artifactId> >> >>>>>>> <configuration> >> >>>>>>> <forkMode>once</forkMode> >> >>>>>>> <argLine>-Xms256m -Xmx512m</argLine> >> >>>>>>> <testFailureIgnore>false</testFailureIgnore> >> >>>>>>> <redirectTestOutputToFile>true</redirectTestOutputToFile> >> >>>>>>> <parallel>classes</parallel> >> >>>>>>> <threadCount>5</threadCount> >> >>>>>>> </configuration> >> >>>>>>> </plugin> >> >>>>>>> >> >>>>>>> Anyone played around with this stuff? I suspect the failures are >> due >> >>>> to >> >>>>>> tests stomping on each other, but I am still digging in. >> >>>>>>> >> >>>>>>> -Grant >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Lance Norskog >> >>>>>> [email protected] >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Lance Norskog >> >>>> [email protected] >> >>>> >> >> >> >> -------------------------------------------- >> >> Grant Ingersoll >> >> >> >> >> > >> >> >> >> -- >> Lance Norskog >> [email protected] >> > -- Lance Norskog [email protected]
