On 04/10/16 07:19, Darin Johnson wrote: > Spent some time working on this tonight. Currently it's here: > https://github.com/DarinJ/incubator-pirk/tree/submodules
Thanks Darin, looking good. My first thought is that the StandaloneResponder look s a bit out of place now in core, but it may be overkill to pull it out into its own module. WDYT? > if anyone wants to > provide a sanity check on how I've modified the ResponderPlugin to handle > the distributed tests (Look at the ResponderPlugin and maybe > SparkResponder). Here we added a hasDistributedTest and > runDistributedTest, the reason for hasDistributedTest is that Storm didn't > have a test in BaseTests. Yeah, though in time I hope that the BaseTests don't differentiate on distributed vs stand alone tests. I'm afraid it is a bit of a Pirk-ism for the code to know about implementation details elsewhere. IMHO the various test queries should run on all available plug-ins, and not be concerned with the distributed/otherwise nature of the implementation. > The basic idea is to provide the Plugins the ability to setup their own > distributed test and provide BaseTests back with the list of objects. I > moved performQuery from DistTestSuite to BaseTests and made it private, > this got rid of a circular dependency. There's a few things I don't > particularly like the setup necessarily handled in two functions and > there's a lot of file reads/writes. I don't think this can be helped due > to the nature of the tests. I'm also torn on calling main, using > ProcessBuilder, each frameworks special pragmatic launcher. It's a journey and this is a big step in the right direction :-) > Please note at present, you have to specify a test to run (i.e. -t 1:JS), > this is because not all responders are in all jars now (so some tests fail > saying plugin not found). That should be corrected tomorrow, I wouldn't > recommend going that far. I'll have the TestSuite look for available > platforms that have a distributed test and then run them. I also plan to > start consolidating functions in DistTestSuite.java and BaseTests.java. > > Also in BaseTests, it seems really awkward that Standalone is there but has > special performQuery methods and special results. Is there a particular > reason for this? It would be nice to make it so standalone was tested the > same way. Definitely. > The other modules should be easier to break out. I have bothered > optimizing maven yet. I assume you mean for the submodule dependencies etc? No problem. Your branch is testing ok for me locally: [INFO] Apache Pirk (incubating) Project ................... SUCCESS [ 1.392 s] [INFO] pirk-core .......................................... SUCCESS [01:17 min] [INFO] pirk-storm ......................................... SUCCESS [ 54.000 s] [INFO] pirk-spark ......................................... SUCCESS [ 3.275 s] [INFO] pirk-mapreduce ..................................... SUCCESS [ 1.095 s] The overall direction is good. Once the release is cut, and you are happy that it is reasonably in shape, I think it would be good to switch over to the modular builds so we can collaborate on the refinements. Regards, Tim > On Thu, Sep 29, 2016 at 7:39 AM, Darin Johnson <[email protected]> > wrote: > >> Great, will do. Unfortunately, I don't think this will be done by end of >> week. We'll see. >> >> On Sep 29, 2016 6:32 AM, "Ellison Anne Williams" <[email protected]> >> wrote: >> >>> I would say that the DistributedTestSuite is ripe for refactor - it grew >>> pretty organically and hasn't had a good refactoring yet. ;) >>> >>> What you suggested sounds reasonable - look forward to seeing the code. >>> >>> On Thu, Sep 29, 2016 at 4:48 AM, Tim Ellison <[email protected]> >>> wrote: >>> >>>> On 29/09/16 01:50, Darin Johnson wrote: >>>>> Hey guys, started working on refactoring into submodules and noticed a >>>> few >>>>> things. >>>>> >>>>> First the class DistributedTestSuite in org.apache.pirk.tests package >>> is >>>>> directly calling SparkLauncher, which can be avoided using platform >>>>> (already a TODO). This will be needed to break the spark responder >>> out. >>>> No >>>>> issues here' expected it. >>>>> >>>>> Also noticed that DistributedTestSuite calls >>>> BaseTests.testDNSHostnameQuery >>>>> which in turn calls the static method performQuery in >>>>> DistTestSuite.performQuery so we've got a cyclic dependency. I'm >>>> thinking >>>>> this should be refactored, which shouldn't be hard - most of the >>> method >>>> are >>>>> static. >>>>> >>>>> I figured I should solicit some thoughts on the best approach to going >>>>> about this before jumping in. Comments welcome, especially as I >>> haven't >>>>> written anything yet :). >>>>> >>>>> My initial thoughts are the DistTestSuite should see what distributed >>>>> frameworks are available (via plugin) and then run those tests. This >>>> will >>>>> likely require a setup helper for each framework though - the >>>> DistTestSuite >>>>> might then need to skip frameworks where the setup helper isn't found. >>>> If >>>>> this seems to be the approach people would like I'll start stubbing it >>>> out >>>>> and get some opinions. >>>> >>>> Yep, sounds reasonable. The distributed tests should discover what >>>> platforms are available and run on them, perhaps printing a warning if >>>> an expected project platform is not found. >>>> >>>> Looking forward to seeing the code. >>>> >>>> Regards, >>>> Tim >>>> >>>> >>> >> >
