On 19/09/16 21:22, Darin Johnson wrote: > Alright, that was in the spirit of what I was thinking when I did this. > > Why don't we take Tim's suggested improvements to my PR (I'll do the > necessary cleanup) and at the same time just remove the platform argument > altogether since backwards compatibility isn't upsetting anyone. > > We'll still need a command line option for the launcher for now as we don't > have submodules we can decide between the two choices after we break out > submodules and improve the config.
Sounds great - let me know how I can help. Regards, Tim > On Sep 19, 2016 12:19 PM, "Tim Ellison" <[email protected]> wrote: > >> On 19/09/16 15:46, Darin Johnson wrote: >>> Hey guys, >>> >>> Thanks for looking at the PR, I apologize if it offended anyone's eyes:). >>> >>> I'm glad it generated some discussion about the configuration. I didn't >>> really like where things were heading with the config. However, didn't >>> want to create to much scope creep. >>> >>> I think any hierarchical config (TypeSafe or yaml) would make things much >>> more maintainable, the plugin could simply grab the appropriate part of >> the >>> config and handle accordingly. I'd also cut down the number of command >>> line options to only those that change between runs often (like >>> input/output) >>> >>>> One option is to make Pirk pluggable, so that a Pirk installation could >>>> use one or more of these in an extensible fashion by adding JAR files. >>>> That would still require selecting one by command-line argument. >>> >>> An argument for this approach is for lambda architecture approaches (say >>> spark/spark-streaming) were the contents of the jars would be so similar >> it >>> seems like to much trouble to create separate jars. >>> >>> Happy to continue working on this given some direction on where you'd >> like >>> it to go. Also, it's a bit of a blocker to refactoring the build into >>> submodules. >> >> FWIW my 2c is to not try and fix all the problems in one go, and rather >> take a compromise on the configurations while you tease apart the >> submodules in to separate source code trees, poms, etc; then come back >> and fix the runtime configs. >> >> Once the submodules are in place it will open up more work for release >> engineering and tinkering that can be done in parallel with the config >> polishing. >> >> Just a thought. >> Tim >> >> >>> On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <[email protected]> >> wrote: >>> >>>> On 19/09/16 13:40, Ellison Anne Williams wrote: >>>>> It seems that it's the same idea as the ResponderLauncher with the >>>> service >>>>> component added to maintain something akin to the 'platform'. I would >>>>> prefer that we just did away with the platform notion altogether and >> make >>>>> the ResponderDriver 'dumb'. We get around needing a platform-aware >>>> service >>>>> by requiring the ResponderLauncher implementation to be passed as a CLI >>>> to >>>>> the ResponderDriver. >>>> >>>> Let me check I understand what you are saying here. >>>> >>>> At the moment, there is a monolithic Pirk that hard codes how to respond >>>> using lots of different backends (mapreduce, spark, sparkstreaming, >>>> storm , standalone), and that is selected by command-line argument. >>>> >>>> One option is to make Pirk pluggable, so that a Pirk installation could >>>> use one or more of these in an extensible fashion by adding JAR files. >>>> That would still require selecting one by command-line argument. >>>> >>>> A second option is to simply pass in the required backend JAR to select >>>> the particular implementation you choose, as a specific Pirk >>>> installation doesn't need to use multiple backends simultaneously. >>>> >>>> ...and you are leaning towards the second option. Do I have that >> correct? >>>> >>>> Regards, >>>> Tim >>>> >>>>> Am I missing something? Is there a good reason to provide a service by >>>>> which platforms are registered? I'm open... >>>>> >>>>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <[email protected]> >>>> wrote: >>>>> >>>>>> How about an approach like this? >>>>>> https://github.com/tellison/incubator-pirk/tree/pirk-63 >>>>>> >>>>>> The "on-ramp" is the driver [1], which calls upon the service to find >> a >>>>>> plug-in [2] that claims to implement the required platform responder, >>>>>> e.g. [3]. >>>>>> >>>>>> The list of plug-ins is given in the provider's JAR file, so the ones >> we >>>>>> provide in Pirk are listed together [4], but if you split these into >>>>>> modules, or somebody brings their own JAR alongside, these would be >>>>>> listed in each JAR's services/ directory. >>>>>> >>>>>> [1] >>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>> src/main/java/org/apache/pirk/responder/wideskies/ >> ResponderDriver.java >>>>>> [2] >>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java >>>>>> [3] >>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>> src/main/java/org/apache/pirk/responder/wideskies/storm/ >>>>>> StormResponder.java >>>>>> [4] >>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>> src/main/services/org.apache.responder.spi.Responder >>>>>> >>>>>> I'm not even going to dignify this with a WIP PR, it is far from >> ready, >>>>>> so proceed with caution. There is hopefully enough there to show the >>>>>> approach, and if it is worth continuing I'm happy to do so. >>>>>> >>>>>> Regards, >>>>>> Tim >>>>>> >>>>>> >>>>> >>>> >>> >> >
