On 19/09/16 15:46, Darin Johnson wrote: > Hey guys, > > Thanks for looking at the PR, I apologize if it offended anyone's eyes:). > > I'm glad it generated some discussion about the configuration. I didn't > really like where things were heading with the config. However, didn't > want to create to much scope creep. > > I think any hierarchical config (TypeSafe or yaml) would make things much > more maintainable, the plugin could simply grab the appropriate part of the > config and handle accordingly. I'd also cut down the number of command > line options to only those that change between runs often (like > input/output) > >> One option is to make Pirk pluggable, so that a Pirk installation could >> use one or more of these in an extensible fashion by adding JAR files. >> That would still require selecting one by command-line argument. > > An argument for this approach is for lambda architecture approaches (say > spark/spark-streaming) were the contents of the jars would be so similar it > seems like to much trouble to create separate jars. > > Happy to continue working on this given some direction on where you'd like > it to go. Also, it's a bit of a blocker to refactoring the build into > submodules.
FWIW my 2c is to not try and fix all the problems in one go, and rather take a compromise on the configurations while you tease apart the submodules in to separate source code trees, poms, etc; then come back and fix the runtime configs. Once the submodules are in place it will open up more work for release engineering and tinkering that can be done in parallel with the config polishing. Just a thought. Tim > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <t.p.elli...@gmail.com> wrote: > >> On 19/09/16 13:40, Ellison Anne Williams wrote: >>> It seems that it's the same idea as the ResponderLauncher with the >> service >>> component added to maintain something akin to the 'platform'. I would >>> prefer that we just did away with the platform notion altogether and make >>> the ResponderDriver 'dumb'. We get around needing a platform-aware >> service >>> by requiring the ResponderLauncher implementation to be passed as a CLI >> to >>> the ResponderDriver. >> >> Let me check I understand what you are saying here. >> >> At the moment, there is a monolithic Pirk that hard codes how to respond >> using lots of different backends (mapreduce, spark, sparkstreaming, >> storm , standalone), and that is selected by command-line argument. >> >> One option is to make Pirk pluggable, so that a Pirk installation could >> use one or more of these in an extensible fashion by adding JAR files. >> That would still require selecting one by command-line argument. >> >> A second option is to simply pass in the required backend JAR to select >> the particular implementation you choose, as a specific Pirk >> installation doesn't need to use multiple backends simultaneously. >> >> ...and you are leaning towards the second option. Do I have that correct? >> >> Regards, >> Tim >> >>> Am I missing something? Is there a good reason to provide a service by >>> which platforms are registered? I'm open... >>> >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <t.p.elli...@gmail.com> >> wrote: >>> >>>> How about an approach like this? >>>> https://github.com/tellison/incubator-pirk/tree/pirk-63 >>>> >>>> The "on-ramp" is the driver [1], which calls upon the service to find a >>>> plug-in [2] that claims to implement the required platform responder, >>>> e.g. [3]. >>>> >>>> The list of plug-ins is given in the provider's JAR file, so the ones we >>>> provide in Pirk are listed together [4], but if you split these into >>>> modules, or somebody brings their own JAR alongside, these would be >>>> listed in each JAR's services/ directory. >>>> >>>> [1] >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>> src/main/java/org/apache/pirk/responder/wideskies/ResponderDriver.java >>>> [2] >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java >>>> [3] >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>> src/main/java/org/apache/pirk/responder/wideskies/storm/ >>>> StormResponder.java >>>> [4] >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>> src/main/services/org.apache.responder.spi.Responder >>>> >>>> I'm not even going to dignify this with a WIP PR, it is far from ready, >>>> so proceed with caution. There is hopefully enough there to show the >>>> approach, and if it is worth continuing I'm happy to do so. >>>> >>>> Regards, >>>> Tim >>>> >>>> >>> >> >
