Sounds good. On Mon, Sep 19, 2016 at 4:22 PM, Darin Johnson <[email protected]> wrote:
> Alright, that was in the spirit of what I was thinking when I did this. > > Why don't we take Tim's suggested improvements to my PR (I'll do the > necessary cleanup) and at the same time just remove the platform argument > altogether since backwards compatibility isn't upsetting anyone. > > We'll still need a command line option for the launcher for now as we don't > have submodules we can decide between the two choices after we break out > submodules and improve the config. > > > On Sep 19, 2016 12:19 PM, "Tim Ellison" <[email protected]> wrote: > > > On 19/09/16 15:46, Darin Johnson wrote: > > > Hey guys, > > > > > > Thanks for looking at the PR, I apologize if it offended anyone's > eyes:). > > > > > > I'm glad it generated some discussion about the configuration. I > didn't > > > really like where things were heading with the config. However, didn't > > > want to create to much scope creep. > > > > > > I think any hierarchical config (TypeSafe or yaml) would make things > much > > > more maintainable, the plugin could simply grab the appropriate part of > > the > > > config and handle accordingly. I'd also cut down the number of command > > > line options to only those that change between runs often (like > > > input/output) > > > > > >> One option is to make Pirk pluggable, so that a Pirk installation > could > > >> use one or more of these in an extensible fashion by adding JAR files. > > >> That would still require selecting one by command-line argument. > > > > > > An argument for this approach is for lambda architecture approaches > (say > > > spark/spark-streaming) were the contents of the jars would be so > similar > > it > > > seems like to much trouble to create separate jars. > > > > > > Happy to continue working on this given some direction on where you'd > > like > > > it to go. Also, it's a bit of a blocker to refactoring the build into > > > submodules. > > > > FWIW my 2c is to not try and fix all the problems in one go, and rather > > take a compromise on the configurations while you tease apart the > > submodules in to separate source code trees, poms, etc; then come back > > and fix the runtime configs. > > > > Once the submodules are in place it will open up more work for release > > engineering and tinkering that can be done in parallel with the config > > polishing. > > > > Just a thought. > > Tim > > > > > > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <[email protected]> > > wrote: > > > > > >> On 19/09/16 13:40, Ellison Anne Williams wrote: > > >>> It seems that it's the same idea as the ResponderLauncher with the > > >> service > > >>> component added to maintain something akin to the 'platform'. I would > > >>> prefer that we just did away with the platform notion altogether and > > make > > >>> the ResponderDriver 'dumb'. We get around needing a platform-aware > > >> service > > >>> by requiring the ResponderLauncher implementation to be passed as a > CLI > > >> to > > >>> the ResponderDriver. > > >> > > >> Let me check I understand what you are saying here. > > >> > > >> At the moment, there is a monolithic Pirk that hard codes how to > respond > > >> using lots of different backends (mapreduce, spark, sparkstreaming, > > >> storm , standalone), and that is selected by command-line argument. > > >> > > >> One option is to make Pirk pluggable, so that a Pirk installation > could > > >> use one or more of these in an extensible fashion by adding JAR files. > > >> That would still require selecting one by command-line argument. > > >> > > >> A second option is to simply pass in the required backend JAR to > select > > >> the particular implementation you choose, as a specific Pirk > > >> installation doesn't need to use multiple backends simultaneously. > > >> > > >> ...and you are leaning towards the second option. Do I have that > > correct? > > >> > > >> Regards, > > >> Tim > > >> > > >>> Am I missing something? Is there a good reason to provide a service > by > > >>> which platforms are registered? I'm open... > > >>> > > >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <[email protected]> > > >> wrote: > > >>> > > >>>> How about an approach like this? > > >>>> https://github.com/tellison/incubator-pirk/tree/pirk-63 > > >>>> > > >>>> The "on-ramp" is the driver [1], which calls upon the service to > find > > a > > >>>> plug-in [2] that claims to implement the required platform > responder, > > >>>> e.g. [3]. > > >>>> > > >>>> The list of plug-ins is given in the provider's JAR file, so the > ones > > we > > >>>> provide in Pirk are listed together [4], but if you split these into > > >>>> modules, or somebody brings their own JAR alongside, these would be > > >>>> listed in each JAR's services/ directory. > > >>>> > > >>>> [1] > > >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ > > >>>> src/main/java/org/apache/pirk/responder/wideskies/ > > ResponderDriver.java > > >>>> [2] > > >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ > > >>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java > > >>>> [3] > > >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ > > >>>> src/main/java/org/apache/pirk/responder/wideskies/storm/ > > >>>> StormResponder.java > > >>>> [4] > > >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ > > >>>> src/main/services/org.apache.responder.spi.Responder > > >>>> > > >>>> I'm not even going to dignify this with a WIP PR, it is far from > > ready, > > >>>> so proceed with caution. There is hopefully enough there to show > the > > >>>> approach, and if it is worth continuing I'm happy to do so. > > >>>> > > >>>> Regards, > > >>>> Tim > > >>>> > > >>>> > > >>> > > >> > > > > > >
