Alright, that was in the spirit of what I was thinking when I did this.

Why don't we take Tim's suggested improvements to my PR (I'll do the
necessary cleanup) and at the same time just remove the platform argument
altogether since backwards compatibility isn't upsetting anyone.

We'll still need a command line option for the launcher for now as we don't
have submodules we can decide between the two choices after we break out
submodules and improve the config.


On Sep 19, 2016 12:19 PM, "Tim Ellison" <t.p.elli...@gmail.com> wrote:

> On 19/09/16 15:46, Darin Johnson wrote:
> > Hey guys,
> >
> > Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
> >
> > I'm glad it generated some discussion about the configuration.  I didn't
> > really like where things were heading with the config.  However, didn't
> > want to create to much scope creep.
> >
> > I think any hierarchical config (TypeSafe or yaml) would make things much
> > more maintainable, the plugin could simply grab the appropriate part of
> the
> > config and handle accordingly.  I'd also cut down the number of command
> > line options to only those that change between runs often (like
> > input/output)
> >
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >
> > An argument for this approach is for lambda architecture approaches (say
> > spark/spark-streaming) were the contents of the jars would be so similar
> it
> > seems like to much trouble to create separate jars.
> >
> > Happy to continue working on this given some direction on where you'd
> like
> > it to go.  Also, it's a bit of a blocker to refactoring the build into
> > submodules.
>
> FWIW my 2c is to not try and fix all the problems in one go, and rather
> take a compromise on the configurations while you tease apart the
> submodules in to separate source code trees, poms, etc; then come back
> and fix the runtime configs.
>
> Once the submodules are in place it will open up more work for release
> engineering and tinkering that can be done in parallel with the config
> polishing.
>
> Just a thought.
> Tim
>
>
> > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <t.p.elli...@gmail.com>
> wrote:
> >
> >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> >>> It seems that it's the same idea as the ResponderLauncher with the
> >> service
> >>> component added to maintain something akin to the 'platform'. I would
> >>> prefer that we just did away with the platform notion altogether and
> make
> >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> >> service
> >>> by requiring the ResponderLauncher implementation to be passed as a CLI
> >> to
> >>> the ResponderDriver.
> >>
> >> Let me check I understand what you are saying here.
> >>
> >> At the moment, there is a monolithic Pirk that hard codes how to respond
> >> using lots of different backends (mapreduce, spark, sparkstreaming,
> >> storm , standalone), and that is selected by command-line argument.
> >>
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >>
> >> A second option is to simply pass in the required backend JAR to select
> >> the particular implementation you choose, as a specific Pirk
> >> installation doesn't need to use multiple backends simultaneously.
> >>
> >> ...and you are leaning towards the second option.  Do I have that
> correct?
> >>
> >> Regards,
> >> Tim
> >>
> >>> Am I missing something? Is there a good reason to provide a service by
> >>> which platforms are registered? I'm open...
> >>>
> >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <t.p.elli...@gmail.com>
> >> wrote:
> >>>
> >>>> How about an approach like this?
> >>>>    https://github.com/tellison/incubator-pirk/tree/pirk-63
> >>>>
> >>>> The "on-ramp" is the driver [1], which calls upon the service to find
> a
> >>>> plug-in [2] that claims to implement the required platform responder,
> >>>> e.g. [3].
> >>>>
> >>>> The list of plug-ins is given in the provider's JAR file, so the ones
> we
> >>>> provide in Pirk are listed together [4], but if you split these into
> >>>> modules, or somebody brings their own JAR alongside, these would be
> >>>> listed in each JAR's services/ directory.
> >>>>
> >>>> [1]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/wideskies/
> ResponderDriver.java
> >>>> [2]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
> >>>> [3]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/wideskies/storm/
> >>>> StormResponder.java
> >>>> [4]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/services/org.apache.responder.spi.Responder
> >>>>
> >>>> I'm not even going to dignify this with a WIP PR, it is far from
> ready,
> >>>> so proceed with caution.  There is hopefully enough there to show the
> >>>> approach, and if it is worth continuing I'm happy to do so.
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>>
> >>>
> >>
> >
>

Reply via email to