Agree - let's leave the config/CLI the way it is for now and tackle that as
a subsequent design discussion and PR.

Also, I think that we should leave the ResponderDriver and the
ResponderProps alone for this PR and push to a subsequent PR (once we
decide if and how we would like to delegate each).

I vote to remove the 'platform' option and the backwards compatibility in
this PR and proceed with having a ResponderLauncher interface and forcing
its implementation by the ResponderDriver.

And, I am not so concerned with having one fat jar vs. multiple jars right
now - to me, at this point, it's a 'nice to have' and not a 'must have' for
Pirk functionality. We do need to break out Pirk into more clearly defined
submodules (which is in progress) - via this re-factor, I think that we
will gain some ability to generate multiple jars which is nice.



On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <t.p.elli...@gmail.com> wrote:

> On 19/09/16 15:46, Darin Johnson wrote:
> > Hey guys,
> >
> > Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
> >
> > I'm glad it generated some discussion about the configuration.  I didn't
> > really like where things were heading with the config.  However, didn't
> > want to create to much scope creep.
> >
> > I think any hierarchical config (TypeSafe or yaml) would make things much
> > more maintainable, the plugin could simply grab the appropriate part of
> the
> > config and handle accordingly.  I'd also cut down the number of command
> > line options to only those that change between runs often (like
> > input/output)
> >
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >
> > An argument for this approach is for lambda architecture approaches (say
> > spark/spark-streaming) were the contents of the jars would be so similar
> it
> > seems like to much trouble to create separate jars.
> >
> > Happy to continue working on this given some direction on where you'd
> like
> > it to go.  Also, it's a bit of a blocker to refactoring the build into
> > submodules.
>
> FWIW my 2c is to not try and fix all the problems in one go, and rather
> take a compromise on the configurations while you tease apart the
> submodules in to separate source code trees, poms, etc; then come back
> and fix the runtime configs.
>
> Once the submodules are in place it will open up more work for release
> engineering and tinkering that can be done in parallel with the config
> polishing.
>
> Just a thought.
> Tim
>
>
> > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <t.p.elli...@gmail.com>
> wrote:
> >
> >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> >>> It seems that it's the same idea as the ResponderLauncher with the
> >> service
> >>> component added to maintain something akin to the 'platform'. I would
> >>> prefer that we just did away with the platform notion altogether and
> make
> >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> >> service
> >>> by requiring the ResponderLauncher implementation to be passed as a CLI
> >> to
> >>> the ResponderDriver.
> >>
> >> Let me check I understand what you are saying here.
> >>
> >> At the moment, there is a monolithic Pirk that hard codes how to respond
> >> using lots of different backends (mapreduce, spark, sparkstreaming,
> >> storm , standalone), and that is selected by command-line argument.
> >>
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >>
> >> A second option is to simply pass in the required backend JAR to select
> >> the particular implementation you choose, as a specific Pirk
> >> installation doesn't need to use multiple backends simultaneously.
> >>
> >> ...and you are leaning towards the second option.  Do I have that
> correct?
> >>
> >> Regards,
> >> Tim
> >>
> >>> Am I missing something? Is there a good reason to provide a service by
> >>> which platforms are registered? I'm open...
> >>>
> >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <t.p.elli...@gmail.com>
> >> wrote:
> >>>
> >>>> How about an approach like this?
> >>>>    https://github.com/tellison/incubator-pirk/tree/pirk-63
> >>>>
> >>>> The "on-ramp" is the driver [1], which calls upon the service to find
> a
> >>>> plug-in [2] that claims to implement the required platform responder,
> >>>> e.g. [3].
> >>>>
> >>>> The list of plug-ins is given in the provider's JAR file, so the ones
> we
> >>>> provide in Pirk are listed together [4], but if you split these into
> >>>> modules, or somebody brings their own JAR alongside, these would be
> >>>> listed in each JAR's services/ directory.
> >>>>
> >>>> [1]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/wideskies/
> ResponderDriver.java
> >>>> [2]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
> >>>> [3]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/java/org/apache/pirk/responder/wideskies/storm/
> >>>> StormResponder.java
> >>>> [4]
> >>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >>>> src/main/services/org.apache.responder.spi.Responder
> >>>>
> >>>> I'm not even going to dignify this with a WIP PR, it is far from
> ready,
> >>>> so proceed with caution.  There is hopefully enough there to show the
> >>>> approach, and if it is worth continuing I'm happy to do so.
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>>
> >>>
> >>
> >
>

Reply via email to