Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Ellison Anne Williams]

Why are we using the service-plugin model when we are dropping the platform
designations? Thought that we were going to drop all central platform
awareness from Pirk in favor of the ResponderLauncher. The service-plugin
model retains it in a roundabout way as we have to statically maintain the
list of known responders in
org.apache.pirk.responder.wideskies.spi.ResponderPlugin.

On Thu, Sep 22, 2016 at 5:36 AM, ellisonanne  wrote:

> Github user ellisonanne commented on a diff in the pull request:
>
> https://github.com/apache/incubator-pirk/pull/93#discussion_r80005357
>
> --- Diff: src/main/resources/META-INF/services/org.apache.pirk.
> responder.wideskies.spi.ResponderPlugin ---
> @@ -0,0 +1,5 @@
> +org.apache.pirk.responder.wideskies.mapreduce.MapReduceResponder
> --- End diff --
>
> Why are we using the service-plugin model when we are dropping the
> platform designations?
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Ellison Anne Williams]

I am in favor of breaking out pirk-core as specified so that our initial
submodule structure would be as follows:

pirk-core (encryption,query, inputformat, serialization, utils)

pirk-responder (core responder incl. standalone)

pirk-querier

pirk-storm

pirk-mapreduce

pirk-spark

pirk-benchmark

pirk-distributed-test


One thing to note is that under this breakdown, pirk-core would not include
the Elasticsearch dependency (es-hadoop). The only submodules that would
have the es-hadoop dependency (those which need it) currently are
pirk-mapreduce, pirk-spark, and pirk-distributed-test.


I believe that we agreed (somewhere :)) in this thread to go ahead and
remove the platform 'backwards compatibility' for PIRK-63. Please holler if
this is not correct.




On Tue, Sep 20, 2016 at 9:40 PM, Darin Johnson 
wrote:

> Suneel, a google doc as promised, only a day late (sorry - sick kid).
>
> https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_
> mMrRQyynQ-Q6MFbI/edit?usp=sharing
>
> I was planning on working on this, but I'm going to take a day or two to
> let others comment.
>
> Darin
>
> On Mon, Sep 19, 2016 at 5:07 PM, Suneel Marthi 
> wrote:
>
> > A shared Google doc would be more convenient than a bunch of Jiras. Its
> > easier to comment and add notes that way.
> >
> >
> > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson  >
> > wrote:
> >
> > > Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > > Based off my pirk-63 I was able to pull spark and storm out with no
> > > issues.  I was planning to pull them out, then tackling elastic search,
> > > then hadoop as it's a little entrenched.  This should keep most PRs to
> > > manageable chunks. I think once that's done addressing the configs will
> > > make more sense.
> > >
> > > I'm open to suggestions. But the hope would be:
> > > Pirk-parent
> > > Pirk-core
> > > Pirk-hadoop
> > > Pirk-storm
> > > Pirk-parent
> > >
> > > Pirk-es is a little weird as it's really just an inputformat, seems
> like
> > > there's a more general solution here than creating submodules for every
> > > inputformat.
> > >
> > > Darin
> > >
> > > On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> > >
> > > >
> > >
> > > > Refactor is definitely a first priority.  Is there a design/proposal
> > > draft
> > > > that we could comment on about how to go about refactoring the
> code.  I
> > > > have been trying to keep up with the emails but definitely would have
> > > > missed some.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > > eawilli...@apache.org > wrote:
> > > >
> > > > > Agree - let's leave the config/CLI the way it is for now and tackle
> > > that as
> > > > > a subsequent design discussion and PR.
> > > > >
> > > > > Also, I think that we should leave the ResponderDriver and the
> > > > > ResponderProps alone for this PR and push to a subsequent PR (once
> we
> > > > > decide if and how we would like to delegate each).
> > > > >
> > > > > I vote to remove the 'platform' option and the backwards
> > compatibility
> > > in
> > > > > this PR and proceed with having a ResponderLauncher interface and
> > > forcing
> > > > > its implementation by the ResponderDriver.
> > > > >
> > > > > And, I am not so concerned with having one fat jar vs. multiple
> jars
> > > right
> > > > > now - to me, at this point, it's a 'nice to have' and not a 'must
> > have'
> > > for
> > > > > Pirk functionality. We do need to break out Pirk into more clearly
> > > defined
> > > > > submodules (which is in progress) - via this re-factor, I think
> that
> > we
> > > > > will gain some ability to generate multiple jars which is nice.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> t.p.elli...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > > Hey guys,
> > > > > > >
> > > > > > > Thanks for looking at the PR, I apologize if it offended
> anyone's
> > > > > eyes:).
> > > > > > >
> > > > > > > I'm glad it generated some discussion about the
> configuration.  I
> > > > > didn't
> > > > > > > really like where things were heading with the config.
> However,
> > > didn't
> > > > > > > want to create to much scope creep.
> > > > > > >
> > > > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > > things
> > > > > much
> > > > > > > more maintainable, the plugin could simply grab the appropriate
> > > part of
> > > > > > the
> > > > > > > config and handle accordingly.  I'd also cut down the number of
> > > command
> > > > > > > line options to only those that change between runs often (like
> > > > > > > input/output)
> > > > > > >
> > > > > > >> One option is to make Pirk pluggable, so that a Pirk
> > installation
> > > > > could
> > > > > > >> use one or more of these in an extensible fashion by 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Suneel, a google doc as promised, only a day late (sorry - sick kid).

https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_mMrRQyynQ-Q6MFbI/edit?usp=sharing

I was planning on working on this, but I'm going to take a day or two to
let others comment.

Darin

On Mon, Sep 19, 2016 at 5:07 PM, Suneel Marthi 
wrote:

> A shared Google doc would be more convenient than a bunch of Jiras. Its
> easier to comment and add notes that way.
>
>
> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
> wrote:
>
> > Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > Based off my pirk-63 I was able to pull spark and storm out with no
> > issues.  I was planning to pull them out, then tackling elastic search,
> > then hadoop as it's a little entrenched.  This should keep most PRs to
> > manageable chunks. I think once that's done addressing the configs will
> > make more sense.
> >
> > I'm open to suggestions. But the hope would be:
> > Pirk-parent
> > Pirk-core
> > Pirk-hadoop
> > Pirk-storm
> > Pirk-parent
> >
> > Pirk-es is a little weird as it's really just an inputformat, seems like
> > there's a more general solution here than creating submodules for every
> > inputformat.
> >
> > Darin
> >
> > On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> >
> > >
> >
> > > Refactor is definitely a first priority.  Is there a design/proposal
> > draft
> > > that we could comment on about how to go about refactoring the code.  I
> > > have been trying to keep up with the emails but definitely would have
> > > missed some.
> > >
> > >
> > >
> > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > eawilli...@apache.org > wrote:
> > >
> > > > Agree - let's leave the config/CLI the way it is for now and tackle
> > that as
> > > > a subsequent design discussion and PR.
> > > >
> > > > Also, I think that we should leave the ResponderDriver and the
> > > > ResponderProps alone for this PR and push to a subsequent PR (once we
> > > > decide if and how we would like to delegate each).
> > > >
> > > > I vote to remove the 'platform' option and the backwards
> compatibility
> > in
> > > > this PR and proceed with having a ResponderLauncher interface and
> > forcing
> > > > its implementation by the ResponderDriver.
> > > >
> > > > And, I am not so concerned with having one fat jar vs. multiple jars
> > right
> > > > now - to me, at this point, it's a 'nice to have' and not a 'must
> have'
> > for
> > > > Pirk functionality. We do need to break out Pirk into more clearly
> > defined
> > > > submodules (which is in progress) - via this re-factor, I think that
> we
> > > > will gain some ability to generate multiple jars which is nice.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison  >
> > > > wrote:
> > > >
> > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > Hey guys,
> > > > > >
> > > > > > Thanks for looking at the PR, I apologize if it offended anyone's
> > > > eyes:).
> > > > > >
> > > > > > I'm glad it generated some discussion about the configuration.  I
> > > > didn't
> > > > > > really like where things were heading with the config.  However,
> > didn't
> > > > > > want to create to much scope creep.
> > > > > >
> > > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > things
> > > > much
> > > > > > more maintainable, the plugin could simply grab the appropriate
> > part of
> > > > > the
> > > > > > config and handle accordingly.  I'd also cut down the number of
> > command
> > > > > > line options to only those that change between runs often (like
> > > > > > input/output)
> > > > > >
> > > > > >> One option is to make Pirk pluggable, so that a Pirk
> installation
> > > > could
> > > > > >> use one or more of these in an extensible fashion by adding JAR
> > files.
> > > > > >> That would still require selecting one by command-line argument.
> > > > > >
> > > > > > An argument for this approach is for lambda architecture
> approaches
> > > > (say
> > > > > > spark/spark-streaming) were the contents of the jars would be so
> > > > similar
> > > > > it
> > > > > > seems like to much trouble to create separate jars.
> > > > > >
> > > > > > Happy to continue working on this given some direction on where
> > you'd
> > > > > like
> > > > > > it to go.  Also, it's a bit of a blocker to refactoring the build
> > into
> > > > > > submodules.
> > > > >
> > > > > FWIW my 2c is to not try and fix all the problems in one go, and
> > rather
> > > > > take a compromise on the configurations while you tease apart the
> > > > > submodules in to separate source code trees, poms, etc; then come
> > back
> > > > > and fix the runtime configs.
> > > > >
> > > > > Once the submodules are in place it will open up more work for
> > release
> > > > > engineering and tinkering that can be done in parallel with the
> > config
> > > > > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Tim Ellison]

On 19/09/16 22:10, Suneel Marthi wrote:
> Here's an example from the Flink project for how they go about new features
> or system breaking API changes, we could start a similar process. The Flink
> guys call these FLIP (Flink Improvement Proposal) and Kafka community
> similarly has something called KLIP.
> 
> We could start a PLIP (??? :-) )

I expect an change 'process' will evolve as required, but +1 for
starting to document some of the architecture on the Pirk wiki.

My notebook has a few scrappy line drawings at the moment.

Regards,
Tim


> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
> 
> 
> On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi 
> wrote:
> 
>> A shared Google doc would be more convenient than a bunch of Jiras. Its
>> easier to comment and add notes that way.
>>
>>
>> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
>> wrote:
>>
>>> Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
>>> Based off my pirk-63 I was able to pull spark and storm out with no
>>> issues.  I was planning to pull them out, then tackling elastic search,
>>> then hadoop as it's a little entrenched.  This should keep most PRs to
>>> manageable chunks. I think once that's done addressing the configs will
>>> make more sense.
>>>
>>> I'm open to suggestions. But the hope would be:
>>> Pirk-parent
>>> Pirk-core
>>> Pirk-hadoop
>>> Pirk-storm
>>> Pirk-parent
>>>
>>> Pirk-es is a little weird as it's really just an inputformat, seems like
>>> there's a more general solution here than creating submodules for every
>>> inputformat.
>>>
>>> Darin
>>>
>>> On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
>>>

>>>
 Refactor is definitely a first priority.  Is there a design/proposal
>>> draft
 that we could comment on about how to go about refactoring the code.  I
 have been trying to keep up with the emails but definitely would have
 missed some.



 On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
 eawilli...@apache.org > wrote:

> Agree - let's leave the config/CLI the way it is for now and tackle
>>> that as
> a subsequent design discussion and PR.
>
> Also, I think that we should leave the ResponderDriver and the
> ResponderProps alone for this PR and push to a subsequent PR (once we
> decide if and how we would like to delegate each).
>
> I vote to remove the 'platform' option and the backwards compatibility
>>> in
> this PR and proceed with having a ResponderLauncher interface and
>>> forcing
> its implementation by the ResponderDriver.
>
> And, I am not so concerned with having one fat jar vs. multiple jars
>>> right
> now - to me, at this point, it's a 'nice to have' and not a 'must
>>> have'
>>> for
> Pirk functionality. We do need to break out Pirk into more clearly
>>> defined
> submodules (which is in progress) - via this re-factor, I think that
>>> we
> will gain some ability to generate multiple jars which is nice.
>
>
>
> On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison 
> wrote:
>
>> On 19/09/16 15:46, Darin Johnson wrote:
>>> Hey guys,
>>>
>>> Thanks for looking at the PR, I apologize if it offended anyone's
> eyes:).
>>>
>>> I'm glad it generated some discussion about the configuration.  I
> didn't
>>> really like where things were heading with the config.  However,
>>> didn't
>>> want to create to much scope creep.
>>>
>>> I think any hierarchical config (TypeSafe or yaml) would make
>>> things
> much
>>> more maintainable, the plugin could simply grab the appropriate
>>> part of
>> the
>>> config and handle accordingly.  I'd also cut down the number of
>>> command
>>> line options to only those that change between runs often (like
>>> input/output)
>>>
 One option is to make Pirk pluggable, so that a Pirk installation
> could
 use one or more of these in an extensible fashion by adding JAR
>>> files.
 That would still require selecting one by command-line argument.
>>>
>>> An argument for this approach is for lambda architecture
>>> approaches
> (say
>>> spark/spark-streaming) were the contents of the jars would be so
> similar
>> it
>>> seems like to much trouble to create separate jars.
>>>
>>> Happy to continue working on this given some direction on where
>>> you'd
>> like
>>> it to go.  Also, it's a bit of a blocker to refactoring the build
>>> into
>>> submodules.
>>
>> FWIW my 2c is to not try and fix all the problems in one go, and
>>> rather
>> take a compromise on the configurations while you tease apart the
>> submodules in to separate source code trees, poms, etc; then come
>>> back
>> and fix the runtime configs.
>>
>> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Tim Ellison]

On 19/09/16 21:22, Darin Johnson wrote:
> Alright, that was in the spirit of what I was thinking when I did this.
> 
> Why don't we take Tim's suggested improvements to my PR (I'll do the
> necessary cleanup) and at the same time just remove the platform argument
> altogether since backwards compatibility isn't upsetting anyone.
> 
> We'll still need a command line option for the launcher for now as we don't
> have submodules we can decide between the two choices after we break out
> submodules and improve the config.

Sounds great - let me know how I can help.

Regards,
Tim


> On Sep 19, 2016 12:19 PM, "Tim Ellison"  wrote:
> 
>> On 19/09/16 15:46, Darin Johnson wrote:
>>> Hey guys,
>>>
>>> Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
>>>
>>> I'm glad it generated some discussion about the configuration.  I didn't
>>> really like where things were heading with the config.  However, didn't
>>> want to create to much scope creep.
>>>
>>> I think any hierarchical config (TypeSafe or yaml) would make things much
>>> more maintainable, the plugin could simply grab the appropriate part of
>> the
>>> config and handle accordingly.  I'd also cut down the number of command
>>> line options to only those that change between runs often (like
>>> input/output)
>>>
 One option is to make Pirk pluggable, so that a Pirk installation could
 use one or more of these in an extensible fashion by adding JAR files.
 That would still require selecting one by command-line argument.
>>>
>>> An argument for this approach is for lambda architecture approaches (say
>>> spark/spark-streaming) were the contents of the jars would be so similar
>> it
>>> seems like to much trouble to create separate jars.
>>>
>>> Happy to continue working on this given some direction on where you'd
>> like
>>> it to go.  Also, it's a bit of a blocker to refactoring the build into
>>> submodules.
>>
>> FWIW my 2c is to not try and fix all the problems in one go, and rather
>> take a compromise on the configurations while you tease apart the
>> submodules in to separate source code trees, poms, etc; then come back
>> and fix the runtime configs.
>>
>> Once the submodules are in place it will open up more work for release
>> engineering and tinkering that can be done in parallel with the config
>> polishing.
>>
>> Just a thought.
>> Tim
>>
>>
>>> On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
>> wrote:
>>>
 On 19/09/16 13:40, Ellison Anne Williams wrote:
> It seems that it's the same idea as the ResponderLauncher with the
 service
> component added to maintain something akin to the 'platform'. I would
> prefer that we just did away with the platform notion altogether and
>> make
> the ResponderDriver 'dumb'. We get around needing a platform-aware
 service
> by requiring the ResponderLauncher implementation to be passed as a CLI
 to
> the ResponderDriver.

 Let me check I understand what you are saying here.

 At the moment, there is a monolithic Pirk that hard codes how to respond
 using lots of different backends (mapreduce, spark, sparkstreaming,
 storm , standalone), and that is selected by command-line argument.

 One option is to make Pirk pluggable, so that a Pirk installation could
 use one or more of these in an extensible fashion by adding JAR files.
 That would still require selecting one by command-line argument.

 A second option is to simply pass in the required backend JAR to select
 the particular implementation you choose, as a specific Pirk
 installation doesn't need to use multiple backends simultaneously.

 ...and you are leaning towards the second option.  Do I have that
>> correct?

 Regards,
 Tim

> Am I missing something? Is there a good reason to provide a service by
> which platforms are registered? I'm open...
>
> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
 wrote:
>
>> How about an approach like this?
>>https://github.com/tellison/incubator-pirk/tree/pirk-63
>>
>> The "on-ramp" is the driver [1], which calls upon the service to find
>> a
>> plug-in [2] that claims to implement the required platform responder,
>> e.g. [3].
>>
>> The list of plug-ins is given in the provider's JAR file, so the ones
>> we
>> provide in Pirk are listed together [4], but if you split these into
>> modules, or somebody brings their own JAR alongside, these would be
>> listed in each JAR's services/ directory.
>>
>> [1]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/wideskies/
>> ResponderDriver.java
>> [2]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
>> [3]
>> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Great will write up the doc link here, finish pirk63 then start this.

On Sep 19, 2016 5:34 PM, "Suneel Marthi"  wrote:

> +100
>
> On Mon, Sep 19, 2016 at 11:24 PM, Ellison Anne Williams <
> eawilli...@apache.org> wrote:
>
> > Yes, ES is just an inputformat (like HDFS, Kafka, etc) - we don't need a
> > separate submodule.
> >
> > Aside from pirk-core, it seems that we would want to break the responder
> > implementations out into submodules. This would leave us with something
> > along the lines of the following (at this point):
> >
> > pirk-core (encryption, core responder incl. standalone, core querier,
> > query, inputformat, serialization, utils)
> > pirk-storm
> > pirk-mapreduce
> > pirk-spark
> > pirk-benchmark
> > pirk-distributed-test
> >
> > Once we add other responder implementations, we can add them as
> submodules
> > - i.e. for Flink, we would have pirk-flink; for Beam, pirk-beam, etc.
> >
> > We could break 'pirk-core' down further...
> >
> > On Mon, Sep 19, 2016 at 5:10 PM, Suneel Marthi 
> > wrote:
> >
> > > Here's an example from the Flink project for how they go about new
> > features
> > > or system breaking API changes, we could start a similar process. The
> > Flink
> > > guys call these FLIP (Flink Improvement Proposal) and Kafka community
> > > similarly has something called KLIP.
> > >
> > > We could start a PLIP (??? :-) )
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=65870673
> > >
> > >
> > > On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi <
> suneel.mar...@gmail.com
> > >
> > > wrote:
> > >
> > > > A shared Google doc would be more convenient than a bunch of Jiras.
> Its
> > > > easier to comment and add notes that way.
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson <
> > dbjohnson1...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Suneel, I'll try to put a couple jiras on it tonight with my
> thoughts.
> > > >> Based off my pirk-63 I was able to pull spark and storm out with no
> > > >> issues.  I was planning to pull them out, then tackling elastic
> > search,
> > > >> then hadoop as it's a little entrenched.  This should keep most PRs
> to
> > > >> manageable chunks. I think once that's done addressing the configs
> > will
> > > >> make more sense.
> > > >>
> > > >> I'm open to suggestions. But the hope would be:
> > > >> Pirk-parent
> > > >> Pirk-core
> > > >> Pirk-hadoop
> > > >> Pirk-storm
> > > >> Pirk-parent
> > > >>
> > > >> Pirk-es is a little weird as it's really just an inputformat, seems
> > like
> > > >> there's a more general solution here than creating submodules for
> > every
> > > >> inputformat.
> > > >>
> > > >> Darin
> > > >>
> > > >> On Sep 19, 2016 1:00 PM, "Suneel Marthi" 
> wrote:
> > > >>
> > > >> >
> > > >>
> > > >> > Refactor is definitely a first priority.  Is there a
> design/proposal
> > > >> draft
> > > >> > that we could comment on about how to go about refactoring the
> code.
> > > I
> > > >> > have been trying to keep up with the emails but definitely would
> > have
> > > >> > missed some.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > >> > eawilli...@apache.org > wrote:
> > > >> >
> > > >> > > Agree - let's leave the config/CLI the way it is for now and
> > tackle
> > > >> that as
> > > >> > > a subsequent design discussion and PR.
> > > >> > >
> > > >> > > Also, I think that we should leave the ResponderDriver and the
> > > >> > > ResponderProps alone for this PR and push to a subsequent PR
> (once
> > > we
> > > >> > > decide if and how we would like to delegate each).
> > > >> > >
> > > >> > > I vote to remove the 'platform' option and the backwards
> > > compatibility
> > > >> in
> > > >> > > this PR and proceed with having a ResponderLauncher interface
> and
> > > >> forcing
> > > >> > > its implementation by the ResponderDriver.
> > > >> > >
> > > >> > > And, I am not so concerned with having one fat jar vs. multiple
> > jars
> > > >> right
> > > >> > > now - to me, at this point, it's a 'nice to have' and not a
> 'must
> > > >> have'
> > > >> for
> > > >> > > Pirk functionality. We do need to break out Pirk into more
> clearly
> > > >> defined
> > > >> > > submodules (which is in progress) - via this re-factor, I think
> > that
> > > >> we
> > > >> > > will gain some ability to generate multiple jars which is nice.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> > > t.p.elli...@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > >> > > > > Hey guys,
> > > >> > > > >
> > > >> > > > > Thanks for looking at the PR, I apologize if it offended
> > > anyone's
> > > >> > > eyes:).
> > > >> > > > >
> > > >> > > > > I'm glad it generated some discussion about the
> configuration.
> > > I
> > > >> > > didn't
> > > >> > > > > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Suneel Marthi]

+100

On Mon, Sep 19, 2016 at 11:24 PM, Ellison Anne Williams <
eawilli...@apache.org> wrote:

> Yes, ES is just an inputformat (like HDFS, Kafka, etc) - we don't need a
> separate submodule.
>
> Aside from pirk-core, it seems that we would want to break the responder
> implementations out into submodules. This would leave us with something
> along the lines of the following (at this point):
>
> pirk-core (encryption, core responder incl. standalone, core querier,
> query, inputformat, serialization, utils)
> pirk-storm
> pirk-mapreduce
> pirk-spark
> pirk-benchmark
> pirk-distributed-test
>
> Once we add other responder implementations, we can add them as submodules
> - i.e. for Flink, we would have pirk-flink; for Beam, pirk-beam, etc.
>
> We could break 'pirk-core' down further...
>
> On Mon, Sep 19, 2016 at 5:10 PM, Suneel Marthi 
> wrote:
>
> > Here's an example from the Flink project for how they go about new
> features
> > or system breaking API changes, we could start a similar process. The
> Flink
> > guys call these FLIP (Flink Improvement Proposal) and Kafka community
> > similarly has something called KLIP.
> >
> > We could start a PLIP (??? :-) )
> >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=65870673
> >
> >
> > On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi  >
> > wrote:
> >
> > > A shared Google doc would be more convenient than a bunch of Jiras. Its
> > > easier to comment and add notes that way.
> > >
> > >
> > > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson <
> dbjohnson1...@gmail.com
> > >
> > > wrote:
> > >
> > >> Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > >> Based off my pirk-63 I was able to pull spark and storm out with no
> > >> issues.  I was planning to pull them out, then tackling elastic
> search,
> > >> then hadoop as it's a little entrenched.  This should keep most PRs to
> > >> manageable chunks. I think once that's done addressing the configs
> will
> > >> make more sense.
> > >>
> > >> I'm open to suggestions. But the hope would be:
> > >> Pirk-parent
> > >> Pirk-core
> > >> Pirk-hadoop
> > >> Pirk-storm
> > >> Pirk-parent
> > >>
> > >> Pirk-es is a little weird as it's really just an inputformat, seems
> like
> > >> there's a more general solution here than creating submodules for
> every
> > >> inputformat.
> > >>
> > >> Darin
> > >>
> > >> On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> > >>
> > >> >
> > >>
> > >> > Refactor is definitely a first priority.  Is there a design/proposal
> > >> draft
> > >> > that we could comment on about how to go about refactoring the code.
> > I
> > >> > have been trying to keep up with the emails but definitely would
> have
> > >> > missed some.
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > >> > eawilli...@apache.org > wrote:
> > >> >
> > >> > > Agree - let's leave the config/CLI the way it is for now and
> tackle
> > >> that as
> > >> > > a subsequent design discussion and PR.
> > >> > >
> > >> > > Also, I think that we should leave the ResponderDriver and the
> > >> > > ResponderProps alone for this PR and push to a subsequent PR (once
> > we
> > >> > > decide if and how we would like to delegate each).
> > >> > >
> > >> > > I vote to remove the 'platform' option and the backwards
> > compatibility
> > >> in
> > >> > > this PR and proceed with having a ResponderLauncher interface and
> > >> forcing
> > >> > > its implementation by the ResponderDriver.
> > >> > >
> > >> > > And, I am not so concerned with having one fat jar vs. multiple
> jars
> > >> right
> > >> > > now - to me, at this point, it's a 'nice to have' and not a 'must
> > >> have'
> > >> for
> > >> > > Pirk functionality. We do need to break out Pirk into more clearly
> > >> defined
> > >> > > submodules (which is in progress) - via this re-factor, I think
> that
> > >> we
> > >> > > will gain some ability to generate multiple jars which is nice.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> > t.p.elli...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > On 19/09/16 15:46, Darin Johnson wrote:
> > >> > > > > Hey guys,
> > >> > > > >
> > >> > > > > Thanks for looking at the PR, I apologize if it offended
> > anyone's
> > >> > > eyes:).
> > >> > > > >
> > >> > > > > I'm glad it generated some discussion about the configuration.
> > I
> > >> > > didn't
> > >> > > > > really like where things were heading with the config.
> However,
> > >> didn't
> > >> > > > > want to create to much scope creep.
> > >> > > > >
> > >> > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > >> things
> > >> > > much
> > >> > > > > more maintainable, the plugin could simply grab the
> appropriate
> > >> part of
> > >> > > > the
> > >> > > > > config and handle accordingly.  I'd also cut down the number
> of
> > >> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Ellison Anne Williams]

Yes, ES is just an inputformat (like HDFS, Kafka, etc) - we don't need a
separate submodule.

Aside from pirk-core, it seems that we would want to break the responder
implementations out into submodules. This would leave us with something
along the lines of the following (at this point):

pirk-core (encryption, core responder incl. standalone, core querier,
query, inputformat, serialization, utils)
pirk-storm
pirk-mapreduce
pirk-spark
pirk-benchmark
pirk-distributed-test

Once we add other responder implementations, we can add them as submodules
- i.e. for Flink, we would have pirk-flink; for Beam, pirk-beam, etc.

We could break 'pirk-core' down further...

On Mon, Sep 19, 2016 at 5:10 PM, Suneel Marthi 
wrote:

> Here's an example from the Flink project for how they go about new features
> or system breaking API changes, we could start a similar process. The Flink
> guys call these FLIP (Flink Improvement Proposal) and Kafka community
> similarly has something called KLIP.
>
> We could start a PLIP (??? :-) )
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673
>
>
> On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi 
> wrote:
>
> > A shared Google doc would be more convenient than a bunch of Jiras. Its
> > easier to comment and add notes that way.
> >
> >
> > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson  >
> > wrote:
> >
> >> Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> >> Based off my pirk-63 I was able to pull spark and storm out with no
> >> issues.  I was planning to pull them out, then tackling elastic search,
> >> then hadoop as it's a little entrenched.  This should keep most PRs to
> >> manageable chunks. I think once that's done addressing the configs will
> >> make more sense.
> >>
> >> I'm open to suggestions. But the hope would be:
> >> Pirk-parent
> >> Pirk-core
> >> Pirk-hadoop
> >> Pirk-storm
> >> Pirk-parent
> >>
> >> Pirk-es is a little weird as it's really just an inputformat, seems like
> >> there's a more general solution here than creating submodules for every
> >> inputformat.
> >>
> >> Darin
> >>
> >> On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> >>
> >> >
> >>
> >> > Refactor is definitely a first priority.  Is there a design/proposal
> >> draft
> >> > that we could comment on about how to go about refactoring the code.
> I
> >> > have been trying to keep up with the emails but definitely would have
> >> > missed some.
> >> >
> >> >
> >> >
> >> > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> >> > eawilli...@apache.org > wrote:
> >> >
> >> > > Agree - let's leave the config/CLI the way it is for now and tackle
> >> that as
> >> > > a subsequent design discussion and PR.
> >> > >
> >> > > Also, I think that we should leave the ResponderDriver and the
> >> > > ResponderProps alone for this PR and push to a subsequent PR (once
> we
> >> > > decide if and how we would like to delegate each).
> >> > >
> >> > > I vote to remove the 'platform' option and the backwards
> compatibility
> >> in
> >> > > this PR and proceed with having a ResponderLauncher interface and
> >> forcing
> >> > > its implementation by the ResponderDriver.
> >> > >
> >> > > And, I am not so concerned with having one fat jar vs. multiple jars
> >> right
> >> > > now - to me, at this point, it's a 'nice to have' and not a 'must
> >> have'
> >> for
> >> > > Pirk functionality. We do need to break out Pirk into more clearly
> >> defined
> >> > > submodules (which is in progress) - via this re-factor, I think that
> >> we
> >> > > will gain some ability to generate multiple jars which is nice.
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> t.p.elli...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > On 19/09/16 15:46, Darin Johnson wrote:
> >> > > > > Hey guys,
> >> > > > >
> >> > > > > Thanks for looking at the PR, I apologize if it offended
> anyone's
> >> > > eyes:).
> >> > > > >
> >> > > > > I'm glad it generated some discussion about the configuration.
> I
> >> > > didn't
> >> > > > > really like where things were heading with the config.  However,
> >> didn't
> >> > > > > want to create to much scope creep.
> >> > > > >
> >> > > > > I think any hierarchical config (TypeSafe or yaml) would make
> >> things
> >> > > much
> >> > > > > more maintainable, the plugin could simply grab the appropriate
> >> part of
> >> > > > the
> >> > > > > config and handle accordingly.  I'd also cut down the number of
> >> command
> >> > > > > line options to only those that change between runs often (like
> >> > > > > input/output)
> >> > > > >
> >> > > > >> One option is to make Pirk pluggable, so that a Pirk
> installation
> >> > > could
> >> > > > >> use one or more of these in an extensible fashion by adding JAR
> >> files.
> >> > > > >> That would still require selecting one by command-line
> argument.
> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Suneel Marthi]

Here's an example from the Flink project for how they go about new features
or system breaking API changes, we could start a similar process. The Flink
guys call these FLIP (Flink Improvement Proposal) and Kafka community
similarly has something called KLIP.

We could start a PLIP (??? :-) )

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673


On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi 
wrote:

> A shared Google doc would be more convenient than a bunch of Jiras. Its
> easier to comment and add notes that way.
>
>
> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
> wrote:
>
>> Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
>> Based off my pirk-63 I was able to pull spark and storm out with no
>> issues.  I was planning to pull them out, then tackling elastic search,
>> then hadoop as it's a little entrenched.  This should keep most PRs to
>> manageable chunks. I think once that's done addressing the configs will
>> make more sense.
>>
>> I'm open to suggestions. But the hope would be:
>> Pirk-parent
>> Pirk-core
>> Pirk-hadoop
>> Pirk-storm
>> Pirk-parent
>>
>> Pirk-es is a little weird as it's really just an inputformat, seems like
>> there's a more general solution here than creating submodules for every
>> inputformat.
>>
>> Darin
>>
>> On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
>>
>> >
>>
>> > Refactor is definitely a first priority.  Is there a design/proposal
>> draft
>> > that we could comment on about how to go about refactoring the code.  I
>> > have been trying to keep up with the emails but definitely would have
>> > missed some.
>> >
>> >
>> >
>> > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
>> > eawilli...@apache.org > wrote:
>> >
>> > > Agree - let's leave the config/CLI the way it is for now and tackle
>> that as
>> > > a subsequent design discussion and PR.
>> > >
>> > > Also, I think that we should leave the ResponderDriver and the
>> > > ResponderProps alone for this PR and push to a subsequent PR (once we
>> > > decide if and how we would like to delegate each).
>> > >
>> > > I vote to remove the 'platform' option and the backwards compatibility
>> in
>> > > this PR and proceed with having a ResponderLauncher interface and
>> forcing
>> > > its implementation by the ResponderDriver.
>> > >
>> > > And, I am not so concerned with having one fat jar vs. multiple jars
>> right
>> > > now - to me, at this point, it's a 'nice to have' and not a 'must
>> have'
>> for
>> > > Pirk functionality. We do need to break out Pirk into more clearly
>> defined
>> > > submodules (which is in progress) - via this re-factor, I think that
>> we
>> > > will gain some ability to generate multiple jars which is nice.
>> > >
>> > >
>> > >
>> > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison 
>> > > wrote:
>> > >
>> > > > On 19/09/16 15:46, Darin Johnson wrote:
>> > > > > Hey guys,
>> > > > >
>> > > > > Thanks for looking at the PR, I apologize if it offended anyone's
>> > > eyes:).
>> > > > >
>> > > > > I'm glad it generated some discussion about the configuration.  I
>> > > didn't
>> > > > > really like where things were heading with the config.  However,
>> didn't
>> > > > > want to create to much scope creep.
>> > > > >
>> > > > > I think any hierarchical config (TypeSafe or yaml) would make
>> things
>> > > much
>> > > > > more maintainable, the plugin could simply grab the appropriate
>> part of
>> > > > the
>> > > > > config and handle accordingly.  I'd also cut down the number of
>> command
>> > > > > line options to only those that change between runs often (like
>> > > > > input/output)
>> > > > >
>> > > > >> One option is to make Pirk pluggable, so that a Pirk installation
>> > > could
>> > > > >> use one or more of these in an extensible fashion by adding JAR
>> files.
>> > > > >> That would still require selecting one by command-line argument.
>> > > > >
>> > > > > An argument for this approach is for lambda architecture
>> approaches
>> > > (say
>> > > > > spark/spark-streaming) were the contents of the jars would be so
>> > > similar
>> > > > it
>> > > > > seems like to much trouble to create separate jars.
>> > > > >
>> > > > > Happy to continue working on this given some direction on where
>> you'd
>> > > > like
>> > > > > it to go.  Also, it's a bit of a blocker to refactoring the build
>> into
>> > > > > submodules.
>> > > >
>> > > > FWIW my 2c is to not try and fix all the problems in one go, and
>> rather
>> > > > take a compromise on the configurations while you tease apart the
>> > > > submodules in to separate source code trees, poms, etc; then come
>> back
>> > > > and fix the runtime configs.
>> > > >
>> > > > Once the submodules are in place it will open up more work for
>> release
>> > > > engineering and tinkering that can be done in parallel with the
>> config
>> > > > polishing.
>> > > >
>> > > > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Ellison Anne Williams]

Sounds good.

On Mon, Sep 19, 2016 at 4:22 PM, Darin Johnson 
wrote:

> Alright, that was in the spirit of what I was thinking when I did this.
>
> Why don't we take Tim's suggested improvements to my PR (I'll do the
> necessary cleanup) and at the same time just remove the platform argument
> altogether since backwards compatibility isn't upsetting anyone.
>
> We'll still need a command line option for the launcher for now as we don't
> have submodules we can decide between the two choices after we break out
> submodules and improve the config.
>
>
> On Sep 19, 2016 12:19 PM, "Tim Ellison"  wrote:
>
> > On 19/09/16 15:46, Darin Johnson wrote:
> > > Hey guys,
> > >
> > > Thanks for looking at the PR, I apologize if it offended anyone's
> eyes:).
> > >
> > > I'm glad it generated some discussion about the configuration.  I
> didn't
> > > really like where things were heading with the config.  However, didn't
> > > want to create to much scope creep.
> > >
> > > I think any hierarchical config (TypeSafe or yaml) would make things
> much
> > > more maintainable, the plugin could simply grab the appropriate part of
> > the
> > > config and handle accordingly.  I'd also cut down the number of command
> > > line options to only those that change between runs often (like
> > > input/output)
> > >
> > >> One option is to make Pirk pluggable, so that a Pirk installation
> could
> > >> use one or more of these in an extensible fashion by adding JAR files.
> > >> That would still require selecting one by command-line argument.
> > >
> > > An argument for this approach is for lambda architecture approaches
> (say
> > > spark/spark-streaming) were the contents of the jars would be so
> similar
> > it
> > > seems like to much trouble to create separate jars.
> > >
> > > Happy to continue working on this given some direction on where you'd
> > like
> > > it to go.  Also, it's a bit of a blocker to refactoring the build into
> > > submodules.
> >
> > FWIW my 2c is to not try and fix all the problems in one go, and rather
> > take a compromise on the configurations while you tease apart the
> > submodules in to separate source code trees, poms, etc; then come back
> > and fix the runtime configs.
> >
> > Once the submodules are in place it will open up more work for release
> > engineering and tinkering that can be done in parallel with the config
> > polishing.
> >
> > Just a thought.
> > Tim
> >
> >
> > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
> > wrote:
> > >
> > >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > >>> It seems that it's the same idea as the ResponderLauncher with the
> > >> service
> > >>> component added to maintain something akin to the 'platform'. I would
> > >>> prefer that we just did away with the platform notion altogether and
> > make
> > >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> > >> service
> > >>> by requiring the ResponderLauncher implementation to be passed as a
> CLI
> > >> to
> > >>> the ResponderDriver.
> > >>
> > >> Let me check I understand what you are saying here.
> > >>
> > >> At the moment, there is a monolithic Pirk that hard codes how to
> respond
> > >> using lots of different backends (mapreduce, spark, sparkstreaming,
> > >> storm , standalone), and that is selected by command-line argument.
> > >>
> > >> One option is to make Pirk pluggable, so that a Pirk installation
> could
> > >> use one or more of these in an extensible fashion by adding JAR files.
> > >> That would still require selecting one by command-line argument.
> > >>
> > >> A second option is to simply pass in the required backend JAR to
> select
> > >> the particular implementation you choose, as a specific Pirk
> > >> installation doesn't need to use multiple backends simultaneously.
> > >>
> > >> ...and you are leaning towards the second option.  Do I have that
> > correct?
> > >>
> > >> Regards,
> > >> Tim
> > >>
> > >>> Am I missing something? Is there a good reason to provide a service
> by
> > >>> which platforms are registered? I'm open...
> > >>>
> > >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
> > >> wrote:
> > >>>
> >  How about an approach like this?
> > https://github.com/tellison/incubator-pirk/tree/pirk-63
> > 
> >  The "on-ramp" is the driver [1], which calls upon the service to
> find
> > a
> >  plug-in [2] that claims to implement the required platform
> responder,
> >  e.g. [3].
> > 
> >  The list of plug-ins is given in the provider's JAR file, so the
> ones
> > we
> >  provide in Pirk are listed together [4], but if you split these into
> >  modules, or somebody brings their own JAR alongside, these would be
> >  listed in each JAR's services/ directory.
> > 
> >  [1]
> >  https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >  src/main/java/org/apache/pirk/responder/wideskies/
> > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Sure will do tonight.

On Sep 19, 2016 5:07 PM, "Suneel Marthi"  wrote:

> A shared Google doc would be more convenient than a bunch of Jiras. Its
> easier to comment and add notes that way.
>
>
> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson 
> wrote:
>
> > Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
> > Based off my pirk-63 I was able to pull spark and storm out with no
> > issues.  I was planning to pull them out, then tackling elastic search,
> > then hadoop as it's a little entrenched.  This should keep most PRs to
> > manageable chunks. I think once that's done addressing the configs will
> > make more sense.
> >
> > I'm open to suggestions. But the hope would be:
> > Pirk-parent
> > Pirk-core
> > Pirk-hadoop
> > Pirk-storm
> > Pirk-parent
> >
> > Pirk-es is a little weird as it's really just an inputformat, seems like
> > there's a more general solution here than creating submodules for every
> > inputformat.
> >
> > Darin
> >
> > On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:
> >
> > >
> >
> > > Refactor is definitely a first priority.  Is there a design/proposal
> > draft
> > > that we could comment on about how to go about refactoring the code.  I
> > > have been trying to keep up with the emails but definitely would have
> > > missed some.
> > >
> > >
> > >
> > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > eawilli...@apache.org > wrote:
> > >
> > > > Agree - let's leave the config/CLI the way it is for now and tackle
> > that as
> > > > a subsequent design discussion and PR.
> > > >
> > > > Also, I think that we should leave the ResponderDriver and the
> > > > ResponderProps alone for this PR and push to a subsequent PR (once we
> > > > decide if and how we would like to delegate each).
> > > >
> > > > I vote to remove the 'platform' option and the backwards
> compatibility
> > in
> > > > this PR and proceed with having a ResponderLauncher interface and
> > forcing
> > > > its implementation by the ResponderDriver.
> > > >
> > > > And, I am not so concerned with having one fat jar vs. multiple jars
> > right
> > > > now - to me, at this point, it's a 'nice to have' and not a 'must
> have'
> > for
> > > > Pirk functionality. We do need to break out Pirk into more clearly
> > defined
> > > > submodules (which is in progress) - via this re-factor, I think that
> we
> > > > will gain some ability to generate multiple jars which is nice.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison  >
> > > > wrote:
> > > >
> > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > Hey guys,
> > > > > >
> > > > > > Thanks for looking at the PR, I apologize if it offended anyone's
> > > > eyes:).
> > > > > >
> > > > > > I'm glad it generated some discussion about the configuration.  I
> > > > didn't
> > > > > > really like where things were heading with the config.  However,
> > didn't
> > > > > > want to create to much scope creep.
> > > > > >
> > > > > > I think any hierarchical config (TypeSafe or yaml) would make
> > things
> > > > much
> > > > > > more maintainable, the plugin could simply grab the appropriate
> > part of
> > > > > the
> > > > > > config and handle accordingly.  I'd also cut down the number of
> > command
> > > > > > line options to only those that change between runs often (like
> > > > > > input/output)
> > > > > >
> > > > > >> One option is to make Pirk pluggable, so that a Pirk
> installation
> > > > could
> > > > > >> use one or more of these in an extensible fashion by adding JAR
> > files.
> > > > > >> That would still require selecting one by command-line argument.
> > > > > >
> > > > > > An argument for this approach is for lambda architecture
> approaches
> > > > (say
> > > > > > spark/spark-streaming) were the contents of the jars would be so
> > > > similar
> > > > > it
> > > > > > seems like to much trouble to create separate jars.
> > > > > >
> > > > > > Happy to continue working on this given some direction on where
> > you'd
> > > > > like
> > > > > > it to go.  Also, it's a bit of a blocker to refactoring the build
> > into
> > > > > > submodules.
> > > > >
> > > > > FWIW my 2c is to not try and fix all the problems in one go, and
> > rather
> > > > > take a compromise on the configurations while you tease apart the
> > > > > submodules in to separate source code trees, poms, etc; then come
> > back
> > > > > and fix the runtime configs.
> > > > >
> > > > > Once the submodules are in place it will open up more work for
> > release
> > > > > engineering and tinkering that can be done in parallel with the
> > config
> > > > > polishing.
> > > > >
> > > > > Just a thought.
> > > > > Tim
> > > > >
> > > > >
> > > > > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <
> > t.p.elli...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Suneel, I'll try to put a couple jiras on it tonight with my thoughts.
Based off my pirk-63 I was able to pull spark and storm out with no
issues.  I was planning to pull them out, then tackling elastic search,
then hadoop as it's a little entrenched.  This should keep most PRs to
manageable chunks. I think once that's done addressing the configs will
make more sense.

I'm open to suggestions. But the hope would be:
Pirk-parent
Pirk-core
Pirk-hadoop
Pirk-storm
Pirk-parent

Pirk-es is a little weird as it's really just an inputformat, seems like
there's a more general solution here than creating submodules for every
inputformat.

Darin

On Sep 19, 2016 1:00 PM, "Suneel Marthi"  wrote:

>

> Refactor is definitely a first priority.  Is there a design/proposal draft
> that we could comment on about how to go about refactoring the code.  I
> have been trying to keep up with the emails but definitely would have
> missed some.
>
>
>
> On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> eawilli...@apache.org > wrote:
>
> > Agree - let's leave the config/CLI the way it is for now and tackle
that as
> > a subsequent design discussion and PR.
> >
> > Also, I think that we should leave the ResponderDriver and the
> > ResponderProps alone for this PR and push to a subsequent PR (once we
> > decide if and how we would like to delegate each).
> >
> > I vote to remove the 'platform' option and the backwards compatibility
in
> > this PR and proceed with having a ResponderLauncher interface and
forcing
> > its implementation by the ResponderDriver.
> >
> > And, I am not so concerned with having one fat jar vs. multiple jars
right
> > now - to me, at this point, it's a 'nice to have' and not a 'must have'
for
> > Pirk functionality. We do need to break out Pirk into more clearly
defined
> > submodules (which is in progress) - via this re-factor, I think that we
> > will gain some ability to generate multiple jars which is nice.
> >
> >
> >
> > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison 
> > wrote:
> >
> > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > Hey guys,
> > > >
> > > > Thanks for looking at the PR, I apologize if it offended anyone's
> > eyes:).
> > > >
> > > > I'm glad it generated some discussion about the configuration.  I
> > didn't
> > > > really like where things were heading with the config.  However,
didn't
> > > > want to create to much scope creep.
> > > >
> > > > I think any hierarchical config (TypeSafe or yaml) would make things
> > much
> > > > more maintainable, the plugin could simply grab the appropriate
part of
> > > the
> > > > config and handle accordingly.  I'd also cut down the number of
command
> > > > line options to only those that change between runs often (like
> > > > input/output)
> > > >
> > > >> One option is to make Pirk pluggable, so that a Pirk installation
> > could
> > > >> use one or more of these in an extensible fashion by adding JAR
files.
> > > >> That would still require selecting one by command-line argument.
> > > >
> > > > An argument for this approach is for lambda architecture approaches
> > (say
> > > > spark/spark-streaming) were the contents of the jars would be so
> > similar
> > > it
> > > > seems like to much trouble to create separate jars.
> > > >
> > > > Happy to continue working on this given some direction on where
you'd
> > > like
> > > > it to go.  Also, it's a bit of a blocker to refactoring the build
into
> > > > submodules.
> > >
> > > FWIW my 2c is to not try and fix all the problems in one go, and
rather
> > > take a compromise on the configurations while you tease apart the
> > > submodules in to separate source code trees, poms, etc; then come back
> > > and fix the runtime configs.
> > >
> > > Once the submodules are in place it will open up more work for release
> > > engineering and tinkering that can be done in parallel with the config
> > > polishing.
> > >
> > > Just a thought.
> > > Tim
> > >
> > >
> > > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
> > > wrote:
> > > >
> > > >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > > >>> It seems that it's the same idea as the ResponderLauncher with the
> > > >> service
> > > >>> component added to maintain something akin to the 'platform'. I
would
> > > >>> prefer that we just did away with the platform notion altogether
and
> > > make
> > > >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> > > >> service
> > > >>> by requiring the ResponderLauncher implementation to be passed as
a
> > CLI
> > > >> to
> > > >>> the ResponderDriver.
> > > >>
> > > >> Let me check I understand what you are saying here.
> > > >>
> > > >> At the moment, there is a monolithic Pirk that hard codes how to
> > respond
> > > >> using lots of different backends (mapreduce, spark, sparkstreaming,
> > > >> storm , standalone), and that is selected by command-line argument.
> > > >>
> > > >> One 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Alright, that was in the spirit of what I was thinking when I did this.

Why don't we take Tim's suggested improvements to my PR (I'll do the
necessary cleanup) and at the same time just remove the platform argument
altogether since backwards compatibility isn't upsetting anyone.

We'll still need a command line option for the launcher for now as we don't
have submodules we can decide between the two choices after we break out
submodules and improve the config.


On Sep 19, 2016 12:19 PM, "Tim Ellison"  wrote:

> On 19/09/16 15:46, Darin Johnson wrote:
> > Hey guys,
> >
> > Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
> >
> > I'm glad it generated some discussion about the configuration.  I didn't
> > really like where things were heading with the config.  However, didn't
> > want to create to much scope creep.
> >
> > I think any hierarchical config (TypeSafe or yaml) would make things much
> > more maintainable, the plugin could simply grab the appropriate part of
> the
> > config and handle accordingly.  I'd also cut down the number of command
> > line options to only those that change between runs often (like
> > input/output)
> >
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >
> > An argument for this approach is for lambda architecture approaches (say
> > spark/spark-streaming) were the contents of the jars would be so similar
> it
> > seems like to much trouble to create separate jars.
> >
> > Happy to continue working on this given some direction on where you'd
> like
> > it to go.  Also, it's a bit of a blocker to refactoring the build into
> > submodules.
>
> FWIW my 2c is to not try and fix all the problems in one go, and rather
> take a compromise on the configurations while you tease apart the
> submodules in to separate source code trees, poms, etc; then come back
> and fix the runtime configs.
>
> Once the submodules are in place it will open up more work for release
> engineering and tinkering that can be done in parallel with the config
> polishing.
>
> Just a thought.
> Tim
>
>
> > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
> wrote:
> >
> >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> >>> It seems that it's the same idea as the ResponderLauncher with the
> >> service
> >>> component added to maintain something akin to the 'platform'. I would
> >>> prefer that we just did away with the platform notion altogether and
> make
> >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> >> service
> >>> by requiring the ResponderLauncher implementation to be passed as a CLI
> >> to
> >>> the ResponderDriver.
> >>
> >> Let me check I understand what you are saying here.
> >>
> >> At the moment, there is a monolithic Pirk that hard codes how to respond
> >> using lots of different backends (mapreduce, spark, sparkstreaming,
> >> storm , standalone), and that is selected by command-line argument.
> >>
> >> One option is to make Pirk pluggable, so that a Pirk installation could
> >> use one or more of these in an extensible fashion by adding JAR files.
> >> That would still require selecting one by command-line argument.
> >>
> >> A second option is to simply pass in the required backend JAR to select
> >> the particular implementation you choose, as a specific Pirk
> >> installation doesn't need to use multiple backends simultaneously.
> >>
> >> ...and you are leaning towards the second option.  Do I have that
> correct?
> >>
> >> Regards,
> >> Tim
> >>
> >>> Am I missing something? Is there a good reason to provide a service by
> >>> which platforms are registered? I'm open...
> >>>
> >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
> >> wrote:
> >>>
>  How about an approach like this?
> https://github.com/tellison/incubator-pirk/tree/pirk-63
> 
>  The "on-ramp" is the driver [1], which calls upon the service to find
> a
>  plug-in [2] that claims to implement the required platform responder,
>  e.g. [3].
> 
>  The list of plug-ins is given in the provider's JAR file, so the ones
> we
>  provide in Pirk are listed together [4], but if you split these into
>  modules, or somebody brings their own JAR alongside, these would be
>  listed in each JAR's services/ directory.
> 
>  [1]
>  https://github.com/tellison/incubator-pirk/blob/pirk-63/
>  src/main/java/org/apache/pirk/responder/wideskies/
> ResponderDriver.java
>  [2]
>  https://github.com/tellison/incubator-pirk/blob/pirk-63/
>  src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
>  [3]
>  https://github.com/tellison/incubator-pirk/blob/pirk-63/
>  src/main/java/org/apache/pirk/responder/wideskies/storm/
>  StormResponder.java
>  [4]
> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Suneel Marthi]

Refactor is definitely a first priority.  Is there a design/proposal draft
that we could comment on about how to go about refactoring the code.  I
have been trying to keep up with the emails but definitely would have
missed some.



On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
eawilli...@apache.org> wrote:

> Agree - let's leave the config/CLI the way it is for now and tackle that as
> a subsequent design discussion and PR.
>
> Also, I think that we should leave the ResponderDriver and the
> ResponderProps alone for this PR and push to a subsequent PR (once we
> decide if and how we would like to delegate each).
>
> I vote to remove the 'platform' option and the backwards compatibility in
> this PR and proceed with having a ResponderLauncher interface and forcing
> its implementation by the ResponderDriver.
>
> And, I am not so concerned with having one fat jar vs. multiple jars right
> now - to me, at this point, it's a 'nice to have' and not a 'must have' for
> Pirk functionality. We do need to break out Pirk into more clearly defined
> submodules (which is in progress) - via this re-factor, I think that we
> will gain some ability to generate multiple jars which is nice.
>
>
>
> On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison 
> wrote:
>
> > On 19/09/16 15:46, Darin Johnson wrote:
> > > Hey guys,
> > >
> > > Thanks for looking at the PR, I apologize if it offended anyone's
> eyes:).
> > >
> > > I'm glad it generated some discussion about the configuration.  I
> didn't
> > > really like where things were heading with the config.  However, didn't
> > > want to create to much scope creep.
> > >
> > > I think any hierarchical config (TypeSafe or yaml) would make things
> much
> > > more maintainable, the plugin could simply grab the appropriate part of
> > the
> > > config and handle accordingly.  I'd also cut down the number of command
> > > line options to only those that change between runs often (like
> > > input/output)
> > >
> > >> One option is to make Pirk pluggable, so that a Pirk installation
> could
> > >> use one or more of these in an extensible fashion by adding JAR files.
> > >> That would still require selecting one by command-line argument.
> > >
> > > An argument for this approach is for lambda architecture approaches
> (say
> > > spark/spark-streaming) were the contents of the jars would be so
> similar
> > it
> > > seems like to much trouble to create separate jars.
> > >
> > > Happy to continue working on this given some direction on where you'd
> > like
> > > it to go.  Also, it's a bit of a blocker to refactoring the build into
> > > submodules.
> >
> > FWIW my 2c is to not try and fix all the problems in one go, and rather
> > take a compromise on the configurations while you tease apart the
> > submodules in to separate source code trees, poms, etc; then come back
> > and fix the runtime configs.
> >
> > Once the submodules are in place it will open up more work for release
> > engineering and tinkering that can be done in parallel with the config
> > polishing.
> >
> > Just a thought.
> > Tim
> >
> >
> > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison 
> > wrote:
> > >
> > >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > >>> It seems that it's the same idea as the ResponderLauncher with the
> > >> service
> > >>> component added to maintain something akin to the 'platform'. I would
> > >>> prefer that we just did away with the platform notion altogether and
> > make
> > >>> the ResponderDriver 'dumb'. We get around needing a platform-aware
> > >> service
> > >>> by requiring the ResponderLauncher implementation to be passed as a
> CLI
> > >> to
> > >>> the ResponderDriver.
> > >>
> > >> Let me check I understand what you are saying here.
> > >>
> > >> At the moment, there is a monolithic Pirk that hard codes how to
> respond
> > >> using lots of different backends (mapreduce, spark, sparkstreaming,
> > >> storm , standalone), and that is selected by command-line argument.
> > >>
> > >> One option is to make Pirk pluggable, so that a Pirk installation
> could
> > >> use one or more of these in an extensible fashion by adding JAR files.
> > >> That would still require selecting one by command-line argument.
> > >>
> > >> A second option is to simply pass in the required backend JAR to
> select
> > >> the particular implementation you choose, as a specific Pirk
> > >> installation doesn't need to use multiple backends simultaneously.
> > >>
> > >> ...and you are leaning towards the second option.  Do I have that
> > correct?
> > >>
> > >> Regards,
> > >> Tim
> > >>
> > >>> Am I missing something? Is there a good reason to provide a service
> by
> > >>> which platforms are registered? I'm open...
> > >>>
> > >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
> > >> wrote:
> > >>>
> >  How about an approach like this?
> > https://github.com/tellison/incubator-pirk/tree/pirk-63
> > 
> > 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Darin Johnson]

Hey guys,

Thanks for looking at the PR, I apologize if it offended anyone's eyes:).

I'm glad it generated some discussion about the configuration.  I didn't
really like where things were heading with the config.  However, didn't
want to create to much scope creep.

I think any hierarchical config (TypeSafe or yaml) would make things much
more maintainable, the plugin could simply grab the appropriate part of the
config and handle accordingly.  I'd also cut down the number of command
line options to only those that change between runs often (like
input/output)

>One option is to make Pirk pluggable, so that a Pirk installation could
>use one or more of these in an extensible fashion by adding JAR files.
>That would still require selecting one by command-line argument.

An argument for this approach is for lambda architecture approaches (say
spark/spark-streaming) were the contents of the jars would be so similar it
seems like to much trouble to create separate jars.

Happy to continue working on this given some direction on where you'd like
it to go.  Also, it's a bit of a blocker to refactoring the build into
submodules.

Darin




On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison  wrote:

> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > It seems that it's the same idea as the ResponderLauncher with the
> service
> > component added to maintain something akin to the 'platform'. I would
> > prefer that we just did away with the platform notion altogether and make
> > the ResponderDriver 'dumb'. We get around needing a platform-aware
> service
> > by requiring the ResponderLauncher implementation to be passed as a CLI
> to
> > the ResponderDriver.
>
> Let me check I understand what you are saying here.
>
> At the moment, there is a monolithic Pirk that hard codes how to respond
> using lots of different backends (mapreduce, spark, sparkstreaming,
> storm , standalone), and that is selected by command-line argument.
>
> One option is to make Pirk pluggable, so that a Pirk installation could
> use one or more of these in an extensible fashion by adding JAR files.
> That would still require selecting one by command-line argument.
>
> A second option is to simply pass in the required backend JAR to select
> the particular implementation you choose, as a specific Pirk
> installation doesn't need to use multiple backends simultaneously.
>
> ...and you are leaning towards the second option.  Do I have that correct?
>
> Regards,
> Tim
>
> > Am I missing something? Is there a good reason to provide a service by
> > which platforms are registered? I'm open...
> >
> > On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison 
> wrote:
> >
> >> How about an approach like this?
> >>https://github.com/tellison/incubator-pirk/tree/pirk-63
> >>
> >> The "on-ramp" is the driver [1], which calls upon the service to find a
> >> plug-in [2] that claims to implement the required platform responder,
> >> e.g. [3].
> >>
> >> The list of plug-ins is given in the provider's JAR file, so the ones we
> >> provide in Pirk are listed together [4], but if you split these into
> >> modules, or somebody brings their own JAR alongside, these would be
> >> listed in each JAR's services/ directory.
> >>
> >> [1]
> >> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >> src/main/java/org/apache/pirk/responder/wideskies/ResponderDriver.java
> >> [2]
> >> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
> >> [3]
> >> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >> src/main/java/org/apache/pirk/responder/wideskies/storm/
> >> StormResponder.java
> >> [4]
> >> https://github.com/tellison/incubator-pirk/blob/pirk-63/
> >> src/main/services/org.apache.responder.spi.Responder
> >>
> >> I'm not even going to dignify this with a WIP PR, it is far from ready,
> >> so proceed with caution.  There is hopefully enough there to show the
> >> approach, and if it is worth continuing I'm happy to do so.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >
>

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Tim Ellison]

On 19/09/16 13:39, Suneel Marthi wrote:
> The way this PR is now is so similar to how bad IBM SystemML which is a
> hackwork of hurriedly put together and something I have often pointed out
> to others as a clear example of "how not to design crappy software".  See
> this gist of an example code snippet from IBM SystemML -
> https://gist.github.com/smarthi/eb848e46621b7444924f

Not sure if you are looking at PR93, or the URL I sent you.

I agree that a large, explicit enumeration via a switch/if statement is
not conducive to extensibility, and that is what PIRK-63 is trying to
address.

> First things for the project:
> 
> 1. Move away from using the java properties (this is so 2002 way of doing
> things) to using TypeSafe style configurations which allow for structured
> properties.

>From a quick look, that covers a different level, namely how the
configurations are represented.  First we need to look at the responder
architecture to allow for different responder types to be plugged in to
the Pirk framework.

Each plug-in responder type can figure out how to depict it's configuration.

> 2. From a Responder design, there would be a Responder-impl-class property
> which would be read from TypeSafe config and the appropriate driver class
> invoked.

I've not used style configurations before.  I think they overlap with
the SystemConfiguration a bit.  It would be interesting to see what changes.

> As an example for the above ^^^ two, please look at at the Oryx 2.0 project
> for reference
> 
> https://github.com/oryxproject/oryx

I'd rather look at a proposed change to Pirk ;-)

Regards,
Tim

> On Mon, Sep 19, 2016 at 2:28 PM, Tim Ellison  wrote:
> 
>> How about an approach like this?
>>https://github.com/tellison/incubator-pirk/tree/pirk-63
>>
>> The "on-ramp" is the driver [1], which calls upon the service to find a
>> plug-in [2] that claims to implement the required platform responder,
>> e.g. [3].
>>
>> The list of plug-ins is given in the provider's JAR file, so the ones we
>> provide in Pirk are listed together [4], but if you split these into
>> modules, or somebody brings their own JAR alongside, these would be
>> listed in each JAR's services/ directory.
>>
>> [1]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/wideskies/ResponderDriver.java
>> [2]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
>> [3]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/java/org/apache/pirk/responder/wideskies/storm/
>> StormResponder.java
>> [4]
>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>> src/main/services/org.apache.responder.spi.Responder
>>
>> I'm not even going to dignify this with a WIP PR, it is far from ready,
>> so proceed with caution.  There is hopefully enough there to show the
>> approach, and if it is worth continuing I'm happy to do so.
>>
>> Regards,
>> Tim
>>
>>
> 

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Tim Ellison]

How about an approach like this?
   https://github.com/tellison/incubator-pirk/tree/pirk-63

The "on-ramp" is the driver [1], which calls upon the service to find a
plug-in [2] that claims to implement the required platform responder,
e.g. [3].

The list of plug-ins is given in the provider's JAR file, so the ones we
provide in Pirk are listed together [4], but if you split these into
modules, or somebody brings their own JAR alongside, these would be
listed in each JAR's services/ directory.

[1]
https://github.com/tellison/incubator-pirk/blob/pirk-63/src/main/java/org/apache/pirk/responder/wideskies/ResponderDriver.java
[2]
https://github.com/tellison/incubator-pirk/blob/pirk-63/src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
[3]
https://github.com/tellison/incubator-pirk/blob/pirk-63/src/main/java/org/apache/pirk/responder/wideskies/storm/StormResponder.java
[4]
https://github.com/tellison/incubator-pirk/blob/pirk-63/src/main/services/org.apache.responder.spi.Responder

I'm not even going to dignify this with a WIP PR, it is far from ready,
so proceed with caution.  There is hopefully enough there to show the
approach, and if it is worth continuing I'm happy to do so.

Regards,
Tim

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

[Tim Ellison]

Darin,

Unless I'm reading this wrong, the patch still has many references from
the ResponderDriver to the set of currently supported responders.  This
code will have to change when somebody wants to add a new responder type.

I thought the plan was to have the responder driver agnostic of the
responders available?  So, for example, having the driver maintain a
list of responders by name, and letting people specify the name on the
command line.

Each responder would then be responsible for implementing a standardised
interface, and registering themselves with the driver by name.

In that model the responders would each know about (a) the driver, and
how to register themselves by name, and (b) implement a standard
life-cycle for building a response.

The driver would be responsible for (a) collecting and maintaining the
registrations of any responder being loaded, and (b) invoking the
correct responder based on user selection.

Make sense?

I can hack something together to show what I mean.

Regards,
Tim



On 19/09/16 07:05, DarinJ wrote:
> GitHub user DarinJ opened a pull request:
> 
> https://github.com/apache/incubator-pirk/pull/93
> 
> WIP-Pirk 63-DO NOT MERGE
> 
> This is a WIP for 
> [PIRK-63](https://issues.apache.org/jira/browse/PIRK-63) to open the door to 
> other responders without having to modify the actual code of Pirk.  It's 
> submitted for feedback only, please DO NOT MERGE.  I've only tested 
> standalone mode.
> 
> It deprecates the "platform" CLI option in favor of the "launcher" option 
> which is the name of a class implementing the `ResponderLauncher` interface 
> which will invoke the run method via reflection.  This allows a developer of 
> a different responder to merely place a jar on the classpath and specify the 
> appropriate `ResponderLauncher` on the classpath.
> 
> The "platform" CLI option is still made available.  However, I removed 
> the explicit dependencies in favor of using reflection.  This was done in 
> anticipation other refactoring the build into submodules, though this does 
> admittedly make the code more fragile.
> 
> ResponderDriver had no unit tests, and unfortunately I saw no good way to 
> create good ones for this particular change, especially as it required 
> multiple frameworks to run.
> 
> I should say that another possible route here is to have each framework 
> responder implement their own ResponderDriver.  We could provide some 
> utilities to check the minimum Pirk required options are set, but leave the 
> rest to the implementation of the responder.  It would clean up the 
> ResponderCLI and ResponderProps which are rather bloated and might continue 
> to grow if left unchecked.
> 
> You can merge this pull request into a Git repository by running:
> 
> $ git pull https://github.com/DarinJ/incubator-pirk Pirk-63
> 
> Alternatively you can review and apply these changes as the patch at:
> 
> https://github.com/apache/incubator-pirk/pull/93.patch
> 
> To close this pull request, make a commit to your master/trunk branch
> with (at least) the following in the commit message:
> 
> This closes #93
> 
> 
> commit dda458bb2ae77fd9e3dc686d17dd8b49095b3395
> Author: Darin Johnson 
> Date:   2016-09-13T03:19:12Z
> 
> This is a WIP for 
> [PIRK-63](https://issues.apache.org/jira/browse/PIRK-63) to open the door to 
> other responders without having to modify the actual code of Pirk.  It's 
> submitted for feedback only, please DO NOT MERGE.
> 
> It deprecates the "platform" CLI option in favor of the "launcher" option 
> which is the name of a class implementing the `ResponderLauncher` interface 
> which will invoke the run method via reflection.  This allows a developer of 
> a different responder to merely place a jar on the classpath and specify the 
> appropriate `ResponderLauncher` on the classpath.
> 
> The "platform" CLI option is still made available.  However, I removed 
> the explicit dependencies in favor of using reflection.  This was done in 
> anticipation other refactoring the build into submodules, though this does 
> admittedly make the code more fragile.
> 
> 
> 
> 
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>