A potential use case (please correct me here) could be that someone needs a
Storm impl of Responder which means we need to generate an artifact like -
'pirk-responder-storm.jar'



On Wed, Sep 21, 2016 at 5:41 AM, Darin Johnson <dbjohnson1...@gmail.com>
wrote:

> So along these lines I want to ask one more set of questions.
>
> Since storm/spark/hadoop are just responders do we want to put them as
> modules below responder? I'm not in favor of this but thought I'd ask.
>
> Does it make sense to put some responders in a contrib section?  Storm does
> this for a lot of spouts and things.  I think it will make sense
> eventually.
>
>
>
> On Tue, Sep 20, 2016 at 10:22 PM, Ellison Anne Williams <
> eawilli...@apache.org> wrote:
>
> > I am in favor of breaking out pirk-core as specified so that our initial
> > submodule structure would be as follows:
> >
> > pirk-core (encryption,query, inputformat, serialization, utils)
> >
> > pirk-responder (core responder incl. standalone)
> >
> > pirk-querier
> >
> > pirk-storm
> >
> > pirk-mapreduce
> >
> > pirk-spark
> >
> > pirk-benchmark
> >
> > pirk-distributed-test
> >
> >
> > One thing to note is that under this breakdown, pirk-core would not
> include
> > the Elasticsearch dependency (es-hadoop). The only submodules that would
> > have the es-hadoop dependency (those which need it) currently are
> > pirk-mapreduce, pirk-spark, and pirk-distributed-test.
> >
> >
> > I believe that we agreed (somewhere :)) in this thread to go ahead and
> > remove the platform 'backwards compatibility' for PIRK-63. Please holler
> if
> > this is not correct.
> >
> >
> >
> >
> > On Tue, Sep 20, 2016 at 9:40 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> >
> > > Suneel, a google doc as promised, only a day late (sorry - sick kid).
> > >
> > > https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_
> > > mMrRQyynQ-Q6MFbI/edit?usp=sharing
> > >
> > > I was planning on working on this, but I'm going to take a day or two
> to
> > > let others comment.
> > >
> > > Darin
> > >
> > > On Mon, Sep 19, 2016 at 5:07 PM, Suneel Marthi <
> suneel.mar...@gmail.com>
> > > wrote:
> > >
> > > > A shared Google doc would be more convenient than a bunch of Jiras.
> Its
> > > > easier to comment and add notes that way.
> > > >
> > > >
> > > > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson <
> > dbjohnson1...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Suneel, I'll try to put a couple jiras on it tonight with my
> > thoughts.
> > > > > Based off my pirk-63 I was able to pull spark and storm out with no
> > > > > issues.  I was planning to pull them out, then tackling elastic
> > search,
> > > > > then hadoop as it's a little entrenched.  This should keep most PRs
> > to
> > > > > manageable chunks. I think once that's done addressing the configs
> > will
> > > > > make more sense.
> > > > >
> > > > > I'm open to suggestions. But the hope would be:
> > > > > Pirk-parent
> > > > > Pirk-core
> > > > > Pirk-hadoop
> > > > > Pirk-storm
> > > > > Pirk-parent
> > > > >
> > > > > Pirk-es is a little weird as it's really just an inputformat, seems
> > > like
> > > > > there's a more general solution here than creating submodules for
> > every
> > > > > inputformat.
> > > > >
> > > > > Darin
> > > > >
> > > > > On Sep 19, 2016 1:00 PM, "Suneel Marthi" <smar...@apache.org>
> wrote:
> > > > >
> > > > > >
> > > > >
> > > > > > Refactor is definitely a first priority.  Is there a
> > design/proposal
> > > > > draft
> > > > > > that we could comment on about how to go about refactoring the
> > > code.  I
> > > > > > have been trying to keep up with the emails but definitely would
> > have
> > > > > > missed some.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <
> > > > > > eawilli...@apache.org <eawilli...@apache.org>> wrote:
> > > > > >
> > > > > > > Agree - let's leave the config/CLI the way it is for now and
> > tackle
> > > > > that as
> > > > > > > a subsequent design discussion and PR.
> > > > > > >
> > > > > > > Also, I think that we should leave the ResponderDriver and the
> > > > > > > ResponderProps alone for this PR and push to a subsequent PR
> > (once
> > > we
> > > > > > > decide if and how we would like to delegate each).
> > > > > > >
> > > > > > > I vote to remove the 'platform' option and the backwards
> > > > compatibility
> > > > > in
> > > > > > > this PR and proceed with having a ResponderLauncher interface
> and
> > > > > forcing
> > > > > > > its implementation by the ResponderDriver.
> > > > > > >
> > > > > > > And, I am not so concerned with having one fat jar vs. multiple
> > > jars
> > > > > right
> > > > > > > now - to me, at this point, it's a 'nice to have' and not a
> 'must
> > > > have'
> > > > > for
> > > > > > > Pirk functionality. We do need to break out Pirk into more
> > clearly
> > > > > defined
> > > > > > > submodules (which is in progress) - via this re-factor, I think
> > > that
> > > > we
> > > > > > > will gain some ability to generate multiple jars which is nice.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <
> > > t.p.elli...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On 19/09/16 15:46, Darin Johnson wrote:
> > > > > > > > > Hey guys,
> > > > > > > > >
> > > > > > > > > Thanks for looking at the PR, I apologize if it offended
> > > anyone's
> > > > > > > eyes:).
> > > > > > > > >
> > > > > > > > > I'm glad it generated some discussion about the
> > > configuration.  I
> > > > > > > didn't
> > > > > > > > > really like where things were heading with the config.
> > > However,
> > > > > didn't
> > > > > > > > > want to create to much scope creep.
> > > > > > > > >
> > > > > > > > > I think any hierarchical config (TypeSafe or yaml) would
> make
> > > > > things
> > > > > > > much
> > > > > > > > > more maintainable, the plugin could simply grab the
> > appropriate
> > > > > part of
> > > > > > > > the
> > > > > > > > > config and handle accordingly.  I'd also cut down the
> number
> > of
> > > > > command
> > > > > > > > > line options to only those that change between runs often
> > (like
> > > > > > > > > input/output)
> > > > > > > > >
> > > > > > > > >> One option is to make Pirk pluggable, so that a Pirk
> > > > installation
> > > > > > > could
> > > > > > > > >> use one or more of these in an extensible fashion by
> adding
> > > JAR
> > > > > files.
> > > > > > > > >> That would still require selecting one by command-line
> > > argument.
> > > > > > > > >
> > > > > > > > > An argument for this approach is for lambda architecture
> > > > approaches
> > > > > > > (say
> > > > > > > > > spark/spark-streaming) were the contents of the jars would
> be
> > > so
> > > > > > > similar
> > > > > > > > it
> > > > > > > > > seems like to much trouble to create separate jars.
> > > > > > > > >
> > > > > > > > > Happy to continue working on this given some direction on
> > where
> > > > > you'd
> > > > > > > > like
> > > > > > > > > it to go.  Also, it's a bit of a blocker to refactoring the
> > > build
> > > > > into
> > > > > > > > > submodules.
> > > > > > > >
> > > > > > > > FWIW my 2c is to not try and fix all the problems in one go,
> > and
> > > > > rather
> > > > > > > > take a compromise on the configurations while you tease apart
> > the
> > > > > > > > submodules in to separate source code trees, poms, etc; then
> > come
> > > > > back
> > > > > > > > and fix the runtime configs.
> > > > > > > >
> > > > > > > > Once the submodules are in place it will open up more work
> for
> > > > > release
> > > > > > > > engineering and tinkering that can be done in parallel with
> the
> > > > > config
> > > > > > > > polishing.
> > > > > > > >
> > > > > > > > Just a thought.
> > > > > > > > Tim
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <
> > > > > t.p.elli...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On 19/09/16 13:40, Ellison Anne Williams wrote:
> > > > > > > > >>> It seems that it's the same idea as the ResponderLauncher
> > > with
> > > > > the
> > > > > > > > >> service
> > > > > > > > >>> component added to maintain something akin to the
> > > 'platform'. I
> > > > > would
> > > > > > > > >>> prefer that we just did away with the platform notion
> > > > altogether
> > > > > and
> > > > > > > > make
> > > > > > > > >>> the ResponderDriver 'dumb'. We get around needing a
> > > > > platform-aware
> > > > > > > > >> service
> > > > > > > > >>> by requiring the ResponderLauncher implementation to be
> > > passed
> > > > as
> > > > > a
> > > > > > > CLI
> > > > > > > > >> to
> > > > > > > > >>> the ResponderDriver.
> > > > > > > > >>
> > > > > > > > >> Let me check I understand what you are saying here.
> > > > > > > > >>
> > > > > > > > >> At the moment, there is a monolithic Pirk that hard codes
> > how
> > > to
> > > > > > > respond
> > > > > > > > >> using lots of different backends (mapreduce, spark,
> > > > > sparkstreaming,
> > > > > > > > >> storm , standalone), and that is selected by command-line
> > > > > argument.
> > > > > > > > >>
> > > > > > > > >> One option is to make Pirk pluggable, so that a Pirk
> > > > installation
> > > > > > > could
> > > > > > > > >> use one or more of these in an extensible fashion by
> adding
> > > JAR
> > > > > files.
> > > > > > > > >> That would still require selecting one by command-line
> > > argument.
> > > > > > > > >>
> > > > > > > > >> A second option is to simply pass in the required backend
> > JAR
> > > to
> > > > > > > select
> > > > > > > > >> the particular implementation you choose, as a specific
> Pirk
> > > > > > > > >> installation doesn't need to use multiple backends
> > > > simultaneously.
> > > > > > > > >>
> > > > > > > > >> ...and you are leaning towards the second option.  Do I
> have
> > > > that
> > > > > > > > correct?
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> Tim
> > > > > > > > >>
> > > > > > > > >>> Am I missing something? Is there a good reason to
> provide a
> > > > > service
> > > > > > > by
> > > > > > > > >>> which platforms are registered? I'm open...
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <
> > > > > t.p.elli...@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> How about an approach like this?
> > > > > > > > >>>>    https://github.com/tellison/
> > incubator-pirk/tree/pirk-63
> > > > > > > > >>>>
> > > > > > > > >>>> The "on-ramp" is the driver [1], which calls upon the
> > > service
> > > > to
> > > > > > > find
> > > > > > > > a
> > > > > > > > >>>> plug-in [2] that claims to implement the required
> platform
> > > > > > > responder,
> > > > > > > > >>>> e.g. [3].
> > > > > > > > >>>>
> > > > > > > > >>>> The list of plug-ins is given in the provider's JAR
> file,
> > so
> > > > the
> > > > > > > ones
> > > > > > > > we
> > > > > > > > >>>> provide in Pirk are listed together [4], but if you
> split
> > > > these
> > > > > into
> > > > > > > > >>>> modules, or somebody brings their own JAR alongside,
> these
> > > > would
> > > > > be
> > > > > > > > >>>> listed in each JAR's services/ directory.
> > > > > > > > >>>>
> > > > > > > > >>>> [1]
> > > > > > > > >>>> https://github.com/tellison/
> incubator-pirk/blob/pirk-63/
> > > > > > > > >>>> src/main/java/org/apache/pirk/responder/wideskies/
> > > > > > > > ResponderDriver.java
> > > > > > > > >>>> [2]
> > > > > > > > >>>> https://github.com/tellison/
> incubator-pirk/blob/pirk-63/
> > > > > > > > >>>> src/main/java/org/apache/pirk/
> > > responder/spi/ResponderPlugin.
> > > > > java
> > > > > > > > >>>> [3]
> > > > > > > > >>>> https://github.com/tellison/
> incubator-pirk/blob/pirk-63/
> > > > > > > > >>>> src/main/java/org/apache/pirk/
> responder/wideskies/storm/
> > > > > > > > >>>> StormResponder.java
> > > > > > > > >>>> [4]
> > > > > > > > >>>> https://github.com/tellison/
> incubator-pirk/blob/pirk-63/
> > > > > > > > >>>> src/main/services/org.apache.responder.spi.Responder
> > > > > > > > >>>>
> > > > > > > > >>>> I'm not even going to dignify this with a WIP PR, it is
> > far
> > > > from
> > > > > > > > ready,
> > > > > > > > >>>> so proceed with caution.  There is hopefully enough
> there
> > to
> > > > > show
> > > > > > > the
> > > > > > > > >>>> approach, and if it is worth continuing I'm happy to do
> > so.
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>> Tim
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to