On 19/09/16 22:10, Suneel Marthi wrote: > Here's an example from the Flink project for how they go about new features > or system breaking API changes, we could start a similar process. The Flink > guys call these FLIP (Flink Improvement Proposal) and Kafka community > similarly has something called KLIP. > > We could start a PLIP (??? :-) )
I expect an change 'process' will evolve as required, but +1 for starting to document some of the architecture on the Pirk wiki. My notebook has a few scrappy line drawings at the moment. Regards, Tim > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673 > > > On Mon, Sep 19, 2016 at 11:07 PM, Suneel Marthi <[email protected]> > wrote: > >> A shared Google doc would be more convenient than a bunch of Jiras. Its >> easier to comment and add notes that way. >> >> >> On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson <[email protected]> >> wrote: >> >>> Suneel, I'll try to put a couple jiras on it tonight with my thoughts. >>> Based off my pirk-63 I was able to pull spark and storm out with no >>> issues. I was planning to pull them out, then tackling elastic search, >>> then hadoop as it's a little entrenched. This should keep most PRs to >>> manageable chunks. I think once that's done addressing the configs will >>> make more sense. >>> >>> I'm open to suggestions. But the hope would be: >>> Pirk-parent >>> Pirk-core >>> Pirk-hadoop >>> Pirk-storm >>> Pirk-parent >>> >>> Pirk-es is a little weird as it's really just an inputformat, seems like >>> there's a more general solution here than creating submodules for every >>> inputformat. >>> >>> Darin >>> >>> On Sep 19, 2016 1:00 PM, "Suneel Marthi" <[email protected]> wrote: >>> >>>> >>> >>>> Refactor is definitely a first priority. Is there a design/proposal >>> draft >>>> that we could comment on about how to go about refactoring the code. I >>>> have been trying to keep up with the emails but definitely would have >>>> missed some. >>>> >>>> >>>> >>>> On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams < >>>> [email protected] <[email protected]>> wrote: >>>> >>>>> Agree - let's leave the config/CLI the way it is for now and tackle >>> that as >>>>> a subsequent design discussion and PR. >>>>> >>>>> Also, I think that we should leave the ResponderDriver and the >>>>> ResponderProps alone for this PR and push to a subsequent PR (once we >>>>> decide if and how we would like to delegate each). >>>>> >>>>> I vote to remove the 'platform' option and the backwards compatibility >>> in >>>>> this PR and proceed with having a ResponderLauncher interface and >>> forcing >>>>> its implementation by the ResponderDriver. >>>>> >>>>> And, I am not so concerned with having one fat jar vs. multiple jars >>> right >>>>> now - to me, at this point, it's a 'nice to have' and not a 'must >>> have' >>> for >>>>> Pirk functionality. We do need to break out Pirk into more clearly >>> defined >>>>> submodules (which is in progress) - via this re-factor, I think that >>> we >>>>> will gain some ability to generate multiple jars which is nice. >>>>> >>>>> >>>>> >>>>> On Mon, Sep 19, 2016 at 12:19 PM, Tim Ellison <[email protected]> >>>>> wrote: >>>>> >>>>>> On 19/09/16 15:46, Darin Johnson wrote: >>>>>>> Hey guys, >>>>>>> >>>>>>> Thanks for looking at the PR, I apologize if it offended anyone's >>>>> eyes:). >>>>>>> >>>>>>> I'm glad it generated some discussion about the configuration. I >>>>> didn't >>>>>>> really like where things were heading with the config. However, >>> didn't >>>>>>> want to create to much scope creep. >>>>>>> >>>>>>> I think any hierarchical config (TypeSafe or yaml) would make >>> things >>>>> much >>>>>>> more maintainable, the plugin could simply grab the appropriate >>> part of >>>>>> the >>>>>>> config and handle accordingly. I'd also cut down the number of >>> command >>>>>>> line options to only those that change between runs often (like >>>>>>> input/output) >>>>>>> >>>>>>>> One option is to make Pirk pluggable, so that a Pirk installation >>>>> could >>>>>>>> use one or more of these in an extensible fashion by adding JAR >>> files. >>>>>>>> That would still require selecting one by command-line argument. >>>>>>> >>>>>>> An argument for this approach is for lambda architecture >>> approaches >>>>> (say >>>>>>> spark/spark-streaming) were the contents of the jars would be so >>>>> similar >>>>>> it >>>>>>> seems like to much trouble to create separate jars. >>>>>>> >>>>>>> Happy to continue working on this given some direction on where >>> you'd >>>>>> like >>>>>>> it to go. Also, it's a bit of a blocker to refactoring the build >>> into >>>>>>> submodules. >>>>>> >>>>>> FWIW my 2c is to not try and fix all the problems in one go, and >>> rather >>>>>> take a compromise on the configurations while you tease apart the >>>>>> submodules in to separate source code trees, poms, etc; then come >>> back >>>>>> and fix the runtime configs. >>>>>> >>>>>> Once the submodules are in place it will open up more work for >>> release >>>>>> engineering and tinkering that can be done in parallel with the >>> config >>>>>> polishing. >>>>>> >>>>>> Just a thought. >>>>>> Tim >>>>>> >>>>>> >>>>>>> On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison < >>> [email protected]> >>>>>> wrote: >>>>>>> >>>>>>>> On 19/09/16 13:40, Ellison Anne Williams wrote: >>>>>>>>> It seems that it's the same idea as the ResponderLauncher with >>> the >>>>>>>> service >>>>>>>>> component added to maintain something akin to the 'platform'. I >>> would >>>>>>>>> prefer that we just did away with the platform notion altogether >>> and >>>>>> make >>>>>>>>> the ResponderDriver 'dumb'. We get around needing a >>> platform-aware >>>>>>>> service >>>>>>>>> by requiring the ResponderLauncher implementation to be passed >>> as >>> a >>>>> CLI >>>>>>>> to >>>>>>>>> the ResponderDriver. >>>>>>>> >>>>>>>> Let me check I understand what you are saying here. >>>>>>>> >>>>>>>> At the moment, there is a monolithic Pirk that hard codes how to >>>>> respond >>>>>>>> using lots of different backends (mapreduce, spark, >>> sparkstreaming, >>>>>>>> storm , standalone), and that is selected by command-line >>> argument. >>>>>>>> >>>>>>>> One option is to make Pirk pluggable, so that a Pirk installation >>>>> could >>>>>>>> use one or more of these in an extensible fashion by adding JAR >>> files. >>>>>>>> That would still require selecting one by command-line argument. >>>>>>>> >>>>>>>> A second option is to simply pass in the required backend JAR to >>>>> select >>>>>>>> the particular implementation you choose, as a specific Pirk >>>>>>>> installation doesn't need to use multiple backends >>> simultaneously. >>>>>>>> >>>>>>>> ...and you are leaning towards the second option. Do I have that >>>>>> correct? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Tim >>>>>>>> >>>>>>>>> Am I missing something? Is there a good reason to provide a >>> service >>>>> by >>>>>>>>> which platforms are registered? I'm open... >>>>>>>>> >>>>>>>>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison < >>> [email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> How about an approach like this? >>>>>>>>>> https://github.com/tellison/incubator-pirk/tree/pirk-63 >>>>>>>>>> >>>>>>>>>> The "on-ramp" is the driver [1], which calls upon the service >>> to >>>>> find >>>>>> a >>>>>>>>>> plug-in [2] that claims to implement the required platform >>>>> responder, >>>>>>>>>> e.g. [3]. >>>>>>>>>> >>>>>>>>>> The list of plug-ins is given in the provider's JAR file, so >>> the >>>>> ones >>>>>> we >>>>>>>>>> provide in Pirk are listed together [4], but if you split these >>> into >>>>>>>>>> modules, or somebody brings their own JAR alongside, these >>> would >>> be >>>>>>>>>> listed in each JAR's services/ directory. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>>>>>> src/main/java/org/apache/pirk/responder/wideskies/ >>>>>> ResponderDriver.java >>>>>>>>>> [2] >>>>>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>>>>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin. >>> java >>>>>>>>>> [3] >>>>>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>>>>>> src/main/java/org/apache/pirk/responder/wideskies/storm/ >>>>>>>>>> StormResponder.java >>>>>>>>>> [4] >>>>>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/ >>>>>>>>>> src/main/services/org.apache.responder.spi.Responder >>>>>>>>>> >>>>>>>>>> I'm not even going to dignify this with a WIP PR, it is far >>> from >>>>>> ready, >>>>>>>>>> so proceed with caution. There is hopefully enough there to >>> show >>>>> the >>>>>>>>>> approach, and if it is worth continuing I'm happy to do so. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> >> >
