Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Why are we using the service-plugin model when we are dropping the platform designations? Thought that we were going to drop all central platform awareness from Pirk in favor of the ResponderLauncher. The service-plugin model retains it in a roundabout way as we have to statically maintain the

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

I am in favor of breaking out pirk-core as specified so that our initial submodule structure would be as follows: pirk-core (encryption,query, inputformat, serialization, utils) pirk-responder (core responder incl. standalone) pirk-querier pirk-storm pirk-mapreduce pirk-spark pirk-benchmark

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Suneel, a google doc as promised, only a day late (sorry - sick kid). https://docs.google.com/document/d/1K8E0TridC1hBfqDwWCqdZ8Dj_5_mMrRQyynQ-Q6MFbI/edit?usp=sharing I was planning on working on this, but I'm going to take a day or two to let others comment. Darin On Mon, Sep 19, 2016 at 5:07

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

On 19/09/16 22:10, Suneel Marthi wrote: > Here's an example from the Flink project for how they go about new features > or system breaking API changes, we could start a similar process. The Flink > guys call these FLIP (Flink Improvement Proposal) and Kafka community > similarly has something

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

On 19/09/16 21:22, Darin Johnson wrote: > Alright, that was in the spirit of what I was thinking when I did this. > > Why don't we take Tim's suggested improvements to my PR (I'll do the > necessary cleanup) and at the same time just remove the platform argument > altogether since backwards

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Great will write up the doc link here, finish pirk63 then start this. On Sep 19, 2016 5:34 PM, "Suneel Marthi" wrote: > +100 > > On Mon, Sep 19, 2016 at 11:24 PM, Ellison Anne Williams < > eawilli...@apache.org> wrote: > > > Yes, ES is just an inputformat (like HDFS,

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

+100 On Mon, Sep 19, 2016 at 11:24 PM, Ellison Anne Williams < eawilli...@apache.org> wrote: > Yes, ES is just an inputformat (like HDFS, Kafka, etc) - we don't need a > separate submodule. > > Aside from pirk-core, it seems that we would want to break the responder > implementations out into

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Yes, ES is just an inputformat (like HDFS, Kafka, etc) - we don't need a separate submodule. Aside from pirk-core, it seems that we would want to break the responder implementations out into submodules. This would leave us with something along the lines of the following (at this point):

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Here's an example from the Flink project for how they go about new features or system breaking API changes, we could start a similar process. The Flink guys call these FLIP (Flink Improvement Proposal) and Kafka community similarly has something called KLIP. We could start a PLIP (??? :-) )

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Sounds good. On Mon, Sep 19, 2016 at 4:22 PM, Darin Johnson wrote: > Alright, that was in the spirit of what I was thinking when I did this. > > Why don't we take Tim's suggested improvements to my PR (I'll do the > necessary cleanup) and at the same time just remove

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Sure will do tonight. On Sep 19, 2016 5:07 PM, "Suneel Marthi" wrote: > A shared Google doc would be more convenient than a bunch of Jiras. Its > easier to comment and add notes that way. > > > On Mon, Sep 19, 2016 at 10:38 PM, Darin Johnson >

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Suneel, I'll try to put a couple jiras on it tonight with my thoughts. Based off my pirk-63 I was able to pull spark and storm out with no issues. I was planning to pull them out, then tackling elastic search, then hadoop as it's a little entrenched. This should keep most PRs to manageable

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Alright, that was in the spirit of what I was thinking when I did this. Why don't we take Tim's suggested improvements to my PR (I'll do the necessary cleanup) and at the same time just remove the platform argument altogether since backwards compatibility isn't upsetting anyone. We'll still need

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Refactor is definitely a first priority. Is there a design/proposal draft that we could comment on about how to go about refactoring the code. I have been trying to keep up with the emails but definitely would have missed some. On Mon, Sep 19, 2016 at 6:57 PM, Ellison Anne Williams <

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Hey guys, Thanks for looking at the PR, I apologize if it offended anyone's eyes:). I'm glad it generated some discussion about the configuration. I didn't really like where things were heading with the config. However, didn't want to create to much scope creep. I think any hierarchical

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

On 19/09/16 13:39, Suneel Marthi wrote: > The way this PR is now is so similar to how bad IBM SystemML which is a > hackwork of hurriedly put together and something I have often pointed out > to others as a clear example of "how not to design crappy software". See > this gist of an example code

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

How about an approach like this? https://github.com/tellison/incubator-pirk/tree/pirk-63 The "on-ramp" is the driver [1], which calls upon the service to find a plug-in [2] that claims to implement the required platform responder, e.g. [3]. The list of plug-ins is given in the provider's JAR

Re: [GitHub] incubator-pirk pull request #93: WIP-Pirk 63-DO NOT MERGE

Darin, Unless I'm reading this wrong, the patch still has many references from the ResponderDriver to the set of currently supported responders. This code will have to change when somebody wants to add a new responder type. I thought the plan was to have the responder driver agnostic of the