On 19/09/16 21:22, Darin Johnson wrote:
> Alright, that was in the spirit of what I was thinking when I did this.
> 
> Why don't we take Tim's suggested improvements to my PR (I'll do the
> necessary cleanup) and at the same time just remove the platform argument
> altogether since backwards compatibility isn't upsetting anyone.
> 
> We'll still need a command line option for the launcher for now as we don't
> have submodules we can decide between the two choices after we break out
> submodules and improve the config.

Sounds great - let me know how I can help.

Regards,
Tim


> On Sep 19, 2016 12:19 PM, "Tim Ellison" <t.p.elli...@gmail.com> wrote:
> 
>> On 19/09/16 15:46, Darin Johnson wrote:
>>> Hey guys,
>>>
>>> Thanks for looking at the PR, I apologize if it offended anyone's eyes:).
>>>
>>> I'm glad it generated some discussion about the configuration.  I didn't
>>> really like where things were heading with the config.  However, didn't
>>> want to create to much scope creep.
>>>
>>> I think any hierarchical config (TypeSafe or yaml) would make things much
>>> more maintainable, the plugin could simply grab the appropriate part of
>> the
>>> config and handle accordingly.  I'd also cut down the number of command
>>> line options to only those that change between runs often (like
>>> input/output)
>>>
>>>> One option is to make Pirk pluggable, so that a Pirk installation could
>>>> use one or more of these in an extensible fashion by adding JAR files.
>>>> That would still require selecting one by command-line argument.
>>>
>>> An argument for this approach is for lambda architecture approaches (say
>>> spark/spark-streaming) were the contents of the jars would be so similar
>> it
>>> seems like to much trouble to create separate jars.
>>>
>>> Happy to continue working on this given some direction on where you'd
>> like
>>> it to go.  Also, it's a bit of a blocker to refactoring the build into
>>> submodules.
>>
>> FWIW my 2c is to not try and fix all the problems in one go, and rather
>> take a compromise on the configurations while you tease apart the
>> submodules in to separate source code trees, poms, etc; then come back
>> and fix the runtime configs.
>>
>> Once the submodules are in place it will open up more work for release
>> engineering and tinkering that can be done in parallel with the config
>> polishing.
>>
>> Just a thought.
>> Tim
>>
>>
>>> On Mon, Sep 19, 2016 at 9:33 AM, Tim Ellison <t.p.elli...@gmail.com>
>> wrote:
>>>
>>>> On 19/09/16 13:40, Ellison Anne Williams wrote:
>>>>> It seems that it's the same idea as the ResponderLauncher with the
>>>> service
>>>>> component added to maintain something akin to the 'platform'. I would
>>>>> prefer that we just did away with the platform notion altogether and
>> make
>>>>> the ResponderDriver 'dumb'. We get around needing a platform-aware
>>>> service
>>>>> by requiring the ResponderLauncher implementation to be passed as a CLI
>>>> to
>>>>> the ResponderDriver.
>>>>
>>>> Let me check I understand what you are saying here.
>>>>
>>>> At the moment, there is a monolithic Pirk that hard codes how to respond
>>>> using lots of different backends (mapreduce, spark, sparkstreaming,
>>>> storm , standalone), and that is selected by command-line argument.
>>>>
>>>> One option is to make Pirk pluggable, so that a Pirk installation could
>>>> use one or more of these in an extensible fashion by adding JAR files.
>>>> That would still require selecting one by command-line argument.
>>>>
>>>> A second option is to simply pass in the required backend JAR to select
>>>> the particular implementation you choose, as a specific Pirk
>>>> installation doesn't need to use multiple backends simultaneously.
>>>>
>>>> ...and you are leaning towards the second option.  Do I have that
>> correct?
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>> Am I missing something? Is there a good reason to provide a service by
>>>>> which platforms are registered? I'm open...
>>>>>
>>>>> On Mon, Sep 19, 2016 at 8:28 AM, Tim Ellison <t.p.elli...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> How about an approach like this?
>>>>>>    https://github.com/tellison/incubator-pirk/tree/pirk-63
>>>>>>
>>>>>> The "on-ramp" is the driver [1], which calls upon the service to find
>> a
>>>>>> plug-in [2] that claims to implement the required platform responder,
>>>>>> e.g. [3].
>>>>>>
>>>>>> The list of plug-ins is given in the provider's JAR file, so the ones
>> we
>>>>>> provide in Pirk are listed together [4], but if you split these into
>>>>>> modules, or somebody brings their own JAR alongside, these would be
>>>>>> listed in each JAR's services/ directory.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>>>>>> src/main/java/org/apache/pirk/responder/wideskies/
>> ResponderDriver.java
>>>>>> [2]
>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>>>>>> src/main/java/org/apache/pirk/responder/spi/ResponderPlugin.java
>>>>>> [3]
>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>>>>>> src/main/java/org/apache/pirk/responder/wideskies/storm/
>>>>>> StormResponder.java
>>>>>> [4]
>>>>>> https://github.com/tellison/incubator-pirk/blob/pirk-63/
>>>>>> src/main/services/org.apache.responder.spi.Responder
>>>>>>
>>>>>> I'm not even going to dignify this with a WIP PR, it is far from
>> ready,
>>>>>> so proceed with caution.  There is hopefully enough there to show the
>>>>>> approach, and if it is worth continuing I'm happy to do so.
>>>>>>
>>>>>> Regards,
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 

Reply via email to