Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Responses in line. Glad you brought this up. On Oct 11, 2016 2:16 PM, "Matt Franklin" wrote: > > Dredging up the past here. After working with Streams for a couple of > years, I think things work fairly well, but still see a need for a more > reactive producer

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Do we have consensus on next steps? From what I can see, everyone agrees that the addition of an isRunning method to the provider makes sense. I will create a ticket and commit that change; but, I encourage others to continue discussion on the next steps for improvement. On Thu, May 15, 2014

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Steve, Thanks for the reply! I think that both can be accommodated, using different interfaces, maybe there is room for 2 types of processor? While, this paradigm might be great for Pig and MR1, it falls short on what can be done with Storm. It also falls short of many complicated ETL problems

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

My biggest concern against the proposed interfaces is that it won't guarantee that all streams components will be able to run in the storm runtime (which I know is not exactly working at the moment). Storm guarantees the processing of every tuple that enters the system from a spout. Therefore

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Hi all, After working with the Streams project a bit, I have noticed some of the same issues that Matt and Ryan have brought up. I think that Matt's idea to implement two interfaces (Producer, Listener) would make a great addition to the project. Not only would it increase efficiency but it would

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Fundamentally processors as initially conceived do not maintain fire events autonomously or maintain state between messages. Changing that paradigm would mean Pig/MR1 would not longer be capable of serving as a full-featured processor runtime. Agree this is limiting, but only in terms of what

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

On Tue, May 6, 2014 at 9:53 PM, Matthew Hager [W2O Digital] mha...@w2odigital.com wrote: Good Day! I would like to throw in my two pents in on this if it pleases the community. Here are my thoughts based on implementations that I have written with streams to ensure timely, high yield

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Matt, As always thanks for your feedback and mentorship as I work to contribute to this project. I feel the current pattern for processor is extremely limiting given the constraint of the return statement rather than the alternative of receive - process - write. It seems that if we revise the

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

On Mon, May 5, 2014 at 1:15 PM, Steve Blackmon sblack...@apache.org wrote: What I meant to say re #1 below is that batch-level metadata could be useful for modules downstream of the StreamsProvider / StreamsPersistReader, and the StreamsResultSet gives us a class to which we can add new

Re: Proposing Changes to the StreamsProvider interface and StreamsProviderTask

Comments on this in-line below. On Thu, May 1, 2014 at 4:38 PM, Ryan Ebanks ryaneba...@gmail.com wrote: The use and implementations of the StreamsProviders seems to have drifted away from what it was originally designed for. I recommend that we change the StreamsProvider interface and