Re: Data (re)processing with Kafka (new wiki page)

[Matthias J. Sax]

Thanks for your input!

For (2), there is currently KafkaStream.toString(). You could
theoretically parse the String to extract all required information -- of
course, this is not a nice solution.

-Matthias

On 11/25/16 2:11 AM, saiprasad mishra wrote:
> This page is really helpful.Thanks for putting this
> 
> Some nice to have features can be (not sure for this wiki page)
> 
> 1) Pause and resume without having to start and stop.
> It should drain all the inflight calculations before doing the actual pause
> and a notifier will be helpful that it is actually paused.
> This can be much light wait if possible rather than stopping all tasks and
> stores and starting them again which might take a lot of time
> 
> 
> 2) Metadata api response if it can have all the topology graphs and sub
> graphs with all the nodes and edges for each sub graphs with corresponding
> state store names then it will be easier to build some UI on top of it and
> also the pipe between the sub graphs which are kafka topics need to be
> called out and also the time semantics can be laid out on top of it. This
> is something like a logical view on top of the physical view which the
> current api has.
> 
> 
> Regards
> Sai
> 
> On Fri, Nov 25, 2016 at 12:53 AM, Michael Noll  wrote:
> 
>> Thanks a lot, Matthias!
>>
>> I have already begun to provide feedback.
>>
>> -Michael
>>
>>
>>
>> On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax 
>> wrote:
>>
>>> Hi,
>>>
>>> we added a new wiki page that is supposed to collect data (re)processing
>>> scenario with Kafka:
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/
>> Kafka+Streams+Data+(Re)
>>> Processing+Scenarios
>>>
>>> We added already a couple of scenarios we think might be common and want
>>> to invite all of you to add more. This helps to get a better overview of
>>> requirements to enable new use cases.
>>>
>>> We are looking forward to your feedback!
>>>
>>>
>>> -Matthias
>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Data (re)processing with Kafka (new wiki page)

[saiprasad mishra]

This page is really helpful.Thanks for putting this

Some nice to have features can be (not sure for this wiki page)

1) Pause and resume without having to start and stop.
It should drain all the inflight calculations before doing the actual pause
and a notifier will be helpful that it is actually paused.
This can be much light wait if possible rather than stopping all tasks and
stores and starting them again which might take a lot of time


2) Metadata api response if it can have all the topology graphs and sub
graphs with all the nodes and edges for each sub graphs with corresponding
state store names then it will be easier to build some UI on top of it and
also the pipe between the sub graphs which are kafka topics need to be
called out and also the time semantics can be laid out on top of it. This
is something like a logical view on top of the physical view which the
current api has.


Regards
Sai

On Fri, Nov 25, 2016 at 12:53 AM, Michael Noll  wrote:

> Thanks a lot, Matthias!
>
> I have already begun to provide feedback.
>
> -Michael
>
>
>
> On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax 
> wrote:
>
> > Hi,
> >
> > we added a new wiki page that is supposed to collect data (re)processing
> > scenario with Kafka:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Streams+Data+(Re)
> > Processing+Scenarios
> >
> > We added already a couple of scenarios we think might be common and want
> > to invite all of you to add more. This helps to get a better overview of
> > requirements to enable new use cases.
> >
> > We are looking forward to your feedback!
> >
> >
> > -Matthias
> >
> >
>

Re: Data (re)processing with Kafka (new wiki page)

[Michael Noll]

Thanks a lot, Matthias!

I have already begun to provide feedback.

-Michael



On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax 
wrote:

> Hi,
>
> we added a new wiki page that is supposed to collect data (re)processing
> scenario with Kafka:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+(Re)
> Processing+Scenarios
>
> We added already a couple of scenarios we think might be common and want
> to invite all of you to add more. This helps to get a better overview of
> requirements to enable new use cases.
>
> We are looking forward to your feedback!
>
>
> -Matthias
>
>

Data (re)processing with Kafka (new wiki page)

[Matthias J. Sax]

Hi,

we added a new wiki page that is supposed to collect data (re)processing
scenario with Kafka:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+(Re)Processing+Scenarios

We added already a couple of scenarios we think might be common and want
to invite all of you to add more. This helps to get a better overview of
requirements to enable new use cases.

We are looking forward to your feedback!


-Matthias



signature.asc
Description: OpenPGP digital signature