Guozhang, By interleaved groups of message, I meant something like this: Say we have message 0,1,2,3, message 0 and 2 together completes a business logic, message 1 and 3 together completes a business logic. In that case, after user processed message 2, they cannot commit offsets because if they crash before processing message 3, message 1 will not be reconsumed. That means it is possible that user are not able to find a point where the current state is safe to be committed.
This is one example in the use case space table. It is still not clear to me which use cases in the use case space table KIP-28 wants to cover. Are we only covering the case for static topic stream with semi-auto commit? i.e. user cannot change topic subscription on the fly and they can only commit the current offset. Thanks, Jiangjie (Becket) Qin On Mon, Aug 10, 2015 at 6:57 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hello folks, > > I have updated the KIP page with some detailed API / architecture / > packaging proposals, along with the long promised first patch in PR: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client > > https://github.com/apache/kafka/pull/130 > > > Any feedbacks / comments are more than welcomed. > > Guozhang > > > On Mon, Aug 10, 2015 at 6:55 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Jun, > > > > 1. I have removed the streamTime in punctuate() since it is not only > > triggered by clock time, detailed explanation can be found here: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client#KIP-28-Addaprocessorclient-StreamTime > > > > 2. Yes, if users do not schedule a task, then punctuate will never fire. > > > > 3. Yes, I agree. The reason it was implemented in this way is that the > > state store registration call is triggered by the users. However I think > it > > is doable to change that API so that it will be more natural to have sth. > > like: > > > > context.createStore(store-name, store-type). > > > > Guozhang > > > > On Tue, Aug 4, 2015 at 9:17 AM, Jun Rao <j...@confluent.io> wrote: > > > >> A few questions/comments. > >> > >> 1. What's streamTime passed to punctuate()? Is that just the current > time? > >> 2. Is punctuate() only called if schedule() is called? > >> 3. The way the KeyValueStore is created seems a bit weird. Since this is > >> part of the internal state managed by KafkaProcessorContext, it seems > >> there > >> should be an api to create the KeyValueStore from KafkaProcessorContext, > >> instead of passing context to the constructor of KeyValueStore? > >> > >> Thanks, > >> > >> Jun > >> > >> On Thu, Jul 23, 2015 at 5:59 PM, Guozhang Wang <wangg...@gmail.com> > >> wrote: > >> > >> > Hi all, > >> > > >> > I just posted KIP-28: Add a transform client for data processing > >> > < > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+transform+client+for+data+processing > >> > > > >> > . > >> > > >> > The wiki page does not yet have the full design / implementation > >> details, > >> > and this email is to kick-off the conversation on whether we should > add > >> > this new client with the described motivations, and if yes what > >> features / > >> > functionalities should be included. > >> > > >> > Looking forward to your feedback! > >> > > >> > -- Guozhang > >> > > >> > > > > > > > > -- > > -- Guozhang > > > > > > -- > -- Guozhang >
