Hi Boyang, Thanks for pointing out the possibility of skipping bad records (never crossed my mind). I suppose we could make it an option for the user if they could skip a bad record. It was never the intention of this KIP though on whether or not to do that. I could log a JIRA on such an issue, but I think this is out of the KIP's scope.
As for the ordering guarantees, if you are using the standard Kafka design of one thread per task. Then everything will pretty much remain the same. However, if we are talking about using multiple threads per task (which is something that this KIP proposes), then we should probably expect the behavior to be somewhat similar to Samza's Async Task as stated in the JIRA for this KIP (second-last comment). Ordering would no longer be possible (so yeah, basically no guarantee at all). And how the user handles out-of-order messages is not something I'm well versed in. I guess they can try to put the messages back in order some time later on. But I honestly don't know what they will do. It would be good if you could give me some insight into this. Cheers, Richard On Fri, Dec 21, 2018 at 4:24 PM Boyang Chen <bche...@outlook.com> wrote: > Thanks Richard for proposing this feature! We also have encountered some > similar feature request that we want to define a generic async processing > API<https://issues.apache.org/jira/browse/KAFKA-7566>. > > However I guess the motivation here is that we should skip big records > during normal processing, or let a separate task handle those records who > takes P99 processing time. Since my feeling is that if some edge cases > happen, could we just skip the bad record and continue processing next > record? > > Also I want to understand what kind of ordering guarantee we are gonna > provide with this new API, or there is no ordering guarantee at all? Could > we discuss any potential issues if consumer needs to process out-of-order > messages? > > Best, > Boyang > ________________________________ > From: Richard Yu <yohan.richard...@gmail.com> > Sent: Saturday, December 22, 2018 2:00 AM > To: dev@kafka.apache.org > Subject: KIP-408: Add Asynchronous Processing to Kafka Streams > > Hi all, > > Lately, there has been considerable interest in adding asynchronous > processing to Kafka Streams. > Here is the KIP for such an addition: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-408%3A+Add+Asynchronous+Processing+To+Kafka+Streams > > I wish to discuss the best ways to approach this problem. > > Thanks, > Richard Yu >
