Thanks for reply Jason! Is it any way to control size of this buffer? Will it fails if I try to commit offsets for 100 topics/partitions?
Sure I can work around it by batching all commits into one call, but the problem I can see is that the API does not enforce me to do this. From client point of view I did some correct stuff and got the exception (even more - asynchronous exception) with no idea what was wrong in my actions. On Tue, Mar 15, 2016 at 1:00 AM, Jason Gustafson <ja...@confluent.io> wrote: > Hey Alexey, > > Asynchronous commit handling could probably be improved quite a bit. > Basically what's happening is that the client's send buffer is getting > filled up, which causes the subsequent commits to fail with > SendFailedException. We don't currently implement any retrying for > asynchronous commits, so even transient failures get bubbled up to the > user. This is.. unfortunate. We probably can implement some retry logic, > but we need to be careful careful about preserving the commit order. Maybe > this is as simple as adding sequence numbers for each partition. I haven't > thought it all the way through, but you're welcome to open a JIRA if you > think it would be helpful. > > For now, the easiest way to mitigate the problem is probably to batch the > commits into a single call to commitAsync(). Would that work for you? > > -Jason > > On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk < > alexey.romanc...@gmail.com> wrote: > > > Hi all! > > > > I am using new client 0.9.0.1. > > > > I found that when I call commitAsync multiple times before calling poll > > most of commits failed with SendFailedException. > > > > Here it is an example of code - > > https://gist.github.com/13h3r/42633bcd64b80ddffe6b > > > > Could you please explain commitAsync in more details? > > > > What is wrong with calling commitAsync multiple times? Should I call poll > > immediate after calling commitAsync? Should I precalculate offsets for > > multiple commits for every topic/partition to minimize number of offsets? > > > > Thanks! > > >
