mjsax commented on PR #20254: URL: https://github.com/apache/kafka/pull/20254#issuecomment-3368526760
Thanks @junrao! > Does KStreams depend on Producer.flush() failing? It seems that KStreams needs to check the future returned in the send() call to know if the record is sent successfully. We do not depend on `flush` to fail, but to block. We do check the send `Callback` (not the `Future`, but it's effectively the same thing, right?) after `flush` returned, but if `flush` does not block, the callbacks are not executed yet, and thus our check if any callback failed does not detect the error. > Do you have a test that can reproduce this? This PR add a test which reproduces the issue. > This is indeed a bug in the producer. ProducerBatch has a produceFuture of type ProduceRequestResult and a list of Thunk, one for each produced record. Each Thunk has a future of type FutureRecordMetadata. When we split a ProducerBatch, we have the logic to chain Thunk.future to the newly split batches. This makes sure that we don't unblock the producer.send() prematurely. But the problem is that flush() doesn't wait on Thunk.future. Instead, it waits on ProducerBatch.produceFuture. After splitting, we immediately call ProducerBatch.produceFuture.done(). This will unblock the flush() call prematurely since the splitted batches haven't been completed. Thanks for confirming. That matches my understanding. Thanks for proposing a fix. I don't know the producer code well enough to propose one :) > As for why the splitting loops forever, this is probably because of a recently fixed bug in https://github.com/apache/kafka/pull/20358/files. Interesting. If we make `flush()` block as it should, and not let is exit early, we should hit `max.block.ms` inside `flush()` and it would throw a `TimeoutException` back into KS code? KS should handle this (or at least just crash). We can look into this as a follow up how a `TimeoutException` is handled for this case, and if KS does the right thing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
