[
https://issues.apache.org/jira/browse/KAFKA-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Chen updated KAFKA-9878:
-------------------------------
Description:
Currently the EndTxn call from Producer will immediately return as the control
record is written to the txn coordinator log. The ongoing transaction will be
going to a pending state to wait for all txn markers to be propagated. In the
meantime, producer client will start another new transaction but being rejected
constantly until the pending state gets resolved, which is unnecessary round
trips and more burden to the broker to handle repetitive requests.
To avoid this situation, we should make the Producer client wait for txn marker
completion instead. This will incur better performance overall, as no more
back-off shall be triggered for a subsequent transaction to begin.
On the other hand, we could also batch complete the AddPartitionsToTxn results
if we buffered more than one request in the queue.
The third change is on the client side, which is to maintain the futures of the
AddPartitionsToTxn calls to make more inflight changes as necessary.
was:
Currently the EndTxn call from Producer will immediately return as the control
record is written to the txn coordinator log. The ongoing transaction will be
going to a pending state to wait for all txn markers to be propagated. In the
meantime, producer client will start another new transaction but being rejected
constantly until the pending state gets resolved, which is unnecessary round
trips and more burden to the broker to handle repetitive requests.
To avoid this situation, we should make the Producer client wait for txn marker
completion instead. This will incur better performance overall, as no more
back-off shall be triggered for a subsequent transaction to begin.
> Block AddPartitionsToTxn call until the txn markers are committed
> -----------------------------------------------------------------
>
> Key: KAFKA-9878
> URL: https://issues.apache.org/jira/browse/KAFKA-9878
> Project: Kafka
> Issue Type: Improvement
> Reporter: Boyang Chen
> Priority: Major
>
> Currently the EndTxn call from Producer will immediately return as the
> control record is written to the txn coordinator log. The ongoing transaction
> will be going to a pending state to wait for all txn markers to be
> propagated. In the meantime, producer client will start another new
> transaction but being rejected constantly until the pending state gets
> resolved, which is unnecessary round trips and more burden to the broker to
> handle repetitive requests.
> To avoid this situation, we should make the Producer client wait for txn
> marker completion instead. This will incur better performance overall, as no
> more back-off shall be triggered for a subsequent transaction to begin.
> On the other hand, we could also batch complete the AddPartitionsToTxn
> results if we buffered more than one request in the queue.
> The third change is on the client side, which is to maintain the futures of
> the AddPartitionsToTxn calls to make more inflight changes as necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
