[
https://issues.apache.org/jira/browse/KAFKA-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justine Olshan updated KAFKA-18227:
-----------------------------------
Description:
With https://issues.apache.org/jira/browse/KAFKA-14563 there are two cases:
either a transaction is already added to the transaction and we continue
producing/committing offsets, or if it is not we add it with an interbroker
call. With TV2, we can tell if a transaction was added in this or a previous
transaction by checking the epoch. However, with the upgrade path, the epoch
check doesn't work because the first TV2 transaction will have the same epoch
as the last TV0 transaction.
If a transaction is in PrepareAbort/PrepareCommit, the producer can return a
EndTxn response and is free to start the next transaction. It is possible that
this new transaction will produce and commit offsets without informing the
coordinator if the partitions are the same as the last transaction. Then on
EndTxn, we can get fenced.
In other scenarios (like there are new partitions)we may also see
InvalidStateTransitions.
Upgrades should not require producer restarts or lose guarantees so we should
fix this.
was:
With https://issues.apache.org/jira/browse/KAFKA-14563 there are two cases:
either a transaction is already added to the transaction and we continue
producing/committing offsets, or if it is not we add it with an interbroker
call. With TV2, we can tell if a transaction was added in this or a previous
transaction by checking the epoch. However, with the upgrade path, the epoch
check doesn't work because the first TV2 transaction will have the same epoch
as the last TV0 transaction.
If a transaction is in PrepareAbort/PrepareComplete, the producer will get
EndTxn response and is free to start the next transaction. It is possible that
this new transaction will produce and commit offsets without informing the
coordinator if the partitions are the same as the last transaction. Then on
EndTxn, we can get fenced.
In other scenarios (like there are new partitions)we may also see
InvalidStateTransitions.
Upgrades should not require producer restarts or lose guarantees so we should
fix this.
> Ensure v2 partitions are not added to last transaction during upgrade
> ---------------------------------------------------------------------
>
> Key: KAFKA-18227
> URL: https://issues.apache.org/jira/browse/KAFKA-18227
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Justine Olshan
> Priority: Critical
>
> With https://issues.apache.org/jira/browse/KAFKA-14563 there are two cases:
> either a transaction is already added to the transaction and we continue
> producing/committing offsets, or if it is not we add it with an interbroker
> call. With TV2, we can tell if a transaction was added in this or a previous
> transaction by checking the epoch. However, with the upgrade path, the epoch
> check doesn't work because the first TV2 transaction will have the same epoch
> as the last TV0 transaction.
> If a transaction is in PrepareAbort/PrepareCommit, the producer can return a
> EndTxn response and is free to start the next transaction. It is possible
> that this new transaction will produce and commit offsets without informing
> the coordinator if the partitions are the same as the last transaction. Then
> on EndTxn, we can get fenced.
> In other scenarios (like there are new partitions)we may also see
> InvalidStateTransitions.
> Upgrades should not require producer restarts or lose guarantees so we should
> fix this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
