Justine Olshan created KAFKA-21551:
--------------------------------------

             Summary: rebuildProducerState skips log replay after truncation 
when the producer state was empty, leaving completed transactions stuck open 
(LSO never advances)
                 Key: KAFKA-21551
                 URL: https://issues.apache.org/jira/browse/KAFKA-21551
             Project: Kafka
          Issue Type: Bug
            Reporter: Justine Olshan


h3. Problem

When a replica truncates its log (e.g. after a leader change), 
{{UnifiedLog.rebuildProducerState}} reloads the latest producer snapshot at or 
below the truncation offset and is then expected to replay the log from the 
snapshot offset up to the truncation point. There is an optimization that skips 
this replay when the producer state was empty before truncation:
{code:java}
boolean isEmptyBeforeTruncation = producerStateManager.isEmpty() && 
producerStateManager.mapEndOffset() >= lastOffset;
producerStateManager.truncateAndReload(logStartOffset, lastOffset, 
time.milliseconds());
...
// Only do the potentially expensive reloading if the last snapshot offset is 
lower than the log end
// offset (which would be the case on first startup) and there were active 
producers prior to truncation
// (which could be the case if truncating after initial loading). If there 
weren't, then truncating
// shouldn't change that fact ...
if (lastOffset > producerStateManager.mapEndOffset() && 
!isEmptyBeforeTruncation) {
    loadProducersFromLog(...);
}{code}
The assumption that "empty before truncation => empty after truncation" does 
not hold. The snapshot being reloaded can predate the end of a transaction, so 
it can contain a producer with an ongoing transaction 
({{{}currentTxnFirstOffset {}}}set). Producers with an ongoing transaction are 
exempt from producer ID expiration ({{{}isProducerExpired{}}}), so 
{{truncateAndReload}} restores that transaction as open. Because the replay is 
skipped, the COMMIT/ABORT marker that ended the transaction (which sits between 
the snapshot offset and the truncation point) is never applied. The transaction 
remains open indefinitely, the LSO is pinned at its first offset, and 
read_committed consumers on the partition make no progress.
h3. Scenario
 # Transaction X for producer P starts in segment 1 and is completed by the 
coordinator (marker written) in segment 2. The producer snapshot taken at the 
roll of segment 1 records X as ongoing.
 # The partition then receives only non-idempotent writes (no producer ID) for 
longer than {{producer.id.expiration.ms }}(default 24h). P expires and the 
producer state becomes empty. No segment roll happens in this window, so no 
newer snapshot is taken.
 # An uncommitted record is appended to the end of segment 2, and a leader 
change causes this replica to truncate it. The truncation point is after X's 
marker
 # {{{}rebuildProducerState{}}}: {{{}isEmptyBeforeTruncation == true{}}}. 
{{truncateAndReload}} loads the snapshot from the end of segment 1, which 
restores X as open. The replay of segment 2 is skipped.
 # X is never closed on this replica. The LSO is stuck until the transaction is 
manually aborted. A manual abort is also wrong here: the replica records the 
abort as spanning from X's original first offset, so records that were already 
committed are hidden from read_committed fetches served by this replica.

Note: the truncated record does not need to be a transaction marker. Any 
truncation of an uncommitted tail record is enough.
h3. Conditions (all required)
 * After the partition's last transaction, only non-idempotent writes arrive 
for longer than {{{}producer.id.expiration.ms{}}}, so the producer state is 
empty.
 * The replica truncates its log (e.g. after a leader change) before any new 
idempotent/transactional write or post-marker snapshot.
 * The latest snapshot at or below the truncation point predates the end marker 
of a transaction whose marker is below the truncation point.

A snapshot taken anywhere between the transaction's end marker and the 
truncation point prevents the bug (e.g. a segment roll or a clean shutdown), 
since that snapshot records the transaction as complete. With default configs 
this protection is weak: {{segment.ms }}defaults to 7 days and time-based rolls 
are only evaluated on append, while {{producer.id.expiration.ms}} defaults to 
24 hours. So a low-volume partition can keep the marker's segment open for days 
after its producer IDs expire. The bug needs a specific sequence, but default 
configs leave a wide window for it, and when it hits it blocks read_committed 
consumers indefinitely. It has been observed in production.
h3. Possible fixes
 # Make the skip condition also check the state after reload, so the log is 
replayed whenever the reloaded snapshot brought back any producer or ongoing 
transaction:

{code:java}
if (lastOffset > producerStateManager.mapEndOffset()
        && (!isEmptyBeforeTruncation || !producerStateManager.isEmpty())) 
{{code}
Expired producers without an ongoing transaction are already filtered out on 
snapshot load, so a non-empty state after reload means there is something the 
replay needs to reconcile.
 # Remove the optimization and always replay from the snapshot offset. The 
optimization targeted users not yet using idempotence/transactions. Idempotence 
is now enabled by default in the producer, so the case it optimizes for is much 
less common, and correctness is easier to reason about without it.

h3. Reproduction

Unit test outline (UnifiedLogTest):
 # Append a transactional batch for producer P, roll the segment (snapshot 
records P with an ongoing txn).
 # Append a COMMIT marker for P, then non-idempotent records.
 # Advance mock time past \{{producer.id.expiration.ms}} and run producer 
expiration, then assert the producer state is empty.
 # Append one more non-idempotent record and \{{truncateTo}} the offset before 
it.
 # Assert the LSO equals the high watermark and there is no ongoing transaction 
for P. This currently fails: P's transaction is reported as open and the LSO is 
stuck at its first offset.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to