Justine Olshan created KAFKA-21551:
--------------------------------------
Summary: rebuildProducerState skips log replay after truncation
when the producer state was empty, leaving completed transactions stuck open
(LSO never advances)
Key: KAFKA-21551
URL: https://issues.apache.org/jira/browse/KAFKA-21551
Project: Kafka
Issue Type: Bug
Reporter: Justine Olshan
h3. Problem
When a replica truncates its log (e.g. after a leader change),
{{UnifiedLog.rebuildProducerState}} reloads the latest producer snapshot at or
below the truncation offset and is then expected to replay the log from the
snapshot offset up to the truncation point. There is an optimization that skips
this replay when the producer state was empty before truncation:
{code:java}
boolean isEmptyBeforeTruncation = producerStateManager.isEmpty() &&
producerStateManager.mapEndOffset() >= lastOffset;
producerStateManager.truncateAndReload(logStartOffset, lastOffset,
time.milliseconds());
...
// Only do the potentially expensive reloading if the last snapshot offset is
lower than the log end
// offset (which would be the case on first startup) and there were active
producers prior to truncation
// (which could be the case if truncating after initial loading). If there
weren't, then truncating
// shouldn't change that fact ...
if (lastOffset > producerStateManager.mapEndOffset() &&
!isEmptyBeforeTruncation) {
loadProducersFromLog(...);
}{code}
The assumption that "empty before truncation => empty after truncation" does
not hold. The snapshot being reloaded can predate the end of a transaction, so
it can contain a producer with an ongoing transaction
({{{}currentTxnFirstOffset {}}}set). Producers with an ongoing transaction are
exempt from producer ID expiration ({{{}isProducerExpired{}}}), so
{{truncateAndReload}} restores that transaction as open. Because the replay is
skipped, the COMMIT/ABORT marker that ended the transaction (which sits between
the snapshot offset and the truncation point) is never applied. The transaction
remains open indefinitely, the LSO is pinned at its first offset, and
read_committed consumers on the partition make no progress.
h3. Scenario
# Transaction X for producer P starts in segment 1 and is completed by the
coordinator (marker written) in segment 2. The producer snapshot taken at the
roll of segment 1 records X as ongoing.
# The partition then receives only non-idempotent writes (no producer ID) for
longer than {{producer.id.expiration.ms }}(default 24h). P expires and the
producer state becomes empty. No segment roll happens in this window, so no
newer snapshot is taken.
# An uncommitted record is appended to the end of segment 2, and a leader
change causes this replica to truncate it. The truncation point is after X's
marker
# {{{}rebuildProducerState{}}}: {{{}isEmptyBeforeTruncation == true{}}}.
{{truncateAndReload}} loads the snapshot from the end of segment 1, which
restores X as open. The replay of segment 2 is skipped.
# X is never closed on this replica. The LSO is stuck until the transaction is
manually aborted. A manual abort is also wrong here: the replica records the
abort as spanning from X's original first offset, so records that were already
committed are hidden from read_committed fetches served by this replica.
Note: the truncated record does not need to be a transaction marker. Any
truncation of an uncommitted tail record is enough.
h3. Conditions (all required)
* After the partition's last transaction, only non-idempotent writes arrive
for longer than {{{}producer.id.expiration.ms{}}}, so the producer state is
empty.
* The replica truncates its log (e.g. after a leader change) before any new
idempotent/transactional write or post-marker snapshot.
* The latest snapshot at or below the truncation point predates the end marker
of a transaction whose marker is below the truncation point.
A snapshot taken anywhere between the transaction's end marker and the
truncation point prevents the bug (e.g. a segment roll or a clean shutdown),
since that snapshot records the transaction as complete. With default configs
this protection is weak: {{segment.ms }}defaults to 7 days and time-based rolls
are only evaluated on append, while {{producer.id.expiration.ms}} defaults to
24 hours. So a low-volume partition can keep the marker's segment open for days
after its producer IDs expire. The bug needs a specific sequence, but default
configs leave a wide window for it, and when it hits it blocks read_committed
consumers indefinitely. It has been observed in production.
h3. Possible fixes
# Make the skip condition also check the state after reload, so the log is
replayed whenever the reloaded snapshot brought back any producer or ongoing
transaction:
{code:java}
if (lastOffset > producerStateManager.mapEndOffset()
&& (!isEmptyBeforeTruncation || !producerStateManager.isEmpty()))
{{code}
Expired producers without an ongoing transaction are already filtered out on
snapshot load, so a non-empty state after reload means there is something the
replay needs to reconcile.
# Remove the optimization and always replay from the snapshot offset. The
optimization targeted users not yet using idempotence/transactions. Idempotence
is now enabled by default in the producer, so the case it optimizes for is much
less common, and correctness is easier to reason about without it.
h3. Reproduction
Unit test outline (UnifiedLogTest):
# Append a transactional batch for producer P, roll the segment (snapshot
records P with an ongoing txn).
# Append a COMMIT marker for P, then non-idempotent records.
# Advance mock time past \{{producer.id.expiration.ms}} and run producer
expiration, then assert the producer state is empty.
# Append one more non-idempotent record and \{{truncateTo}} the offset before
it.
# Assert the LSO equals the high watermark and there is no ongoing transaction
for P. This currently fails: P's transaction is reported as open and the LSO is
stuck at its first offset.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)