jong su park created KAFKA-21479:
------------------------------------
Summary: Stale joinPrepareTimer after WakeupException in
ConsumerCoordinator#onJoinPrepare causes immediate "joinPrepare timeout" on a
later rebalance
Key: KAFKA-21479
URL: https://issues.apache.org/jira/browse/KAFKA-21479
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 4.4.0, 3.9.2
Environment: kafka-clients 3.9.2 and 4.4.0, classic group protocol,
broker apache/kafka:3.7.0 (KRaft, single node), JDK 17
Reporter: jong su park
Attachments: StaleJoinPrepareTimer.java
With the classic group protocol and {{enable.auto.commit=true}}, if
{{consumer.wakeup()}} fires while {{ConsumerCoordinator#onJoinPrepare}} is
waiting for the pre-rebalance auto-commit
({{client.poll(autoCommitOffsetRequestFuture, pollTimer)}}), the
{{WakeupException}} leaves {{joinPrepareTimer}} and
{{autoCommitOffsetRequestFuture}} set.
{{AbstractCoordinator#joinGroupIfNeeded}} sets {{needsJoinPrepare = false}}
before calling {{onJoinPrepare}}, so on the next {{poll()}} the join continues
without re-entering {{onJoinPrepare}} and completes normally. Nothing resets
the timer (it is only reset at the end of a successful {{onJoinPrepare}}).
On the next rebalance that happens more than {{rebalanceTimeoutMs}}
({{max.poll.interval.ms}}) later, {{onJoinPrepare}} reuses the old timer, which
is already expired, and immediately logs:
{noformat}
ERROR ... Asynchronous auto-commit of offsets failed: joinPrepare timeout. Will
continue to join group
{noformat}
Because {{autoCommitOffsetRequestFuture}} is also stale (already completed), no
fresh commit is sent before the rebalance proceeds (based on code reading; not
measured), so records consumed since the last periodic auto-commit can be
redelivered after reassignment.
In practice {{wakeup()}} is called from other threads by frameworks, e.g.
Spring Kafka's {{enforceRebalance()}} / {{pause()}} / {{resume()}}. We see this
ERROR in production a few milliseconds after a rebalance starts, hours after
the previous one.
h3. Steps to reproduce
Attached {{StaleJoinPrepareTimer.java}} (single file, uses reflection only to
print internal state).
# Start a single broker on localhost:9092 (we used apache/kafka:3.7.0 in KRaft
mode).
# Compile and run with kafka-clients and an slf4j binding on the classpath:
{noformat}
javac -cp <cp> StaleJoinPrepareTimer.java && java -cp .:<cp>
StaleJoinPrepareTimer localhost:9092 10000
{noformat}
Consumer config: {{enable.auto.commit=true}}, {{auto.commit.interval.ms=500}},
{{max.poll.interval.ms=8000}}, {{CooperativeStickyAssignor}}.
A background thread calls {{consumer.wakeup()}} repeatedly while the main
thread loops {{enforceRebalance()}} + {{poll()}}. Within a few rounds one
{{WakeupException}} is thrown from {{onJoinPrepare}}, and after the join
completes {{joinPrepareTimer}} is still set.
# The program then stops the wakeups, polls for 10s (>
{{max.poll.interval.ms}}) and calls {{enforceRebalance()}} once more.
h3. Observed (3.9.2 and 4.4.0)
{noformat}
rounds=8 wakeupsInOnJoinPrepare=1 staleAfterCompletedJoin=true
[joinPrepareTimer=set(elapsedMs=302, expired=false),
autoCommitOffsetRequestFuture=set]
waited 10000ms (max.poll.interval.ms=8000):
[joinPrepareTimer=set(elapsedMs=10378, expired=true),
autoCommitOffsetRequestFuture=set]
next rebalance without any wakeup
ERROR ConsumerCoordinator - ... Asynchronous auto-commit of offsets failed:
joinPrepare timeout. Will continue to join group (1 ms later)
{noformat}
Control: same steps with waitMs=2000 (< {{max.poll.interval.ms}}): no ERROR,
the timer is reset by the next {{onJoinPrepare}}. Without the wakeup thread: no
ERROR.
h3. Expected
A {{WakeupException}} (or any exception) leaving {{onJoinPrepare}} should not
leave {{joinPrepareTimer}} / {{autoCommitOffsetRequestFuture}} behind, or
{{onJoinPrepare}} should be re-entered before the join continues.
h3. Possible fix (suggestion)
Reset {{joinPrepareTimer}} (and {{autoCommitOffsetRequestFuture}}) when
{{onJoinPrepare}} exits exceptionally, or restore {{needsJoinPrepare = true}}
when the exception is a {{WakeupException}} / {{InterruptException}} from the
commit wait.
Related: KAFKA-14024 (introduced joinPrepareTimer, PR #12349), KAFKA-13310,
KAFKA-14196, KAFKA-12983.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)