jong su park created KAFKA-21479:
------------------------------------

             Summary: Stale joinPrepareTimer after WakeupException in 
ConsumerCoordinator#onJoinPrepare causes immediate "joinPrepare timeout" on a 
later rebalance
                 Key: KAFKA-21479
                 URL: https://issues.apache.org/jira/browse/KAFKA-21479
             Project: Kafka
          Issue Type: Bug
          Components: clients
    Affects Versions: 4.4.0, 3.9.2
         Environment: kafka-clients 3.9.2 and 4.4.0, classic group protocol, 
broker apache/kafka:3.7.0 (KRaft, single node), JDK 17
            Reporter: jong su park
         Attachments: StaleJoinPrepareTimer.java

With the classic group protocol and {{enable.auto.commit=true}}, if 
{{consumer.wakeup()}} fires while {{ConsumerCoordinator#onJoinPrepare}} is 
waiting for the pre-rebalance auto-commit 
({{client.poll(autoCommitOffsetRequestFuture, pollTimer)}}), the 
{{WakeupException}} leaves {{joinPrepareTimer}} and 
{{autoCommitOffsetRequestFuture}} set.

{{AbstractCoordinator#joinGroupIfNeeded}} sets {{needsJoinPrepare = false}} 
before calling {{onJoinPrepare}}, so on the next {{poll()}} the join continues 
without re-entering {{onJoinPrepare}} and completes normally. Nothing resets 
the timer (it is only reset at the end of a successful {{onJoinPrepare}}).

On the next rebalance that happens more than {{rebalanceTimeoutMs}} 
({{max.poll.interval.ms}}) later, {{onJoinPrepare}} reuses the old timer, which 
is already expired, and immediately logs:
{noformat}
ERROR ... Asynchronous auto-commit of offsets failed: joinPrepare timeout. Will 
continue to join group
{noformat}
Because {{autoCommitOffsetRequestFuture}} is also stale (already completed), no 
fresh commit is sent before the rebalance proceeds (based on code reading; not 
measured), so records consumed since the last periodic auto-commit can be 
redelivered after reassignment.

In practice {{wakeup()}} is called from other threads by frameworks, e.g. 
Spring Kafka's {{enforceRebalance()}} / {{pause()}} / {{resume()}}. We see this 
ERROR in production a few milliseconds after a rebalance starts, hours after 
the previous one.

h3. Steps to reproduce
Attached {{StaleJoinPrepareTimer.java}} (single file, uses reflection only to 
print internal state).
# Start a single broker on localhost:9092 (we used apache/kafka:3.7.0 in KRaft 
mode).
# Compile and run with kafka-clients and an slf4j binding on the classpath:
{noformat}
javac -cp <cp> StaleJoinPrepareTimer.java && java -cp .:<cp> 
StaleJoinPrepareTimer localhost:9092 10000
{noformat}
Consumer config: {{enable.auto.commit=true}}, {{auto.commit.interval.ms=500}}, 
{{max.poll.interval.ms=8000}}, {{CooperativeStickyAssignor}}.
A background thread calls {{consumer.wakeup()}} repeatedly while the main 
thread loops {{enforceRebalance()}} + {{poll()}}. Within a few rounds one 
{{WakeupException}} is thrown from {{onJoinPrepare}}, and after the join 
completes {{joinPrepareTimer}} is still set.
# The program then stops the wakeups, polls for 10s (> 
{{max.poll.interval.ms}}) and calls {{enforceRebalance()}} once more.

h3. Observed (3.9.2 and 4.4.0)
{noformat}
rounds=8 wakeupsInOnJoinPrepare=1 staleAfterCompletedJoin=true 
[joinPrepareTimer=set(elapsedMs=302, expired=false), 
autoCommitOffsetRequestFuture=set]
waited 10000ms (max.poll.interval.ms=8000): 
[joinPrepareTimer=set(elapsedMs=10378, expired=true), 
autoCommitOffsetRequestFuture=set]
next rebalance without any wakeup
ERROR ConsumerCoordinator - ... Asynchronous auto-commit of offsets failed: 
joinPrepare timeout. Will continue to join group   (1 ms later)
{noformat}
Control: same steps with waitMs=2000 (< {{max.poll.interval.ms}}): no ERROR, 
the timer is reset by the next {{onJoinPrepare}}. Without the wakeup thread: no 
ERROR.

h3. Expected
A {{WakeupException}} (or any exception) leaving {{onJoinPrepare}} should not 
leave {{joinPrepareTimer}} / {{autoCommitOffsetRequestFuture}} behind, or 
{{onJoinPrepare}} should be re-entered before the join continues.

h3. Possible fix (suggestion)
Reset {{joinPrepareTimer}} (and {{autoCommitOffsetRequestFuture}}) when 
{{onJoinPrepare}} exits exceptionally, or restore {{needsJoinPrepare = true}} 
when the exception is a {{WakeupException}} / {{InterruptException}} from the 
commit wait.

Related: KAFKA-14024 (introduced joinPrepareTimer, PR #12349), KAFKA-13310, 
KAFKA-14196, KAFKA-12983.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to