Nicholas Telford created KAFKA-21230:
----------------------------------------
Summary: Long rebalances can cause OOM errors with transactional
state stores enabled
Key: KAFKA-21230
URL: https://issues.apache.org/jira/browse/KAFKA-21230
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Nicholas Telford
When transactional state stores (KIP-892) are enabled, we loosely bound our
transaction buffers using the new {{statestore.uncommitted.max.bytes}}, which
is checked in {{StreamThread#maybeCommit()}}.
However, {{TaskManager.commit()}} does not guarantee that it commits: if the
thread is currently in the middle of a rebalance, *commits are skipped until it
completes*. When this happens, processing continues without a commit until the
rebalance has completed.
Long rebalances can therefore cause threads to exceed this limit by a large
degree, potentially causing OOM errors (native memory, most likely).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
