[
https://issues.apache.org/jira/browse/KAFKA-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-21230:
---------------------------------------
Assignee: Nicholas Telford
> Long rebalances can cause OOM errors with transactional state stores enabled
> ----------------------------------------------------------------------------
>
> Key: KAFKA-21230
> URL: https://issues.apache.org/jira/browse/KAFKA-21230
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Nicholas Telford
> Assignee: Nicholas Telford
> Priority: Major
>
> When transactional state stores (KIP-892) are enabled, we loosely bound our
> transaction buffers using the new {{statestore.uncommitted.max.bytes}}, which
> is checked in {{StreamThread#maybeCommit()}}.
> However, {{TaskManager.commit()}} does not guarantee that it commits: if the
> thread is currently in the middle of a rebalance, *commits are skipped until
> it completes*. When this happens, processing continues without a commit until
> the rebalance has completed.
> Long rebalances can therefore cause threads to exceed this limit by a large
> degree, potentially causing OOM errors (native memory, most likely).
> I do not think this is a blocker for 4.4.0, because it only affects
> applications when transactional state stores have been explicitly enabled;
> and only under very specific circumstances.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
