[
https://issues.apache.org/jira/browse/KAFKA-21276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126998#comment-18126998
]
Eric Chang commented on KAFKA-21276:
------------------------------------
Logs from the two reproductions in the description, on trunk at commit
4a8164b8. In both runs the standby instance runs in its own JVM with {{-Xmx192m
-XX:+ExitOnOutOfMemoryError}}, and the active task processes about 1,300
records per second.
Attached:
* {{standby-trunk-oom-1-partition-1kb.log}}: one input partition, 300,000
records of 1 KB.
* {{standby-trunk-oom-8-partitions-50kb.log}}: eight input partitions, 12,000
records of 50 KB.
* {{CdcStandbyOomTest.java}}: the test that produced the first run. It uses
{{EmbeddedKafkaCluster}} and starts the standby with {{ProcessBuilder}}, so it
can be dropped into {{streams/integration-tests}}. It is a measurement, not a
test for the PR.
First run, the standby instance sampled every 500 ms. {{fetched}} is
{{records-consumed-total}} of its restore consumer, {{applied}} is the sum of
{{batchSize}} from {{StandbyUpdateListener#onBatchLoaded}}:
{noformat}
t_ms,fetched,applied,waiting,heap_used_mb,heap_max_mb,active_tasks,standby_tasks
3027,0,0,0,47,192,0,0
3528,17085,7620,9465,78,192,0,1
4029,99495,7620,91875,163,192,0,1
4531,148740,10050,138690,183,192,0,1
{noformat}
End of the standby's log:
{noformat}
[2026-10-14 13:22:58,057] INFO [Consumer
clientId=app-cdc-oom-...-StateUpdater-1-restore-consumer, groupId=null]
Assigned to partition(s): cdc-oom-0
(org.apache.kafka.clients.consumer.internals.ClassicKafkaConsumer:647)
...
Terminating due to java.lang.OutOfMemoryError: Java heap space
{noformat}
With the fix from the PR, the same run finished all 300,000 records in about
186 seconds, and the standby's heap usage stayed at or below 67 MB.
> Standby task with source-topic changelog fetches and buffers records beyond
> the committed-offset limit without bound
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21276
> URL: https://issues.apache.org/jira/browse/KAFKA-21276
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.1
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
> Attachments: CdcStandbyOomTest.java,
> standby-trunk-oom-1-partition-1kb.log, standby-trunk-oom-8-partitions-50kb.log
>
>
> A standby task whose state store uses the input topic as its changelog may
> only apply records up to the offset committed by the active task. This
> happens for {{builder.table(topic)}} when {{topology.optimization}} is set to
> {{all}} or {{reuse.ktable.source.topics}}; without this optimization the
> store has its own changelog topic and is not affected.
> {{StoreChangelogReader}} keeps fetching beyond the committed offset, and the
> records that cannot be applied yet stay in
> {{ChangelogMetadata#bufferedRecords}}. Nothing pauses the partition, so the
> buffer grows with every poll.
> When the active task is far behind the end of the input topic, for example
> when the application starts on a topic that already holds a lot of data, the
> standby buffers almost the whole backlog and can run out of memory.
> Before 2.5.0, {{StreamThread}} paused a standby partition when
> {{StandbyTask#update}} returned records beyond the offset limit, and resumed
> it after they were applied. This behavior was lost in the refactoring of
> KAFKA-9113, so 2.5.0 and later versions are affected.
> *Reproduction*
> Setup: one broker, two application instances, {{num.standby.replicas=1}},
> {{topology.optimization=all}}, a topology with {{builder.table(inputTopic)}},
> an active task slowed down by a sleep in a downstream processor, and the
> instance with the standby task in its own JVM with {{-Xmx192m}}.
> * One input partition, 300,000 records of 1 KB: the standby fails with
> {{OutOfMemoryError: Java heap space}} after about 5 seconds, with about
> 139,000 records in the restore buffer.
> * Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby
> fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with
> about 2,250 records (about 113 MB) in the restore buffers of its standby
> partitions.
> The same behavior can be shown without an OOM. With
> {{restore.consumer.max.poll.records=100}} and {{commit.interval.ms=100}}, the
> active instance is paused with {{KafkaStreams#pause()}} (so the committed
> offset does not move) and 10,000 new records are written to the input topic.
> Within a few seconds the restore consumer of the standby instance fetched
> 8,703 of them ({{records-consumed-total}}), although it could not apply any
> of them.
> Other cases do not grow: a standby with its own changelog topic, and an
> active task that is restoring (records that can be applied are applied right
> after they are fetched).
> *Proposed fix*
> Pause such a changelog partition while its buffer holds records beyond the
> limit offset, and resume it once the limit has moved past them. This restores
> the behavior from before KAFKA-9113 and adds no new config or public API. The
> restore buffer then holds at most one poll of records per partition
> ({{max.poll.records}}), and the restore consumer keeps at most one fetch per
> partition (about {{max.partition.fetch.bytes}}) until the partition is
> resumed.
> The cost: when the committed offset of the active task moves, the standby
> applies the records it already has and then needs one more fetch for the rest.
> Related: KAFKA-19992 / KIP-1325 proposes a configurable limit for the restore
> buffer. The two changes are independent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)