[ 
https://issues.apache.org/jira/browse/KAFKA-21276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126998#comment-18126998
 ] 

Eric Chang commented on KAFKA-21276:
------------------------------------

Logs from the two reproductions in the description, on trunk at commit 
4a8164b8. In both runs the standby instance runs in its own JVM with {{-Xmx192m 
-XX:+ExitOnOutOfMemoryError}}, and the active task processes about 1,300 
records per second.

Attached:
* {{standby-trunk-oom-1-partition-1kb.log}}: one input partition, 300,000 
records of 1 KB.
* {{standby-trunk-oom-8-partitions-50kb.log}}: eight input partitions, 12,000 
records of 50 KB.
* {{CdcStandbyOomTest.java}}: the test that produced the first run. It uses 
{{EmbeddedKafkaCluster}} and starts the standby with {{ProcessBuilder}}, so it 
can be dropped into {{streams/integration-tests}}. It is a measurement, not a 
test for the PR.

First run, the standby instance sampled every 500 ms. {{fetched}} is 
{{records-consumed-total}} of its restore consumer, {{applied}} is the sum of 
{{batchSize}} from {{StandbyUpdateListener#onBatchLoaded}}:
{noformat}
t_ms,fetched,applied,waiting,heap_used_mb,heap_max_mb,active_tasks,standby_tasks
3027,0,0,0,47,192,0,0
3528,17085,7620,9465,78,192,0,1
4029,99495,7620,91875,163,192,0,1
4531,148740,10050,138690,183,192,0,1
{noformat}

End of the standby's log:
{noformat}
[2026-10-14 13:22:58,057] INFO [Consumer 
clientId=app-cdc-oom-...-StateUpdater-1-restore-consumer, groupId=null] 
Assigned to partition(s): cdc-oom-0 
(org.apache.kafka.clients.consumer.internals.ClassicKafkaConsumer:647)
...
Terminating due to java.lang.OutOfMemoryError: Java heap space
{noformat}

With the fix from the PR, the same run finished all 300,000 records in about 
186 seconds, and the standby's heap usage stayed at or below 67 MB.

> Standby task with source-topic changelog fetches and buffers records beyond 
> the committed-offset limit without bound
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21276
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21276
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.3.1
>            Reporter: Eric Chang
>            Assignee: Eric Chang
>            Priority: Major
>         Attachments: CdcStandbyOomTest.java, 
> standby-trunk-oom-1-partition-1kb.log, standby-trunk-oom-8-partitions-50kb.log
>
>
> A standby task whose state store uses the input topic as its changelog may 
> only apply records up to the offset committed by the active task. This 
> happens for {{builder.table(topic)}} when {{topology.optimization}} is set to 
> {{all}} or {{reuse.ktable.source.topics}}; without this optimization the 
> store has its own changelog topic and is not affected.
> {{StoreChangelogReader}} keeps fetching beyond the committed offset, and the 
> records that cannot be applied yet stay in 
> {{ChangelogMetadata#bufferedRecords}}. Nothing pauses the partition, so the 
> buffer grows with every poll.
> When the active task is far behind the end of the input topic, for example 
> when the application starts on a topic that already holds a lot of data, the 
> standby buffers almost the whole backlog and can run out of memory.
> Before 2.5.0, {{StreamThread}} paused a standby partition when 
> {{StandbyTask#update}} returned records beyond the offset limit, and resumed 
> it after they were applied. This behavior was lost in the refactoring of 
> KAFKA-9113, so 2.5.0 and later versions are affected.
> *Reproduction*
> Setup: one broker, two application instances, {{num.standby.replicas=1}}, 
> {{topology.optimization=all}}, a topology with {{builder.table(inputTopic)}}, 
> an active task slowed down by a sleep in a downstream processor, and the 
> instance with the standby task in its own JVM with {{-Xmx192m}}.
> * One input partition, 300,000 records of 1 KB: the standby fails with 
> {{OutOfMemoryError: Java heap space}} after about 5 seconds, with about 
> 139,000 records in the restore buffer.
> * Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby 
> fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with 
> about 2,250 records (about 113 MB) in the restore buffers of its standby 
> partitions.
> The same behavior can be shown without an OOM. With 
> {{restore.consumer.max.poll.records=100}} and {{commit.interval.ms=100}}, the 
> active instance is paused with {{KafkaStreams#pause()}} (so the committed 
> offset does not move) and 10,000 new records are written to the input topic. 
> Within a few seconds the restore consumer of the standby instance fetched 
> 8,703 of them ({{records-consumed-total}}), although it could not apply any 
> of them.
> Other cases do not grow: a standby with its own changelog topic, and an 
> active task that is restoring (records that can be applied are applied right 
> after they are fetched).
> *Proposed fix*
> Pause such a changelog partition while its buffer holds records beyond the 
> limit offset, and resume it once the limit has moved past them. This restores 
> the behavior from before KAFKA-9113 and adds no new config or public API. The 
> restore buffer then holds at most one poll of records per partition 
> ({{max.poll.records}}), and the restore consumer keeps at most one fetch per 
> partition (about {{max.partition.fetch.bytes}}) until the partition is 
> resumed.
> The cost: when the committed offset of the active task moves, the standby 
> applies the records it already has and then needs one more fetch for the rest.
> Related: KAFKA-19992 / KIP-1325 proposes a configurable limit for the restore 
> buffer. The two changes are independent.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to