[
https://issues.apache.org/jira/browse/KAFKA-21276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-21276:
-------------------------------
Affects Version/s: 4.3.1
> Standby task with source-topic changelog fetches and buffers records beyond
> the committed-offset limit without bound
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21276
> URL: https://issues.apache.org/jira/browse/KAFKA-21276
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.1
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
>
> A standby task whose state store uses the input topic as its changelog may
> only apply records up to the offset committed by the active task. This
> happens for {{builder.table(topic)}} when {{topology.optimization}} is set to
> {{all}} or {{reuse.ktable.source.topics}}; without this optimization the
> store has its own changelog topic and is not affected.
> {{StoreChangelogReader}} keeps fetching beyond the committed offset, and the
> records that cannot be applied yet stay in
> {{ChangelogMetadata#bufferedRecords}}. Nothing pauses the partition, so the
> buffer grows with every poll.
> When the active task is far behind the end of the input topic, for example
> when the application starts on a topic that already holds a lot of data, the
> standby buffers almost the whole backlog and can run out of memory.
> Before 2.5.0, {{StreamThread}} paused a standby partition when
> {{StandbyTask#update}} returned records beyond the offset limit, and resumed
> it after they were applied. This behavior was lost in the refactoring of
> KAFKA-9113, so 2.5.0 and later versions are affected.
> *Reproduction*
> Setup: one broker, two application instances, {{num.standby.replicas=1}},
> {{topology.optimization=all}}, a topology with {{builder.table(inputTopic)}},
> an active task slowed down by a sleep in a downstream processor, and the
> instance with the standby task in its own JVM with {{-Xmx192m}}.
> * One input partition, 300,000 records of 1 KB: the standby fails with
> {{OutOfMemoryError: Java heap space}} after about 5 seconds, with about
> 139,000 records in the restore buffer.
> * Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby
> fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with
> about 2,250 records (about 113 MB) in the restore buffers of its standby
> partitions.
> The same behavior can be shown without an OOM. With
> {{restore.consumer.max.poll.records=100}} and {{commit.interval.ms=100}}, the
> active instance is paused with {{KafkaStreams#pause()}} (so the committed
> offset does not move) and 10,000 new records are written to the input topic.
> Within a few seconds the restore consumer of the standby instance fetched
> 8,703 of them ({{records-consumed-total}}), although it could not apply any
> of them.
> Other cases do not grow: a standby with its own changelog topic, and an
> active task that is restoring (records that can be applied are applied right
> after they are fetched).
> *Proposed fix*
> Pause such a changelog partition while its buffer holds records beyond the
> limit offset, and resume it once the limit has moved past them. This restores
> the behavior from before KAFKA-9113 and adds no new config or public API. The
> restore buffer then holds at most one poll of records per partition
> ({{max.poll.records}}), and the restore consumer keeps at most one fetch per
> partition (about {{max.partition.fetch.bytes}}) until the partition is
> resumed.
> The cost: when the committed offset of the active task moves, the standby
> applies the records it already has and then needs one more fetch for the rest.
> Related: KAFKA-19992 / KIP-1325 proposes a configurable limit for the restore
> buffer. The two changes are independent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)