[ 
https://issues.apache.org/jira/browse/KAFKA-21276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Chang updated KAFKA-21276:
-------------------------------
    Description: 
A standby task whose state store uses the input topic as its changelog may only 
apply records up to the offset committed by the active task. This happens for 
{{builder.table(topic)}} when {{topology.optimization}} is set to {{all}} or 
{{reuse.ktable.source.topics}}; without this optimization the store has its own 
changelog topic and is not affected.

{{StoreChangelogReader}} keeps fetching beyond the committed offset, and the 
records that cannot be applied yet stay in 
{{ChangelogMetadata#bufferedRecords}}. Nothing pauses the partition, so the 
buffer grows with every poll.

When the active task is far behind the end of the input topic, for example when 
the application starts on a topic that already holds a lot of data, the standby 
buffers almost the whole backlog and can run out of memory.

Before 2.5.0, {{StreamThread}} paused a standby partition when 
{{StandbyTask#update}} returned records beyond the offset limit, and resumed it 
after they were applied. This behavior was lost in the refactoring of 
KAFKA-9113, so 2.5.0 and later versions are affected.

*Reproduction*
Setup: one broker, two application instances, {{num.standby.replicas=1}}, 
{{topology.optimization=all}}, a topology with {{builder.table(inputTopic)}}, 
an active task slowed down by a sleep in a downstream processor, and the 
instance with the standby task in its own JVM with {{-Xmx192m}}.
* One input partition, 300,000 records of 1 KB: the standby fails with 
{{OutOfMemoryError: Java heap space}} after about 5 seconds, with about 139,000 
records in the restore buffer.
* Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby 
fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with 
about 2,250 records (about 113 MB) in the restore buffers of its standby 
partitions.

The same behavior can be shown without an OOM. With 
{{restore.consumer.max.poll.records=100}} and {{commit.interval.ms=100}}, the 
active instance is paused with {{KafkaStreams#pause()}} (so the committed 
offset does not move) and 10,000 new records are written to the input topic. 
Within a few seconds the restore consumer of the standby instance fetched 8,703 
of them ({{records-consumed-total}}), although it could not apply any of them.

Other cases do not grow: a standby with its own changelog topic, and an active 
task that is restoring (records that can be applied are applied right after 
they are fetched).

*Proposed fix*
Pause such a changelog partition while its buffer holds records beyond the 
limit offset, and resume it once the limit has moved past them, as before 
2.5.0. The restore buffer then holds at most one poll of records per partition 
({{max.poll.records}}), and the restore consumer keeps at most one fetch per 
partition (about {{max.partition.fetch.bytes}}) until the partition is resumed.

The cost: when the committed offset of the active task moves, the standby 
applies the records it already has and then needs one more fetch for the rest.

Related: KAFKA-19992 / KIP-1325 proposes a configurable limit for the restore 
buffer. The two changes are independent.

  was:
A standby task whose state store uses the input topic as its changelog (for 
example {{builder.table(topic)}} with {{topology.optimization=all}}) may only 
apply records up to the offset committed by the active task. 
{{StoreChangelogReader}} keeps fetching beyond that offset, and the records 
that cannot be applied yet stay in {{ChangelogMetadata#bufferedRecords}}. 
Nothing pauses the partition, so the buffer grows with every poll.

When the active task is far behind the end of the input topic, for example when 
the application starts on a topic that already holds a lot of data, the standby 
buffers almost the whole backlog and can run out of memory.

*Reproduction* (one broker, a slow active task, the standby in its own JVM with 
{{-Xmx192m}})
* One input partition, 300,000 records of 1 KB: the standby fails with 
{{OutOfMemoryError: Java heap space}} after about 5 seconds, with about 139,000 
records in the restore buffer.
* Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby 
fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with 
about 2,250 records (about 113 MB) in the restore buffers of its standby 
partitions.

The same behavior can be shown without an OOM: with the active task paused (so 
its committed offset does not move) and 10,000 new records in the input topic, 
the standby fetched 8,703 of them within a few seconds, although it could not 
apply any of them.

Other cases do not grow: a standby with its own changelog topic, and an active 
task that is restoring (records that can be applied are applied right after 
they are fetched).

*Proposed fix*
Pause such a changelog partition while its buffer holds records beyond the 
limit offset, and resume it once the limit has moved past them. The buffer then 
holds at most one fetch per partition ({{max.partition.fetch.bytes}}).

Related: KAFKA-19992 / KIP-1325, which proposes a configurable limit for the 
restore buffer.

        Summary: Standby task with source-topic changelog fetches and buffers 
records beyond the committed-offset limit without bound  (was: Standby tasks 
with a source-topic changelog buffer restore records without limit)

> Standby task with source-topic changelog fetches and buffers records beyond 
> the committed-offset limit without bound
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21276
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21276
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Eric Chang
>            Assignee: Eric Chang
>            Priority: Major
>
> A standby task whose state store uses the input topic as its changelog may 
> only apply records up to the offset committed by the active task. This 
> happens for {{builder.table(topic)}} when {{topology.optimization}} is set to 
> {{all}} or {{reuse.ktable.source.topics}}; without this optimization the 
> store has its own changelog topic and is not affected.
> {{StoreChangelogReader}} keeps fetching beyond the committed offset, and the 
> records that cannot be applied yet stay in 
> {{ChangelogMetadata#bufferedRecords}}. Nothing pauses the partition, so the 
> buffer grows with every poll.
> When the active task is far behind the end of the input topic, for example 
> when the application starts on a topic that already holds a lot of data, the 
> standby buffers almost the whole backlog and can run out of memory.
> Before 2.5.0, {{StreamThread}} paused a standby partition when 
> {{StandbyTask#update}} returned records beyond the offset limit, and resumed 
> it after they were applied. This behavior was lost in the refactoring of 
> KAFKA-9113, so 2.5.0 and later versions are affected.
> *Reproduction*
> Setup: one broker, two application instances, {{num.standby.replicas=1}}, 
> {{topology.optimization=all}}, a topology with {{builder.table(inputTopic)}}, 
> an active task slowed down by a sleep in a downstream processor, and the 
> instance with the standby task in its own JVM with {{-Xmx192m}}.
> * One input partition, 300,000 records of 1 KB: the standby fails with 
> {{OutOfMemoryError: Java heap space}} after about 5 seconds, with about 
> 139,000 records in the restore buffer.
> * Eight input partitions, 12,000 records of 50 KB (about 600 MB): the standby 
> fails with {{OutOfMemoryError: Java heap space}} after about 8 seconds, with 
> about 2,250 records (about 113 MB) in the restore buffers of its standby 
> partitions.
> The same behavior can be shown without an OOM. With 
> {{restore.consumer.max.poll.records=100}} and {{commit.interval.ms=100}}, the 
> active instance is paused with {{KafkaStreams#pause()}} (so the committed 
> offset does not move) and 10,000 new records are written to the input topic. 
> Within a few seconds the restore consumer of the standby instance fetched 
> 8,703 of them ({{records-consumed-total}}), although it could not apply any 
> of them.
> Other cases do not grow: a standby with its own changelog topic, and an 
> active task that is restoring (records that can be applied are applied right 
> after they are fetched).
> *Proposed fix*
> Pause such a changelog partition while its buffer holds records beyond the 
> limit offset, and resume it once the limit has moved past them, as before 
> 2.5.0. The restore buffer then holds at most one poll of records per 
> partition ({{max.poll.records}}), and the restore consumer keeps at most one 
> fetch per partition (about {{max.partition.fetch.bytes}}) until the partition 
> is resumed.
> The cost: when the committed offset of the active task moves, the standby 
> applies the records it already has and then needs one more fetch for the rest.
> Related: KAFKA-19992 / KIP-1325 proposes a configurable limit for the restore 
> buffer. The two changes are independent.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to