[
https://issues.apache.org/jira/browse/KAFKA-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-20711.
-------------------------------------
Fix Version/s: 4.4.0
4.3.2
Resolution: Fixed
> Streams task restore-remaining-records metric never reaches 0
> -------------------------------------------------------------
>
> Key: KAFKA-20711
> URL: https://issues.apache.org/jira/browse/KAFKA-20711
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0
> Reporter: Nicholas Telford
> Assignee: Nicholas Telford
> Priority: Minor
> Fix For: 4.4.0, 4.3.2
>
>
> The Kafka Streams Task-level metric {{restore-remaining-records}} is intended
> to track the total number of records that still need to be restored.
> This metric is inaccurate, never actually dropping to 0 for fully restored
> tasks, and always showing values substantially higher than reality,
> especially under EOS.
> The root-cause is that the metric is initialized with a total number of
> records to restored derived as {{logEndOffset - committedOffset}}, using a
> READ_UNCOMMITTED consumer.
> This offset range naturally includes uncommitted records and transaction
> markers, as well as offsets that have been compacted away, in addition to the
> actual records to restore.
> When decrementing the metric during restore, we decrement by the actual
> number of (committed) records that were restored. Since this excludes
> uncommitted records and transaction markers, we will never decrement the
> metric by the total it was initialized with.
> I have a fix that I will raise a PR for.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
