Eric Chang created KAFKA-21589:
----------------------------------

             Summary: Consumer fetch metrics are recorded more than once when a 
fetched partition fails to initialize
                 Key: KAFKA-21589
                 URL: https://issues.apache.org/jira/browse/KAFKA-21589
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 4.4.0, 4.3.0, 4.2.0, 4.1.0, 4.0.0
            Reporter: Eric Chang
            Assignee: Eric Chang


{{FetchCollector.initialize()}} reports a partition to 
{{FetchMetricsAggregator}} in a {{finally}} block, also when it throws. When 
{{initialize()}} throws, the {{CompletedFetch}} can stay in the {{FetchBuffer}}:

* If the same {{collectFetch()}} call already returned records for another 
partition, the exception is not thrown and the {{CompletedFetch}} stays in the 
buffer.
* If the {{CompletedFetch}} still has bytes, it is not removed even when the 
exception is thrown.

The {{CompletedFetch}} is then reported again by the next {{poll()}}, which 
calls {{initialize()}} again, or by {{drain()}} when the partition is no longer 
assigned ({{FetchBuffer.retainAll}}).

{{FetchMetricsAggregator}} records the metrics of the whole fetch response when 
every partition of the response has reported. After that, each extra report 
records them again. So, at the client level and the topic level:

* {{records-consumed-total}}, {{records-consumed-rate}}, 
{{bytes-consumed-total}} and {{bytes-consumed-rate}} are too high.
* {{fetch-size-avg}} and {{records-per-request-avg}} count that response more 
than once.

Example: one fetch response has 10 records for partition A and 
{{TOPIC_AUTHORIZATION_FAILED}} for partition B. The first {{poll()}} returns 
the 10 records of A and keeps B in the buffer. The second {{poll()}} throws 
{{TopicAuthorizationException}}. After the second {{poll()}}, 
{{records-consumed-total}} is 20 instead of 10.

Other errors that can do the same: {{OFFSET_OUT_OF_RANGE}} without an offset 
reset policy, and a non-empty partition with no complete record batch.

The same behavior exists in older releases, in {{Fetcher.java}} (checked back 
to 2.0.0).

Proposed fix: {{FetchMetricsAggregator.record()}} ignores a partition that has 
already reported.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to