unknowntpo opened a new pull request, #23815:
URL: https://github.com/apache/kafka/pull/23815

   A standby task whose store uses the input topic as its changelog may only 
apply records up to the offset committed by the active task, but 
`StoreChangelogReader` kept fetching beyond that offset. Those records stayed 
in the restore buffer, which could grow until `OutOfMemoryError` when the 
active task was far behind the end of the topic.
   
   This PR pauses such a changelog partition while its buffer holds records 
beyond the limit offset, and resumes it once the limit has moved past them. 
This is how standby tasks worked before 2.5.0: `StreamThread` paused the 
partition when `StandbyTask#update` returned records beyond the limit, and this 
was lost in the refactoring of KAFKA-9113 (#7997). The change adds no new 
config or public API.
   
   The new integration test pauses the active task and produces 10,000 records: 
without the fix the standby polls all 10,000 records beyond the committed 
offset; with the fix it polls them once (at most `max.poll.records`) and then 
stops.
   
   Related to KAFKA-19992 / KIP-1325.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to