Lianet Magrans created KAFKA-19804:
--------------------------------------
Summary: Improve heartbeat request manager initial HB interval
Key: KAFKA-19804
URL: https://issues.apache.org/jira/browse/KAFKA-19804
Project: Kafka
Issue Type: Task
Components: clients, consumer
Reporter: Lianet Magrans
Fix For: 4.2.0
With KIP-848, consumer HB interval config moved to the broker, so currently,
the consumer HB request manager starts with a 0ms interval (mainly to send a
first HB right away after the consumer subscribe + poll). Once a response is
received, the consumer takes the interval from the response and starts using
it.
That 0ms initial interval makes that the HB mgr poll continuously executes
logic on a tight loop that may not really be needed. It mostly has to wait for
a response (or a failure).
Probably worse that this, is the impact on the app thread, given that
pollTimeout takes into account the maxTimeToWait from the network thread, that
is directly impacted by the timeToNextHeartbeat
*
[https://github.com/apache/kafka/blob/388739f5d847d7a16e389d9891f806547f023476/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1764-L1766]
*
[https://github.com/apache/kafka/blob/781bc7a54b8c4f7c86f0d6bb9ef8399d86d0735e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java#L255]
We should review and consider setting a non-zero initial interval (while we
wait for the actual interval from the broker). One option to consider would be
using the request timeout maybe (just a first thought)
High level goals here would be to:
* maintain the behaviour of sending a first HB without delay
* ensure no unneeded activity on the HB mgr poll in the background, in tight
loop, while we're just waiting for the first HB response with an interval
* ensure the app thread poll timeout is not affected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
