Lincong Li created KAFKA-12854: ---------------------------------- Summary: Add a config to allow skipping metadata cache update when topic partition is unassigned Key: KAFKA-12854 URL: https://issues.apache.org/jira/browse/KAFKA-12854 Project: Kafka Issue Type: Improvement Components: clients Reporter: Lincong Li
The "assign" method in the consumer triggers a metadata cache update if the new partition assignment is different from the current assignment. It makes sense to update the MD cache if the new assignment contains partitions that do not exist in the current assignment. However, I wonder why is updating its MD cache necessary if the new partition assignment is a subset of the current assignment. For example, the new assignment is tp0, tp1, and the current assignment is tp0, tp1, tp2. The current behavior does too many drawbacks in most cases. However, if the number of consumer instances is large and each consumer instance is constantly getting some topic partitions unassigned, the QPS of the MD request sent out to update the MD cache becomes high as a result. Proposed changes: Add a config to allow skipping metadata cache update when topic partition(s) is unassigned Existing PR to a forked repo: https://github.com/linkedin/kafka/pull/166/files -- This message was sent by Atlassian Jira (v8.3.4#803005)