flashmouse commented on issue #31531: URL: https://github.com/apache/doris/issues/31531#issuecomment-1972897990
> Hi, @flashmouse would you like to submit this PR ? :) generally kafka rack-aware has 2 parts: 1. when client send fetch request to broker A, A know this partition has replica hold by broker B(B and client are in the same rack), so A tell client redirect this fetch request to Broker B (Fetch From Follower) 2. during kafka consumer rebalance, assignor only assign partitions to client when any replica's rack is the same with client(this one is the [confluentinc/librdkafka#4252](https://github.com/confluentinc/librdkafka/pull/4252) implement). for implement method 1, we need add a new parameter ("region") into BE configuration, and rdkafka client set `client.rack={region_value}`. I think this would be simple retaively. for implement method 2, since doris assign partitions itself, this may be more complex, the function `KafkaDataConsumerGroup::assign_topic_partitions` need do a refactor(ex. use assign logic from rdkafka_assignor) I can try implement method 1 first to familiar with pr flow. If you think this is viable I will try do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
