flashmouse opened a new issue, #31531: URL: https://github.com/apache/doris/issues/31531
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description for the sake of better data persistance and system availability , a aws kafka cluster may deploy brokers into mutiple regions, and each replica of the partitions would deploy into different regions. since netflow cross different regions in aws would incurs traffic charge(this expense often much more expensive than ec instance), kafka implement "[Rack-aware Partition Assignment for Kafka Consumers](https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers)" and "[Rack aware replica assignment](https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment)" so that kafka consumer could fetch partition data from the broker that it has the same rack with consumer. we are now trying to store log data from kafka into doris by routine load. since both kafka and doris are deployed in aws ec instances with 3 regions, we hope to decline the cross-netflow to minimum degree. so we need 2 functions as described below: 1. BE need add config (ex. called region), then BE should only fetch partitions from kafka brokers from the same region. so kafka-BE has no cross-netflow. 2. when BE table bucket by random and load_to_single_tablet=true, data fetched from kafka should all store into the tablet that hold by this BE itself,so BE-BE has no cross-netflow. I have searched issue seems nobody talk this function now. BE configuration seems not have such configuration like "region", and in the source code, when load_to_single_tablet=true, BE choose tablet by round-robin. so for now kafka-BE has cross-netflow, BE fetched data and BE store data may not the same BE so BE-BE has cross-netflow, am I right? and we want to ensure whether these functions above could implement? thx! ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
