[
https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610168#comment-17610168
]
Jeremy DeGroot commented on FLINK-29398:
----------------------------------------
[~renqs] Yes, I'd be happy to lead the design of this feature!
We currently run Flink 1.14 in production, and so we wanted to move from the
FlinkKafkaConsumer to the KafkaSource API and that's what our first POC of this
feature was based on. However we weren't happy with the performance and
behavior of the KafkaSource in the version we were running at the time (the
1.14.2 release, if I remember correctly) so we implemented our final version on
FlinkKafkaConsumer. I'd be happy to target the KafkaSource API for this.
> Utilize Rack Awareness in Flink Consumer
> ----------------------------------------
>
> Key: FLINK-29398
> URL: https://issues.apache.org/jira/browse/FLINK-29398
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Reporter: Jeremy DeGroot
> Priority: Major
>
> [KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams]
> was implemented some time ago in Kafka. This allows brokers and consumers to
> communicate about the rack (or AWS Availability Zone) they're located in.
> Reading from a local broker can save money in bandwidth and improve latency
> for your consumers.
> Flink Kafka consumers currently cannot easily rack awareness if they're
> deployed across multiple racks or availability zones, because they have no
> control over which rack the Task Manager they'll be assigned to may be in.
> This improvement proposes that a Kafka Consumer could be configured with a
> callback or Future that could be run when it's being configured on the task
> manager, that will set the appropriate value at runtime if a value is
> provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
