[
https://issues.apache.org/jira/browse/FLINK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986135#comment-15986135
]
ASF GitHub Bot commented on FLINK-6288:
---------------------------------------
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3766
One thing to be careful with, though:
Since now we're querying Kafka for partition metadata within the `invoke`
method, the query must be handled robustly and make sure it doesn't result in
unexpectedly longer checkpoint times by blocking the whole stream at the Kafka
sink.
Most notably, we need to consider the corner case where Kafka isn't
cooperating nicely:
1. how to handle arbitrary long response time for fetching the partition
metadata?
2. how to handle the case where, due to some Kafka brokers temporary
unavailable, the returned partitions is not complete?
For 2., I can also forsee that we have a separate "partitions update
thread" that refreshes the `Map<String, int[]>` cache continuously at a fixed
interval. This can also involve to a `FlinkKafkaPartitioner` that can provide
dynamically changing `int[] partitions` when invoking the `partition` method.
Perhaps we shouldn't include that with this PR, as its orthogonal to the
API change. Just some food for though :)
> FlinkKafkaProducer's custom Partitioner is always invoked with number of
> partitions of default topic
> ----------------------------------------------------------------------------------------------------
>
> Key: FLINK-6288
> URL: https://issues.apache.org/jira/browse/FLINK-6288
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Fang Yong
>
> The {{FlinkKafkaProducerBase}} supports routing records to topics besides the
> default topic, but the custom {{Partitioner}} interface does not follow this
> semantic.
> The partitioner is always invoked the {{partition}} method with the number of
> partitions in the default topic, and not the number of partitions of the
> current {{targetTopic}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
