becketqin commented on issue #8535: [FLINK-11693] Add KafkaSerializationSchema that uses ProducerRecord URL: https://github.com/apache/flink/pull/8535#issuecomment-505296858 @aljoscha Thanks for updating the patch and the quick response. Regarding the partitioner issue, just thinking loudly, from design principle perspective, there might be two reasonable approaches: 1. Introduce a generic partitioner that is used across all the connectors that supports partitioning. Flink owns the interface and ensure unified user experience in all the connectors. So this would become a pattern for all the connector implementations. 2. Instead of introducing unified partitioner shared by all the connectors, follow the suggested approach of each connected system. E.g. in Kafka the partitioner is specified via a `partitioner.class` config. This way, user experience with the connected system is honored. For partitioners, it seems Flink does not follow design principle 1, but following principle 2 here is a little tricky because we do not use the key and value serializer the way Kafka expected, therefore the configured Kafka partitioner will only be able to see `byte[]` instead of original objects. Given the current status, I agree that we may as well move towards design principle 1. i.e. Introduce something like: ``` interface Partitioner<T, P, CONTEXT extends RuntimeContext> { P partition(T, CONTEXT extend RuntimeContext); } ``` P is the partition id, T is the type, and CONTEXT is the context for this particular system. RuntimeContext might be an overkill. We can have a simpler one. For Kafka it would be: ``` interface KafkaPartitioner extends Partitioner<IN, Integer, KafkaContext> { } ``` Does this sound reasonable to you?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
