[
https://issues.apache.org/jira/browse/FLINK-16396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-16396:
----------------------------------------
Priority: Blocker (was: Critical)
> Support binding Kafka egresses with YAML-ized modules
> -----------------------------------------------------
>
> Key: FLINK-16396
> URL: https://issues.apache.org/jira/browse/FLINK-16396
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
> Affects Versions: statefun-1.1
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Blocker
>
> For a full multi-language story in Stateful Functions, we're still lacking a
> YAML-ized way to bind egresses. We should start with a Kafka one, as that is
> the currently only supported ingress for multi-language applications.
> We propose to initially add a more "manual" approach for this, which gives
> the user full freedom on partitioning and record serialization.
> A more specialized egress with built-in routing / protobuf serialization can
> maybe be added in the future once we collect more knowledge on usage from
> users.
> The plan is as follows:
> * Add a {{KafkaProducerRecord}} protobuf message with schema:
> {code}
> message KafkaProducerRecord {
> string topic
> string key
> bytes valueBytes
> }
> {code}
> How the value was serialized would be completely user-space; the bytes are
> written as is to Kafka. Partitioning depends on the partitioner used by the
> Kafka client, which may be configured with Kafka properties via YAML.
> * The Flink sink provider for this egress should expect only messages of
> {{KafkaProducerRecord}} (wrapped within a protobuf Any). It simply transforms
> the information in the received {{KafkaProducerRecord}} to Kafka's own
> {{ProducerRecord}}, and writes that to Kafka.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
