tzulitai opened a new pull request #36: [FLINK-16123] Add auto-routable Protobuf Kafka ingress URL: https://github.com/apache/flink-statefun/pull/36 This PR adds support for a Protobuf Kafka ingress that automatically routes values as Protobuf `Any` messages to target functions using the Kafka record's key (as UTF8 strings) as the function id. The YAML definition looks like the following: ``` ingress: meta: type: org.apache.flink.statefun.sdk.kafka/routable-protobuf-kafka-connector id: com.mycomp.foo/bar spec: address: kafka-broker:9092 consumerGroupId: my-group-id topics: - topic: topic-1 typeUrl: com.googleapis/com.mycomp.foo.MessageA targets: - com.mycomp.foo/function-1 - com.mycomp.foo/function-2 - topic: topic-2 typeUrl: com.googleapis/com.mycomp.foo.MessageB targets: - com.mycomp.foo/function-2 autoOffsetResetPosition: earliest startupPosition: type: earliest properties: - foo.config: bar ``` Summary: - For each topic, users define the Protobuf type URL of the message type in that topic, and the list of target functions to route records of that topic to. - Users must use as key the target function ids, as a UTF8 string. When binding this ingress to a module, `JsonModule` automatically binds a `AutoRoutableProtobufRouter` for the ingress. Therefore, users do not need to explicitly define a router for the ingress in the YAML. --- ### Verifying - A unit test `RoutableProtobufKafkaSourceProviderTest` is added to demonstrate example YAML and usage. - An E2E test for the feature will follow-up after this PR.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
