[
https://issues.apache.org/jira/browse/FLINK-19538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464355#comment-17464355
]
Mingmin Xu edited comment on FLINK-19538 at 12/23/21, 8:21 AM:
---------------------------------------------------------------
hello [~sjwiesman] [~tzulitai] I'm looking for a similar feature and would like
to hear your opinions to reopen the task.
The feature I'm looking for is quite straightforward, and close to what is
described in task description, to generate a customized ID from kafka key,
instead of converting it to a string like
[https://github.com/apache/flink-statefun/blob/release-3.1/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/binders/ingress/v1/RoutableKafkaIngressDeserializer.java#L48-L49]
. We store quite a few metadata inside of the key object which make it hard
to use directly.
Currently I use a function which acts as a router solely as describe above. Not
sure how to make it generic for all IOs, I would like to start with Kafka only,
to provide a customized class to implement the method `String
genMessageId(ConsumerRecord<byte[], byte[]> inputRecord)`.
------
I really like the ideas mentioned in the comments, which we're also looking at:
1. With one ingress message, how to forward it to one or more targets based on
message content. The use case is, one message could be categorized in multiple
levels, we want to sent it to different functions to perform aggregations
separately;
2. An advanced router to decide how data is shuffled, to leverage region
failover in Flink. Our use case is close to online serving, stop-the-world
restart is not the preferred mode. --More details need to clarify here, we
haven't done any work on this yet.
Any thoughts? I'm happy to take the task if that's the pattern we want to
support.
was (Author: mingmxu):
hello [~sjwiesman] [~tzulitai] I'm looking for a similar feature and would like
to hear your opinions to reopen the task.
The feature I'm looking for is quite straightforward, and more close to what is
described in task description, to generate a customized ID from kafka key,
instead of converting it to a string like
[https://github.com/apache/flink-statefun/blob/release-3.1/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/binders/ingress/v1/RoutableKafkaIngressDeserializer.java#L48-L49]
. We store quite a few metadata inside of the key object which make it hard
to use directly.
Currently I use a function which acts as a router solely as describe above. Not
sure how to make it generic for all IOs, I would like to start with Kafka only,
to provide a customized class to implement the method `String
genMessageId(ConsumerRecord<byte[], byte[]> inputRecord)`.
------
I really like the ideas mentioned in the comments, which we're also looking at:
1. With one ingress message, how to forward it to one or more targets based on
message content. The use case is, one message could be categorized in multiple
levels, we want to sent it to different functions to perform aggregations
separately;
2. An advanced router to decide how data is shuffled, to leverage region
failover in Flink. Our use case is close to online serving, stop-the-world
restart is not the preferred mode. --More details need to clarify here, we
haven't done any work on this yet.
Any thoughts? I'm happy to take the task if that's the pattern we want to
support.
> Support more complex routers in remote ingress definitions
> ----------------------------------------------------------
>
> Key: FLINK-19538
> URL: https://issues.apache.org/jira/browse/FLINK-19538
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
> Reporter: Seth Wiesman
> Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> Ingresses defined via Yaml route messages with an implicit key pulled via the
> header of the source. If users need to route based on another key they must
> either be first sent to a function that serves solely as a router and then
> forward it or write their ingress in Java. We should support more complex
> routers in Yaml.
> See ML Question:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Native-State-in-Python-Stateful-Functions-td38563.html
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
