[
https://issues.apache.org/jira/browse/FLINK-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047367#comment-17047367
]
Dan Pettersson commented on FLINK-16319:
----------------------------------------
Hi Gordon and Thanks for your detailed response!
Yes, your understanding is correct how the code works today.
Your suggestion to remove the interfaces and let the context handle the topic
routing
would indeed make it much more clear. As said, the function would then be just
a
"TopicPartitionFunction" that holds a list of subscriptions.
How the function instances would subscribe to topics without being
depended on the context is something I also have thought of.
The current solution is not good enough as the function instance must do an ugly
if(firstMessage()){
-> set up subscription
}
to make the subscription registration performant.
I haven't looked in the SF code enough but for this and also for other
initial instance tasks an optional method "initFunction" would be nice to have
:-)
But maybe that won't work considering that the functions are virtual?
I'll be happy to work with you on this and exchange ideas and get requirements
from you when you have more time.
Have a nice weekend,
/Dan
> Pubsub-/Broadcast implementation
> --------------------------------
>
> Key: FLINK-16319
> URL: https://issues.apache.org/jira/browse/FLINK-16319
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
> Reporter: Dan Pettersson
> Priority: Major
>
> Hi everyone,
>
> I have a use case where the id of the functions are brokerId + instrumentId
> that receives trades and orders. The instrument has state changes (Open,
> halted, closed etc) that almost all the functions are interested in. Some
> functions only wants for example the Close message whereas other functions
> wants all state changes for the specific instrument.
>
> I've built a statefun pubsub module that exposes two interfaces, Subscriber
> and Publisher, with these two methods:
>
> default void subscribe(Context context, Subscription... subscriptions)
>
> default void publish(Context context, PublishMessage publishMessage)
>
> Behind the interfaces is a hidden StatefulPubSubFunction that keeps track of
> which partition the subscriber is located in and to which topic it listens to.
>
> Code is located under
> [https://github.com/danp11/flink-statefun/tree/master/statefun-pubsub] if
> anyone is interested.
>
> This code is a "classic pub sub" pattern and I think that this kind of
> functionality would be a great addition to Stateful functions. I create this
> Jira to see if there is an interest to discuss how a optimal
> pubsub-/broadcast solution would look like in SF? Igal has previously
> mentioned that Broadcast could be a good fit for this kind of flow.
> At the moment I don't know the internals of SF and-/or Flink good enough to
> come up with a proposal myself unfortunately.
>
> I know you are very busy at the moment (Its impressive how much you have
> produced only the last couple of weeks!:-) but if someone, on a high level,
> has any ideas on where and how a pub sub pattern could be implemented I'd
> really appreciate it. In the future I hope we can come up with a proposal
> together as I need your help here. If you think that a pubsub-/broadcast
> solution would make SF better that is :-)
>
> Hope to hear your thoughts on this!
>
> Thanks,
>
> /Dan
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
