[
https://issues.apache.org/jira/browse/FLINK-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054617#comment-17054617
]
Tzu-Li (Gordon) Tai commented on FLINK-16319:
---------------------------------------------
Hi [~danp11]! Would like to take a step back here.
I briefly discussed pubsub with [~igal] offline, and we were thinking that
implementing this in user-space (i.e. as a {{TopicPartitionFunction}}) might
not be the best idea.
A approach where we implement this in framework-scope, where we can directly
implement Flink operators and have access to broadcast state, might be a better
and more efficient way to do this.
I'm not 100% sure about this yet, but we definitely need to slow down and think
a bit.
As I mentioned in my previous comments, having a proper consensus on how this
should be implemented written down is important before moving on with the code
implementation, and I would strongly suggest that as it would make this easier
to follow for other people as well.
> Pubsub-/Broadcast implementation
> --------------------------------
>
> Key: FLINK-16319
> URL: https://issues.apache.org/jira/browse/FLINK-16319
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
> Reporter: Dan Pettersson
> Priority: Major
>
> Hi everyone,
>
> I have a use case where the id of the functions are brokerId + instrumentId
> that receives trades and orders. The instrument has state changes (Open,
> halted, closed etc) that almost all the functions are interested in. Some
> functions only wants for example the Close message whereas other functions
> wants all state changes for the specific instrument.
>
> I've built a statefun pubsub module that exposes two interfaces, Subscriber
> and Publisher, with these two methods:
>
> default void subscribe(Context context, Subscription... subscriptions)
>
> default void publish(Context context, PublishMessage publishMessage)
>
> Behind the interfaces is a hidden StatefulPubSubFunction that keeps track of
> which partition the subscriber is located in and to which topic it listens to.
>
> Code is located under
> [https://github.com/danp11/flink-statefun/tree/master/statefun-pubsub] if
> anyone is interested.
>
> This code is a "classic pub sub" pattern and I think that this kind of
> functionality would be a great addition to Stateful functions. I create this
> Jira to see if there is an interest to discuss how a optimal
> pubsub-/broadcast solution would look like in SF? Igal has previously
> mentioned that Broadcast could be a good fit for this kind of flow.
> At the moment I don't know the internals of SF and-/or Flink good enough to
> come up with a proposal myself unfortunately.
>
> I know you are very busy at the moment (Its impressive how much you have
> produced only the last couple of weeks!:-) but if someone, on a high level,
> has any ideas on where and how a pub sub pattern could be implemented I'd
> really appreciate it. In the future I hope we can come up with a proposal
> together as I need your help here. If you think that a pubsub-/broadcast
> solution would make SF better that is :-)
>
> Hope to hear your thoughts on this!
>
> Thanks,
>
> /Dan
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
