[ 
https://issues.apache.org/jira/browse/FLINK-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626962#comment-15626962
 ] 

Jamie Grier commented on FLINK-3659:
------------------------------------

I would like to suggest that rather than adding a new API method, 
connectWithBroadcast(), we just enable this functionality via the current API 
and check access to state variants at runtime.

In other words all of the following will work:

{code:java}
DataStream stream = new DataStream();
    DataStream keyedStream = new DataStream().keyBy("...");
    DataStream broadcastStream = new DataStream().broadcast();

    stream.connect(stream);
    stream.connect(keyedStream);
    stream.connect(broadcastStream);

    keyedStream.connect(stream);
    keyedStream.connect(keyedStream);
    keyedStream.connect(broadcastStream);

    broadcastStream.connect(stream);
    broadcastStream.connect(keyedStream);
    broadcastStream.connect(broadcastStream);
{code}

... and based on the actual input types to the LHS and RHS of these connected 
streams we check at runtime what they can do.  For example:

{code:java}
 keyedStream.connect(broadcastStream).flatMap(...)
{code}

In the above the user can access the keyed and broadcast state from his 
flatMap1() method (LHS), and can access only broadcast state from his 
flatMap2() method (RHS).

The reason I suggest this is that it keeps the API simpler and more intuitive 
and there aren't any new APIs to learn -- other than for the new broadcast 
state access itself.  People are already building things exactly this way -- 
they are just being forced to use Checkpointed to make their state 
fault-tolerant.  This allows the same API as before just with some additional 
capabilities and this will work with re-scalable state properly.

In a future version of Flink (2.0+) maybe we can start to think about 
@annotation based APIs more like the current Beam approach which I think is 
very nice.  It allows both flexible and dynamic API evolution as well as 
"static" verification.  Anyway maybe in the future we could do something more 
like this:

{code:java}
class MyCoFlatMap { // doesn't even need to extend anything
   @FlatMap(input=KeyedStream)
   void dataFunc(@BroadcastState("name") String s, @KeyedState("name") Integer 
i) { … }

   @FlatMap(input=BroadcastStream)
   void controlFunc(@BroadcastState("name") String s) { … }
}
{code}

> Add ConnectWithBroadcast Operation
> ----------------------------------
>
>                 Key: FLINK-3659
>                 URL: https://issues.apache.org/jira/browse/FLINK-3659
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>
> We should add a new operation that has a main input that can be keyed (but 
> doesn't have to be) and a second input that is always broadcast. This is 
> similar to a {{CoFlatMap}} or {{CoMap}} but there either both inputs have to 
> be keyed or non-keyed.
> This builds on FLINK-4940 which aims at adding broadcast/global state. When 
> processing an element from the broadcast input only access to broadcast state 
> is allowed. When processing an element from the main input access both the 
> regular keyed state and the broadcast state can be accessed.
> I'm proposing this as an intermediate/low-level operation because it will 
> probably take a while until we add support for side-inputs in the API. This 
> new operation would allow expressing new patterns that cannot be expressed 
> with the currently expressed operations.
> This is the new proposed API (names are non-final): 
> 1) Add {{DataStream.connectWithBroadcast(DataStream)}} and 
> {{KeyedStream.connectWithBroadcast(DataStream)}}
> 2) Add {{ConnectedWithBroadcastStream}}, akin to {{ConnectedStreams}}/
> 3) Add {{BroadcastFlatMap}} and {{TimelyBroadcastFlatMap}} as the user 
> functions.
> Sketch of the user function:
> {code}
> interface BroadcastFlatMapFunction {
>   public void flatMap(IN in, Collector out);
>   public void processBroadcastInput(BIN in);
> }
> {code}
> The API names, function names are a bit verbose and we have to add two new 
> different ones but I don't see a way around this with the current way the 
> Flink API works.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to