Hi Hequn,

I am implementing the broadcast and the regular join. As you said I need
different functions. My question is more about if I can have an operator
which decides beteween broadcast and regular join dynamically. I suppose I
will have to extend the generic TwoInputStreamOperator in Flink. Do you
have any suggestion?

Thanks

On Wed, 14 Aug 2019, 03:59 Hequn Cheng, <chenghe...@gmail.com> wrote:

> Hi Felipe,
>
> > I want to implement a join operator which can use different strategies
> for joining tuples.
> Not all kinds of join strategies can be applied to streaming jobs. Take
> sort-merge join as an example, it's impossible to sort an unbounded data.
> However, you can perform a window join and use the sort-merge strategy to
> join the data within a window. Even though, I'm not sure it's worth to do
> it considering the performance.
>
> > Therefore, I am not sure if I will need to implement my own operator to
> do this or if it is still possible to do with CoProcessFunction.
> You can't implement broadcast join with CoProcessFunction. But you can
> implement it with BroadcastProcessFunction or
> KeyedBroadcastProcessFunction, more details here[1].
>
> Furthermore, you can take a look at the implementation of both window join
> and non-window join in Table API & SQL[2]. The code can be found here[3].
> Hope this helps.
>
> Best, Hequn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/sql.html#joins
> [3]
> https://github.com/apache/flink/tree/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/runtime/join
>
>
> On Tue, Aug 13, 2019 at 11:30 PM Felipe Gutierrez <
> felipe.o.gutier...@gmail.com> wrote:
>
>> Hi all,
>>
>> I want to implement a join operator which can use different strategies
>> for joining tuples. I saw that with CoProcessFunction I am able to
>> implement low-level joins [1]. However, I do know how to decide between
>> different algorithms to join my tuples.
>>
>> On the other hand, to do a broadcast join I will need to use the
>> broadcast operator [2] which yields a BroadcastStream. Therefore, I am not
>> sure if I will need to implement my own operator to do this or if it is
>> still possible to do with CoProcessFunction.
>>
>> Does anyone have some clues for this matter?
>> Thanks
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/process_function.html#low-level-joins
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>

Reply via email to