Re: Implementing a low level join

2019-08-15 Thread Felipe Gutierrez
Thanks for the advice. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 9:59 AM Fabian Hueske wrote: > Hi Felipe, > > No, this is not possible (with reasonable effort). > A

Re: Implementing a low level join

2019-08-15 Thread Fabian Hueske
Hi Felipe, No, this is not possible (with reasonable effort). A checkpoint would be the right time to do the change, but this would be very involved, IMO. As I said, you need a global decision to switch the strategy. This could be communicated with a checkpoint barrier. Then all operator would

Re: Implementing a low level join

2019-08-15 Thread Felipe Gutierrez
Hi Fabian, thanks for jumping within this thread. Do you think there is possible to extend any join generic operator in order to make it a little dynamic? I was thinking that after I process a checkpoint I can change the join strategy. and if you do, do you have any toy example of this? Thanks,

Re: Implementing a low level join

2019-08-15 Thread Fabian Hueske
Hi, Just to clarify. You cannot dynamically switch the join strategy while a job is running. What Hequn suggested was to have a util method Util.joinDynamically(ds1, ds2) that chooses the join strategy when the program is generated (before it is submitted for execution). The problem is that

Re: Implementing a low level join

2019-08-15 Thread Felipe Gutierrez
I see, I am gonna try this. Thanks Hequn *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 4:01 AM Hequn Cheng wrote: > Hi Felipe, > > If I understand correctly, you also have

Re: Implementing a low level join

2019-08-14 Thread Hequn Cheng
Hi Felipe, If I understand correctly, you also have to decide whether to broadcast the datastream from the right side before performing the function? One option is you can add a Util method to join dynamically, e.g., Util.joinDynamically(ds1, ds2). In the util method, you can implement your own

Re: Implementing a low level join

2019-08-14 Thread Felipe Gutierrez
Hi Hequn, I am implementing the broadcast and the regular join. As you said I need different functions. My question is more about if I can have an operator which decides beteween broadcast and regular join dynamically. I suppose I will have to extend the generic TwoInputStreamOperator in Flink.

Re: Implementing a low level join

2019-08-13 Thread Hequn Cheng
Hi Felipe, > I want to implement a join operator which can use different strategies for joining tuples. Not all kinds of join strategies can be applied to streaming jobs. Take sort-merge join as an example, it's impossible to sort an unbounded data. However, you can perform a window join and use