That would be very helpful... Thanks for the support, Flavio
On Fri, Apr 17, 2015 at 10:04 AM, Till Rohrmann <till.rohrm...@gmail.com> wrote: > No its not, but at the moment there is afaik no other way around it. There > is an issue for proper outer join support [1] > > [1] https://issues.apache.org/jira/browse/FLINK-687 > > On Fri, Apr 17, 2015 at 10:01 AM, Flavio Pompermaier <pomperma...@okkam.it > > wrote: > >> Could resolve the problem but the fact to accumulate stuff in a local >> variable is it safe if datasets are huge..? >> >> On Fri, Apr 17, 2015 at 9:54 AM, Till Rohrmann <till.rohrm...@gmail.com> >> wrote: >> >>> If it's fine when you have null string values in the cases where >>> D1.f1!="a1" or D1.f2!="a2" then a possible solution could look like (with >>> Scala API): >>> >>> val ds1: DataSet[(String, String, String)] = getDS1 >>> val ds2: DataSet[(String, String, String)] = getDS2 >>> >>> ds1.coGroup(ds2).where(2).equalTo(0) { >>> (left, right, collector: Collector[(String, String, String, String)]) >>> => { >>> if(right.isEmpty) { >>> left foreach { >>> element => { >>> val value1 = if(element._2 == "a1") element._3 else null >>> val value2 = if(element._2 == "a2") element._3 else null >>> collector.collect((element._1, null, value1, value2)) >>> } >>> } >>> } else { >>> val array = right.toArray >>> for(leftElement <- left) { >>> val value1 = if(leftElement._2 == "a1") leftElement._3 else null >>> val value2 = if(leftElement._2 == "a2") leftElement._3 else null >>> >>> for(rightElement <- array) { >>> collector.collect(leftElement._1, rightElement._1, value1, value2)) >>> } >>> } >>> } >>> } >>> } >>> >>> Does this solve your problem? >>> >>> On Fri, Apr 17, 2015 at 9:30 AM, Flavio Pompermaier < >>> pomperma...@okkam.it> wrote: >>> >>>> Hi Till, >>>> thanks for the reply. >>>> What I'd like to do is to merge D1 and D2 if there's a ref from D1 to >>>> D2 (D1.f2==D2.f0). >>>> If this condition is true, I would like to produce a set of tuples with >>>> the matching elements >>>> at the first to places (D1.*f2*, D2.*f0*) and the other two values (if >>>> present) of the matching tuple >>>> in D1 when D1.f1==*"a1"* and D1.f2=*"a2"* (string values) respectively. >>>> (PS: For each value of D1.f0 you can have at most one value of a1 and >>>> a2) >>>> >>>> Is it more clear? >>>> >>>> On Fri, Apr 17, 2015 at 9:03 AM, Till Rohrmann <till.rohrm...@gmail.com >>>> > wrote: >>>> >>>>> Hi Flavio, >>>>> >>>>> I don't really understand what you try to do. What does >>>>> D1.f2(D1.f1==p1) mean? What does happen if the condition in D1.f2(if >>>>> D1.f1==p2) is false? >>>>> >>>>> Where does the values a1 and a2 in (A, X, a1, a2) come from when you >>>>> join [(A, p3, X), (X, s, V)] and [(A, p3, X), (X, r, 2)]? Maybe you can >>>>> elaborate a bit more on your example. >>>>> >>>>> Cheers, >>>>> >>>>> Till >>>>> >>>>> On Thu, Apr 16, 2015 at 10:09 PM, Flavio Pompermaier < >>>>> pomperma...@okkam.it> wrote: >>>>> >>>>>> I cannot find a solution to my use case :( >>>>>> I have 2 datasets D1 and D2 like: >>>>>> >>>>>> D1: >>>>>> A,p1,a1 >>>>>> A,p2,a2 >>>>>> A,p3,X >>>>>> B,p3,Y >>>>>> B,p1,b1 >>>>>> >>>>>> D2: >>>>>> X,s,V >>>>>> X,r,2 >>>>>> Y,j,k >>>>>> >>>>>> I'd like to have a unique dataset D3(Tuple4) like >>>>>> >>>>>> A,X,a1,a2 >>>>>> B,Y,b1,null >>>>>> >>>>>> Basically filling with <D1.f0,D2.f0,D1.f2(D1.f1==p1),D1.f2(if >>>>>> D1.f1==p2)> when D1.f2==D2.f0. >>>>>> Is that possible and how? >>>>>> Could you show me a simple snippet? >>>>>> >>>>>> Thanks in advance, >>>>>> Flavio >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Apr 16, 2015 at 9:48 PM, Till Rohrmann <trohrm...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> You can materialize the input of the right input by creating an >>>>>>> array out of it, for example. Then you can reiterate over it. >>>>>>> >>>>>>> Cheers, >>>>>>> Till >>>>>>> On Apr 16, 2015 7:37 PM, "Flavio Pompermaier" <pomperma...@okkam.it> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Maximilian, >>>>>>>> I tried your solution but it doesn't work because the rightElements >>>>>>>> iterator cannot be used more than once: >>>>>>>> >>>>>>>> Caused by: org.apache.flink.util.TraversableOnceException: The >>>>>>>> Iterable can be iterated over only once. Only the first call to >>>>>>>> 'iterator()' will succeed. >>>>>>>> >>>>>>>> On Wed, Apr 15, 2015 at 12:59 PM, Maximilian Michels < >>>>>>>> m...@apache.org> wrote: >>>>>>>> >>>>>>>>> Hi Flavio, >>>>>>>>> >>>>>>>>> Here's an simple example of a Left Outer Join: >>>>>>>>> https://gist.github.com/mxm/c2e9c459a9d82c18d789 >>>>>>>>> >>>>>>>>> As Stephan pointed out, this can be very easily modified to >>>>>>>>> construct a Right Outer Join (just exchange leftElements and >>>>>>>>> rightElements >>>>>>>>> in the two loops). >>>>>>>>> >>>>>>>>> Here's an excerpt with the most important part, the coGroup >>>>>>>>> function: >>>>>>>>> >>>>>>>>> public static class LeftOuterJoin implements >>>>>>>>> CoGroupFunction<Tuple2<Integer, String>, Tuple2<Integer, String>, >>>>>>>>> Tuple2<Integer, Integer>> { >>>>>>>>> >>>>>>>>> @Override >>>>>>>>> public void coGroup(Iterable<Tuple2<Integer, String>> leftElements, >>>>>>>>> Iterable<Tuple2<Integer, String>> >>>>>>>>> rightElements, >>>>>>>>> Collector<Tuple2<Integer, Integer>> out) >>>>>>>>> throws Exception { >>>>>>>>> >>>>>>>>> final int NULL_ELEMENT = -1; >>>>>>>>> >>>>>>>>> for (Tuple2<Integer, String> leftElem : leftElements) { >>>>>>>>> boolean hadElements = false; >>>>>>>>> for (Tuple2<Integer, String> rightElem : rightElements) { >>>>>>>>> out.collect(new Tuple2<Integer, Integer>(leftElem.f0, >>>>>>>>> rightElem.f0)); >>>>>>>>> hadElements = true; >>>>>>>>> } >>>>>>>>> if (!hadElements) { >>>>>>>>> out.collect(new Tuple2<Integer, Integer>(leftElem.f0, >>>>>>>>> NULL_ELEMENT)); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Apr 15, 2015 at 11:01 AM, Stephan Ewen <se...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I think this may be a great example to add as a utility function. >>>>>>>>>> >>>>>>>>>> Or actually add as an function to the DataSet, internally >>>>>>>>>> realized as a special case of coGroup. >>>>>>>>>> >>>>>>>>>> We do not have a ready example of that, but it should be >>>>>>>>>> straightforward to realize. Similar as for the join, coGroup on the >>>>>>>>>> join >>>>>>>>>> keys. Inside the coGroup function, emit the combination of all >>>>>>>>>> values from >>>>>>>>>> the two iterators. If one of them is empty (the one that is not >>>>>>>>>> outer) then >>>>>>>>>> emit all values from the outer side. >>>>>>>>>> >>>>>>>>>> Greetings, >>>>>>>>>> Stephan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Apr 15, 2015 at 10:36 AM, Flavio Pompermaier < >>>>>>>>>> pomperma...@okkam.it> wrote: >>>>>>>>>> >>>>>>>>>>> Do you have an already working example of it? :) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 15, 2015 at 10:32 AM, Ufuk Celebi <u...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15 Apr 2015, at 10:30, Flavio Pompermaier < >>>>>>>>>>>> pomperma...@okkam.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > Hi to all, >>>>>>>>>>>> > I have to join two datasets but I'd like to keep all data in >>>>>>>>>>>> the left also if there' no right dataset. >>>>>>>>>>>> > How can you achieve that in Flink? maybe I should use coGroup? >>>>>>>>>>>> >>>>>>>>>>>> Yes, currently you have to implement this manually with a >>>>>>>>>>>> coGroup >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>> >>>> >>> >> >> >
