Any help on this?
On 9 Feb 2016 18:03, "Flavio Pompermaier" <[email protected]> wrote:

> Hi to all,
>
> in my program I have a Dataset that generated different types of object
> wrt the incoming element.
> Thus it's like a Map<Tuple2,Object>.
> In order to type the different generated datasets I do something:
>
> Dataset<Tuple2> start =...
>
> Dataset<MyObj1> ds1 = start.filter().map(..);
> Dataset<MyObj1> ds2 = start.filter().map(..);
> Dataset<MyObj3> ds3 = start.filter().map(..);
> Dataset<MyObj3> ds4 = start.filter().map(..);
>
> However this is very inefficient (I think because Flink needs to
> materialize the entire source dataset for every slot).
>
> It's much more efficient to group the generation of objects of the same
> type. E.g.:
>
> Dataset<Tuple2> start =..
>
> Dataset<MyObj1> tmp1 = start.map(..);
> Dataset<MyObj3> tmp2 = start.map(..);
> Dataset<MyObj1> ds1 = tmp1.filter();
> Dataset<MyObj1> ds2 = tmp1.filter();
> Dataset<MyObj3> ds3 = tmp2.filter();
> Dataset<MyObj3> ds4 = tmp2.filter();
>
> Increasing the number of slots per task manager make things worse and
> worse :)
> Is there a way to improve this situation? Is it possible to write a "map"
> generating different type of object and then filter them by generated class
> type?
>
> Best,
> Flavio
>
>
>
>
>
>
>

Reply via email to