Re: Split one dataset into multiple

2018-11-06 Thread Fabian Hueske
You have to define a common type, like an n-ary Either type and return that from your source / operator. The resulting DataSet can be consumed by multiple FlatmapFunctions, each extracting and forwarding one of the the result types. Cheers, Fabian Am Di., 6. Nov. 2018 um 10:40 Uhr schrieb madan

Re: Split one dataset into multiple

2018-11-06 Thread madan
Hi Vino, Thank you for suggestions. In my case I am using DataSet since data is limited, and split/select is not available on DataSet api. I doubt even hash partition might not work for me. By doing hash partition, I do not know which partition is having which entity data (Dept, Emp in my

Re: Split one dataset into multiple

2018-11-05 Thread vino yang
Hi madan, I think you need to hash partition your records. Flink supports hash partitioning of data. The operator is keyBy. If the value of your tag field is enumerable, you can also use split/select to achieve your purpose. Thanks, vino. madan 于2018年11月5日周一 下午6:37写道: > Hi, > > I have a

Split one dataset into multiple

2018-11-05 Thread madan
Hi, I have a custom iterator which gives data of multitple entities. For example iterator gives data of Department, Employee and Address. Record's entity type is identified by a field value. And I need to apply different set of operations on each dataset. Ex., Department data may have