Re: Request multiple subpartitions of one partition

2019-01-23 Thread Chris Miller
Hi Zhijang, thank you for your replay. I was playing around a little in the last days and ended up in a solution where I change the ResultPartitionView's subpartitionIndex as soon as it returns an EndOfPartition Event. This way I can, sequentially, receive multiple subpartitions at one single

Re: Request multiple subpartitions of one partition

2019-01-09 Thread zhijiang
Hi Chris, I think your requirement seems like this: 1. Determine the number of logic output partitions on upstream side. 2. Determine the number of logic input channels on downstream side. 3. Determine which input channel consumes corresponding output partition. I remembered Tez has similar

Request multiple subpartitions of one partition

2019-01-09 Thread Chris Miller
Hello, let's image we do a hash join of two DataSources. For the join operation we choose parallelism=5. This way Flink uses 5 TaskManagerRunners to do the join job. In particular, the DataSource tasks, EACH ARE CREATING 5 SUBPARTITIONS. Every worker, now requests ONE SUBPARTITION from both