Re: Rebalance to subtasks in same TaskManager instance

2018-02-07 Thread johannes.barn...@clarivate.com
Hi Piotrek Yes I've compared rebalance with rescale. I adjusted the parallelism of the source and target operators so that rescale would behave more or less like the "local or shuffle grouping" option. I was able to show that for my use case a "local or shuffle grouping" option would yield at

Rebalance to subtasks in same TaskManager instance

2018-02-05 Thread johannes.barn...@clarivate.com
Hi, I have a streaming topology with source parallelism of M and a target operator parallelism of N. For optimum performance I have found that I need to choose M and N independently. Also, the source subtasks do not all produce the same number of records and therefor I have to rebalance to the