Changing the parallelism works in Flink by taking a savepoint, shutting down the job, and restarting it from the savepoint with another parallelism.
The rescale() operator defines how records are exchanged between two operators with different parallelism. Rescale prefers local data exchange over uniform distribution (which would be rebalance()). For example if you have a pipeline A -rescale-> B, where operator A has 2 tasks and operator B 4 tasks, then A(1) would send data to B(1) and B(3) and A(2) to B(2) and B(4). Since A(1) / B(1) and A(2) / B(2) run on the same machine (unless explicitly differently scheduled), the data exchange between them is local. Best, Fabian 2018-02-13 16:22 GMT+01:00 m@xi <makisnt...@gmail.com>: > Thanks a lot Fabian and Xingcan! > > @ Fabian : Nice answer. It enhanced my intuition on the topic. So, how one > may change the parallelism while the Flink job is running, e.g. lower the > parallelism during the weekend? > > > Also, it is not clear to me how to use the rescale() operator. If you may > provide a more thorough example, cause the one in the documentation is not > so good in my humble opinion. With some code/pseudo code, it would be > great. > > Thanks in advance. > > Best, > Max > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
