Tzu-Li (Gordon) Tai created FLINK-13093:
-------------------------------------------
Summary: Provide an easy way to modify max parallelism using the
State Processor API
Key: FLINK-13093
URL: https://issues.apache.org/jira/browse/FLINK-13093
Project: Flink
Issue Type: Sub-task
Reporter: Tzu-Li (Gordon) Tai
Currently, the State Processor API does not easily allow one to modify the max
parallelism of a job. To do that with the current state of the API, one would
have to read all state that exists in a loaded {{ExistingSavepoint}}, extract
them as {{DataSet}}s, and then create a {{NewSavepoint}} that has the new max
parallelism with all the extracted data sets bootstrapped as new state.
It would be nice if the user could simply do something like the following (API
is TBD):
{code}
ExistingSavepoint savepoint = Savepoint.load("path", env, backend);
savepoint.modifyMaxParallelism("newPath", newParallelism);
{code}
Under the hood, a batch job is launched that repartitions all existing operator
state using the new max parallelism, and writes to the re-partitioned state
data into the new savepoint path.
The API can be designed such that the user cannot modify the max parallelism
and add / new operators at the same time, to not over complicate the batch job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
