mxm opened a new pull request, #21023: URL: https://github.com/apache/flink/pull/21023
## What is the purpose of the change This allows to change the job vertex parallelism of a JobGraph during job submission time without having to modify the JobGraph upfront. The initial idea was to add a new field to the payload of the job submit Rest endpoint. However, it is probably more practical to handle the overrides in the same way as other PipelineOptions already do it, i.e. via the configuration. The implementation is deliberately lenient with respect to the presence of job vertices. If vertices have been removed or new ones have been added, only the ones found will have their parallelism overrides. The verification should be performed by the caller, not by Flink. In particular, we want to support scenarios where users modify the deployment and we might not yet have overrides for all vertices. Based on feedback in https://github.com/apache/flink/pull/20953. ## Brief change log - Add option to override job vertex parallelisms during job submission - Allow Map<String, T> ConfigOptions ## Verifying this change Tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
