austince opened a new pull request #15348:
URL: https://github.com/apache/flink/pull/15348
## What is the purpose of the change
To maintain compatibility between reactive and default scheduling modes, the
`maxParallelism` of operators must be set consistently. Currently, there are
different defaults applied depending on the mode. This PR unifies setting those
defaults for `JobGraphs` to the `jobgraph` package via a util class. I also
considered adding the defaulting logic to the `JobGraphBuilder`, but that seems
to be mostly used for tests.
The main change is bringing the defaulting logic out of the
`ExecutionJobVertex`, and then allowing the `JobVertex` to know if its
`maxParallelism` been explicitly set by the user or the system. I've also tried
to consolidate the number of places where the `-1` default magic number is
defined.
## Brief change log
- Both reactive and default JobGraphs are configured in the
DefaultJobManagerRunnerFactory
- JobVertex has a package-private method for setting the auto-configured
flag, which is used when restoring from operator state among other things
- Tests updated to use the common helper methods
## Verifying this change
This change added tests and can be verified as follows:
- Refactored current auto-configuration test into a parameterized
JobGraphConfigurationUtilsAutoConfigurtionTest that tests both default and
reactive parallelism auto-configuration
- Updated existing to explicitly add parallelism when creating vertices
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? JavaDocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
