MartijnVisser opened a new pull request, #29662:
URL: https://github.com/apache/flink/pull/29662
## What is the purpose of the change
`pipeline.jobvertex-parallelism-overrides` is applied by the Dispatcher only
to a submitted JobGraph. Since FLINK-36065 clients submit a StreamGraph, so on
2.x the option has no effect; the operator autoscaler writes its decisions into
it. Verified with a probe job: 1.20.5 applies it in session and application
mode, 2.3.0 does not; `git bisect` names 64cbcd2d4b3. The REST jar run, the SQL
gateway and local execution submit a StreamGraph too.
The Dispatcher writes the resolved map into the submitted plan's job
configuration (by reference over the embedded executor's local RPC; MiniCluster
clones, REST deserializes). A StreamGraph persisted in HA before this fix
recovers without overrides, as a JobGraph does today.
Supersedes #27334, which moved the overrides into the scheduler factories;
the diagnosis and the `env.execute()` test idea are Barak's.
## Brief change log
- Resolve the map once in the Dispatcher (`JobVertexParallelismOverrides`)
and store it in a StreamGraph's job configuration
- `StreamingJobGraphGenerator.createJobVertex` applies it after the vertex
id is computed, for both generators
- `AdaptiveGraphManager` decides the forward group of an overridden vertex,
or fails on a conflict
- Option description updated, docs regenerated
## Verifying this change
This change added tests and can be verified as follows:
- Commit 1 adds the tests listed in its message; they fail on master and
pass with commit 2
- Probe job on a dist built from this branch (uid on the map, override to
4): `GET /jobs/<id>` shows 4 and the job master logs `Changing job vertex ...
parallelism from 1 to 4`, in session and application mode, as on 1.20.5
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Fable 5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
