zhuzhurk opened a new pull request, #21743:
URL: https://github.com/apache/flink/pull/21743
## What is the purpose of the change
This change improves the `DefaultPreferredLocationsRetriever` so that it
ignores the input locations of a ConsumePartitionGroup if the corresponding
ConsumerVertexGroup is too large. This helps to avoid tasks to be unevenly
distributed on nodes when running batch jobs or running jobs in
session/standalone mode, because the consumers in this case will tend to be
placed on the same node of the input vertex.
## Brief change log
*(for example:)*
- Changed `EdgeManagerBuildUtil` to set the
ConsumedPartitionGroup/ConsumerVertexGroup to its corresponding
ConsumerVertexGroup/ConsumedPartitionGroup
- Changed `DefaultPreferredLocationsRetriever` to ignore the input
locations of a ConsumePartitionGroup if the corresponding ConsumerVertexGroup
is too large(compared to the ConsumePartitionGroup)
## Verifying this change
- *Added unit tests in EdgeManagerBuildUtilTest and
DefaultPreferredLocationsRetrieverTest*
- *The change is also covered by existing tests of preferred location
retriever*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't
know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
