huwh opened a new pull request, #22674:
URL: https://github.com/apache/flink/pull/22674
## What is the purpose of the change
Flink support distributes shuffle descriptors via the blob server to reduce
JobManager overhead. But the default threshold to enable it is 1MB, which never
reaches. Users need to set a proper value for this, but it requires advanced
knowledge before configuring it.
I would like to enable this feature by the number of connections of a group
of shuffle descriptors. For examples, a simple streaming job with two
operators, each with 10,000 parallelism and connected via all-to-all
distribution. In this job, we only get one set of shuffle descriptors, and this
group has 10000 * 10000 connections. This means that JobManager needs to send
this set of shuffle descriptors to 10000 tasks.
Since it's difficult for users to configure, I would like to give it a
default value.
## Brief change log
- *determine if the shuffle descriptor needs to be offloaded by the blob
server based on the number of ShuffleDescriptor edges.*
## Verifying this change
This change added tests and can be verified as follows:
- *Manually verified the change by running a cluster with 1 JobManagers
and 2000 TaskManagers (10 slots per TaskManager), a streaming program with
20000 parallelism, and verifying that the task deploy was successful.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: Task Deployment: (yes)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
