darshan-sj opened a new pull request, #40842:
URL: https://github.com/apache/beam/pull/40842
## [Java SDK - WriteFiles] Allow `maxNumWritersPerBundle` to exceed
`DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE` (20)
### Summary & Motivation
Currently, `WriteFiles.withMaxNumWritersPerBundle(int)` enforces an upper
bound of `DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE` (`20`):
While 20 is a conservative default to bound per-bundle buffer memory,
pipelines writing to many dynamic destinations (especially when paired with
.withEvictWritersWhenFull() introduced in #40200, or on workers with higher
memory capacity) need the ability to increase maxNumWritersPerBundle above 20
(e.g., 50) to reduce writer eviction frequency or shuffle spilling.
As already documented in the WriteFiles.withMaxNumWritersPerBundle(int)
Javadoc trade-offs, tuning maxNumWritersPerBundle higher trades additional
worker memory for fewer writer evictions / less shuffle spilling.
### Changes Made
1. WriteFiles.withMaxNumWritersPerBundle(int) (WriteFiles.java):
• Removed the maxNumWritersPerBundle <=
DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE upper-bound check while retaining
maxNumWritersPerBundle > 0.
• Updated the Javadoc to note that DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE
(20) is the default value rather than a hard maximum.
2. Unit Tests (WriteFilesTest.java):
• Added assertions in WriteFilesTest.testBuildWrite verifying that
withMaxNumWritersPerBundle(50) succeeds and withMaxNumWritersPerBundle(0)
throws IllegalArgumentException.
<!--
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
- [ ] Replace these instructions with a description of your change and why
you have made it.
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123` or `fixes #456`).
- [ ] Update `CHANGES.md` with noteworthy changes.
See [CONTRIBUTING.md]() for more details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
