vishalmore90 opened a new pull request, #41039:
URL: https://github.com/apache/beam/pull/41039
<!--
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
- [x] Replace these instructions with a description of your change and why
you have made it.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123` or `fixes #456`).
- [ ] Update `CHANGES.md` with noteworthy changes.
See [CONTRIBUTING.md]() for more details.
-->
fixes #41038
## Summary
In
`org.apache.beam.runners.spark.translation.ValueAndCoderLazySerializable`,
`writeCommon` checks `coder.isRegisterByteSizeObserverCheap(value)` to pre-size
the serialization buffer `ByteArrayOutputStream(bufferSize)`.
However, `ByteSizeObserver.reportElementSize(long)` was never invoked
because `ElementByteSizeObserver.advance()` was never called (in
`ElementByteSizeObserver`, `update` only accumulates into `totalSize`, and
`reportElementSize` is only called during `advance()`).
As a result:
- `observer.observedSize` remained `0`.
- `bufferSize` was set to `0` instead of the element's actual size or
default capacity (1024 bytes).
- A 0-capacity `ByteArrayOutputStream(0)` was instantiated for every cheap
element (e.g. `BigEndianIntegerCoder`, `BigEndianLongCoder`, `ByteArrayCoder`,
`StringUtf8Coder`, `KvCoder`, etc.), forcing immediate buffer reallocation and
array copying on every write.
This PR fixes the issue by:
1. Checking `!observer.getIsLazy()` and calling `observer.advance()` before
reading `observer.observedSize`.
2. Guarding against non-positive sizes so `bufferSize` retains a sensible
minimum (the default 1024 bytes) and capping at `Integer.MAX_VALUE - 8`.
3. Adding unit test coverage in `ValueAndCoderLazySerializableTest` for
cheap coders under both Java serialization and Kryo serialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
