tejasiyer-dev opened a new pull request, #41468:
URL: https://github.com/apache/beam/pull/41468
Split out from #40543 (addresses part of #40533).
### Changes
- **`BundleLifter.withKeyFn`:** Evaluates the threshold separately per
`(window, pane, key)` (or per bundle when unkeyed), streaming subsequent
elements for a lifted key directly to the large output once the threshold is
reached.
- **Coder-based sizing (`BundleLifter.of(..., Coder<T>)`):** Estimates
element byte sizes via `Coder.registerByteSizeObserver` without allocating byte
arrays.
- **Memory & key caps:**
- `withMaxBufferedBytes` (defaults to `2 * threshold`) evicts the smallest
buffered keys to the small output once exceeded and routes newly seen keys
directly to the small output.
- `withMaxLiftedKeys` bounds how many keys can be lifted in a single
bundle.
- Sets output coders on both output `PCollection`s and clears key buffers
incrementally as elements are emitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]