tejasiyer-dev opened a new pull request, #41468:
URL: https://github.com/apache/beam/pull/41468

   Split out from #40543 (addresses part of #40533).
   ### Changes
   - **`BundleLifter.withKeyFn`:** Evaluates the threshold separately per 
`(window, pane, key)` (or per bundle when unkeyed), streaming subsequent 
elements for a lifted key directly to the large output once the threshold is 
reached.
   - **Coder-based sizing (`BundleLifter.of(..., Coder<T>)`):** Estimates 
element byte sizes via `Coder.registerByteSizeObserver` without allocating byte 
arrays.
   - **Memory & key caps:**
     - `withMaxBufferedBytes` (defaults to `2 * threshold`) evicts the smallest 
buffered keys to the small output once exceeded and routes newly seen keys 
directly to the small output.
     - `withMaxLiftedKeys` bounds how many keys can be lifted in a single 
bundle.
   - Sets output coders on both output `PCollection`s and clears key buffers 
incrementally as elements are emitted.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to