tejasiyer-dev commented on PR #40543: URL: https://github.com/apache/beam/pull/40543#issuecomment-6786268693
#40978 (moving BundleLifter from Iceberg to sdk/util) has merged, so I've updated this PR to use it. **Per-WriterKey threshold:** BundleLifter now takes an optional key function, and WriteFiles uses it to check the threshold per window/pane/destination (WriterKey). Without a key function, BundleLifter still checks the threshold against the whole bundle. Iceberg can also check the threshold per table/partition instead of per bundle by passing a key function but I've left its threshold behavior unchanged for now. Maybe it can be incorporated in a follow up PR. **Streaming:** Once a key reaches the threshold, its buffered elements are flushed to the large output, and later elements for that key are output directly without sizing or buffering. @arunpandianp @ahmedabu98 @scwhittle PTAL, and could someone also approve the workflows and merge afterwards? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
