stankiewicz opened a new pull request, #41394: URL: https://github.com/apache/beam/pull/41394
CombiningValueRuntimeState previously lacked an in-memory per-bundle accumulator: add() read and rewrote the underlying BagState on 50% of calls (random.random() < 0.5), and read() re-read, merged, cleared, and rewrote the underlying BagState on every call. In transforms such as GroupIntoBatches and stateful BatchElements that call count_state.add(1) followed by count_state.read() on every element, this resulted in 1.5 underlying BagState reads and rewrites per element and defeated blind writes on multi-element bundles. Maintain a per-bundle _pending_accumulator in CombiningValueRuntimeState so add() accumulates in memory during the bundle, read() folds the persisted bag state at most once per bundle, and commit() flushes to the underlying BagState once at the end of the bundle (compacting blind writes with 2% probability). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
