stankiewicz opened a new pull request, #41394:
URL: https://github.com/apache/beam/pull/41394

   CombiningValueRuntimeState previously lacked an in-memory per-bundle 
accumulator: add() read and rewrote the underlying BagState on 50% of calls 
(random.random() < 0.5), and read() re-read, merged, cleared, and rewrote the 
underlying BagState on every call. In transforms such as GroupIntoBatches and 
stateful BatchElements that call count_state.add(1) followed by 
count_state.read() on every element, this resulted in 1.5 underlying BagState 
reads and rewrites per element and defeated blind writes on multi-element 
bundles.
   
   Maintain a per-bundle _pending_accumulator in CombiningValueRuntimeState so 
add() accumulates in memory during the bundle, read() folds the persisted bag 
state at most once per bundle, and commit() flushes to the underlying BagState 
once at the end of the bundle (compacting blind writes with 2% probability).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to