stankiewicz commented on code in PR #41394:
URL: https://github.com/apache/beam/pull/41394#discussion_r4626874337


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -560,6 +560,9 @@ def commit(self) -> None:
 
 
 class CombiningValueRuntimeState(userstate.CombiningValueRuntimeState):
+  # Probability of compacting persisted bag accumulators on blind-write 
commits.
+  _BLIND_WRITE_COMPACT_PROBABILITY = 0.02

Review Comment:
   good point. this was arbitrary number that was better than 50%. 
   it's for situation where you have long window (e.g. 10 minutes) over which 
you're adding new values, periodically, and eventually when you get value, you 
will receive too many elements on reduce side, as they haven't been compacted 
as they were blindlly written (e.g. single value per bundle).  
   50% was too aggressive. 0% is risky. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to