stankiewicz commented on code in PR #41394: URL: https://github.com/apache/beam/pull/41394#discussion_r4626874337
########## sdks/python/apache_beam/runners/worker/bundle_processor.py: ########## @@ -560,6 +560,9 @@ def commit(self) -> None: class CombiningValueRuntimeState(userstate.CombiningValueRuntimeState): + # Probability of compacting persisted bag accumulators on blind-write commits. + _BLIND_WRITE_COMPACT_PROBABILITY = 0.02 Review Comment: good point. this was arbitrary number that was better than 50%. it's for situation where you have long window (e.g. 10 minutes) over which you're adding new values, periodically, and eventually when you get value, you will receive too many elements on reduce side, as they haven't been compacted as they were blindlly written (e.g. single value per bundle). 50% was too aggressive. 0% is risky. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
