nchint opened a new issue, #39754: URL: https://github.com/apache/beam/issues/39754
### What would you like to happen? `PGBKCVOperation` (`sdks/python/apache_beam/runners/worker/operations.py`) caps its precombine table by key count (`max_keys`, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap: ```python # TODO(b/36567833): Bound by in-memory size rather than key count. ``` A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below `max_keys` (or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count. Notes: - `b/36567833` is a Google-internal Buganizer ref with no public page; this issue tracks it publicly. - Sibling `PGBKOperation` has the same shape (`max_size = 10 * 1000` elements). - Sizing accurately is the hard part — `sys.getsizeof` undercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed. ### Issue Priority Priority: 3 (nice-to-have improvement) ### Issue Components - Component: Python SDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
