uditjainstjis opened a new pull request, #39466: URL: https://github.com/apache/beam/pull/39466
Addresses #39244 ## What Expands the SDK worker heap dump (enabled by `--experiments=enable_heap_dump`) with a new `memory_stats()` section so operators can distinguish memory growth on the **native (C) heap** from **Python-object** allocations, and can spot native-heap fragmentation — the two things the issue asks for. The dump previously contained only the guppy Python-object heap. `memory_stats()` adds: - **Process memory** — peak RSS (`ru_maxrss`, unit-corrected: bytes on macOS/BSD, KiB on Linux) and, on Linux, current RSS from `/proc/self/statm`. - **Python allocations** — `sys.getallocatedblocks()` and per-generation `gc` collection stats. - **glibc malloc (native heap)** — `mallinfo2()` `arena`/`hblkhd`/`uordblks`/`fordblks`/`keepcost`, plus a `fordblks/(arena+hblkhd)` fragmentation ratio. `fordblks` (free space the allocator retains rather than returning to the OS) growing relative to `uordblks` is the fragmentation signal referenced in the issue. `mallinfo2` is used rather than the legacy `mallinfo`, whose `int` fields overflow past 2GB and would misreport exactly when memory is the concern; if the symbol is absent it reports `unavailable (needs glibc >= 2.33)`. No new dependencies — stdlib only (`resource`, `gc`, `sys`, `ctypes`, `/proc`). Every collector degrades gracefully and never raises when a source is unavailable, and `memory_stats()` is emitted even when guppy is not importable. ## Example output (Linux) ``` ========== MEMORY STATS ========== --- Process memory --- peak RSS (ru_maxrss): 116162560 bytes current RSS (/proc/self/statm): 98533376 bytes --- Python allocations --- sys.getallocatedblocks: 423303 gc.get_count (gen0, gen1, gen2): (635, 2, 9) gc gen0: collections=222 collected=992 uncollectable=0 ... --- glibc malloc (native heap) --- arena (non-mmapped bytes from sbrk): 999424 hblkhd (mmapped bytes): 401408 uordblks (in-use bytes): 874320 fordblks (free bytes retained by allocator): 125104 keepcost (releasable top-most free bytes): 120976 fragmentation (fordblks / (arena + hblkhd)): 8.93% ============================== ``` ## Testing - New unit tests in `worker_status_test.py`: section presence, memory stats emitted without guppy, and graceful degradation of the glibc section on non-glibc platforms. - Full `worker_status_test.py` passes locally (9 passed). - The `mallinfo2` ctypes path (by-value struct `restype`) was executed on real Linux glibc 2.39 to confirm the struct layout and that it does not segfault; values are internally consistent (`uordblks + fordblks == arena`). - `yapf==0.43.0` clean, pylint 10.00/10. This is a first cut; happy to also wire the section outside the guppy-gated dump or add `malloc_info()` XML if maintainers prefer. - [x] Update `CHANGES.md` with noteworthy changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
