GitHub user olmaks added a comment to the discussion: Task memory diagnostics in Airflow 3: measurement scopes and trade-offs
Thanks for writing this up. One data point on Q1 and Q2 from a listener I've been using for my project, meant to complement the V1 runner metric you outlined in #49983 (`task.runner_rss_bytes`). **On Q1, what the runner's RSS shows at startup.** I read `/proc/self/status` at `on_starting(TaskRunnerMarker())` (before the DAG file is loaded), at `on_task_instance_running` (after the task is prepared and its templates rendered) and at the terminal hook, on 3.3.1 (Linux in Docker, x86_64 and arm64, synthetic DAGs; medians over 10 runs per DAG unless noted): - Before the runner loaded its DAG file, its RSS was already about 137 MiB under LocalExecutor and 219 MiB under CeleryExecutor (arm64). A runner-RSS gauge includes that starting footprint; recording it would help separate it from the net change during startup. - A top-level `import pandas` raised RSS by 97 MiB over startup on x86_64 (84 MiB on arm64), against 2 MiB for a light DAG. In a separate breakdown test, about 82 of those 84 MiB happened between readings near the start and end of the DAG file. Under CeleryExecutor the startup increase was 54 MiB; that runner also began with more modules loaded, including numpy. - A task holding 128 MiB showed +128 MiB between the running and terminal hooks. **On Q2.** A log event per runner start, like #66138's "Worker startup parse complete", avoids sharing a metric series, but still needs its own identity: one try spanned two runner processes when a sensor deferred and three for a reschedule-mode sensor, all with the same `try_number`. I include host, PID and process start time alongside the task-instance fields. That shows successive producers within one try; it doesn't settle how concurrent gauge producers should be identified or aggregated. What these readings don't cover: - Endpoint differences miss memory allocated and freed in between: 256 MiB allocated and freed left about 0 RSS change while `VmHWM` rose, and a later 150 MiB allocation below an earlier 300 MiB peak changed neither the endpoint difference nor `VmHWM`. Periodic sampling can reveal such use, though it can still miss short spikes. - A runner killed by SIGKILL can't run its terminal listener hook (#60858 is the related callback issue). In three OOM runs (SIGKILL, cgroup `oom_kill` counter +1 each), the event logged at the running hook survived; a runner killed before that hook leaves no listener startup event. - Only the Python runner process is measured; subprocesses are excluded. If useful as a separate startup diagnostic, I'd be happy to open a small draft PR adding best-effort RSS readings around `parse()` to the existing "Worker startup parse complete" event, via psutil (already an SDK dependency). I'd validate those exact boundaries, `run_as_user` (where `parse()` runs twice) and failure paths separately from the listener measurements; the pandas breakdown suggests that narrower window is worth testing. I can also run any metric change through the same tests (deferral, reschedule, retries, OOM, Local and Celery). Listener, DAGs and results: https://gist.github.com/olmaks/c6831ab0f6f5a9cb909a58ad11a2f070 GitHub link: https://github.com/apache/airflow/discussions/73623#discussioncomment-19039673 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
