GitHub user PrithwishDas101 added a comment to the discussion: Task memory diagnostics in Airflow 3: measurement scopes and trade-offs
I think the most important distinction here is between a metric that's useful for diagnostics and one that's reliable for comparing memory consumption across task executions. For a first implementation, I'd favor the narrowly defined runner RSS diagnostic, provided its limitations are explicit. It seems like a useful starting point without requiring Airflow to solve cross-executor aggregation and workload-level accounting immediately. A few things to preserve in the design: -Keep the measurement boundary explicit.** Runner RSS should not be presented as total task memory, especially when operators launch subprocesses or external runtimes. -Avoid ambiguous aggregation.** Concurrent executions of the same task shouldn't silently overwrite or conflate one another's measurements. If aggregation is needed, its ownership and semantics should be defined separately. -Treat observability as best-effort.** Sampling or exporting metrics should never interfere with task execution, retries, or cleanup. -Keep historical peaks distinct from periodic samples.** A periodic RSS sampler can miss short-lived memory spikes, so a sampled distribution shouldn't be presented as the distribution of actual per-execution peaks. I'd suggest starting with one precisely documented metric and validating its lifecycle and concurrency behavior across the supported execution paths. That would give users an immediate diagnostic signal while leaving room for a separate workload-level metric if there's enough demand. The key is making the metric's meaning consistent and trustworthy before expanding its scope. GitHub link: https://github.com/apache/airflow/discussions/73623#discussioncomment-18946361 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
