potiuk commented on code in PR #56821:
URL: https://github.com/apache/airflow/pull/56821#discussion_r2462961546
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -2561,3 +2561,33 @@ dag_processor:
type: boolean
example: ~
default: "True"
+profiling:
+ description: |
+ Configuration for memory profiling in Airflow component.
+ Currently, we provide profiling using Memray and additional tools may be
added in the future
+ Also, see the guide in Link (TBD)
+ options:
+ memray_trace_enabled:
+ description: |
+ Whether to enable memory allocation tracing by memray in the
scheduler. If enabled, Airflow will
+ start tracing memory allocation and store the metrics in
"$AIRFLOW_HOME/dag_processor_memory.bin"
+ To generate analyzed view, run this command in base directory where
the bin file is generated
+ ```
+ # see also
https://bloomberg.github.io/memray/run.html#aggregated-capture-files
+ memray flamegraph $AIRFLOW_HOME/<component>_memory.bin
+ ```
+ This is an expensive operation and generally should not be used except
for debugging purposes.
+ version_added: 3.1.2
+ type: boolean
+ example: ~
+ default: "False"
+ memray_trace_components:
+ description: |
+ Comma-separated list of Airflow components to profile with memray.
+ Valid components are: scheduler, api, dag_processor
+
+ This option only takes effect when memray_trace_enabled is set to True.
+ version_added: 3.1.2
+ type: string
+ example: "scheduler,api,dag_processor"
+ default: ""
Review Comment:
Better indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
