howardyoo commented on PR #39650: URL: https://github.com/apache/airflow/pull/39650#issuecomment-2143453554
> I think (@howardyoo - @ferruzzi can you confirm?) the addition of traces, should make all the resource inormation automatically available if you enable it via Open-Telemetry (and traces will link the metrics about resources to tasks/dags automatically). From what I know OTEL has a way to enable all the "system"/ "python" etc. metrics out-of-the-box and the "traces" addition, shoudl (IMHO) label such metrics with appropriate labels for Airlfow "logical" tags - i.e. dags/task etc. > > See #37948 > > But maybe I am too optimistic there :) ? OpenTelemetry for Python SDK does provide 'auto-instrumentation' feature where it can automatically detect and produce traces, but those will not include metrics like cpu, memory usage, fs i/o, net i/o, processes, etc. However, we can definitely implement those as additional instrumentation, utilizing psutil package. It would also be very helpful if these metrics could also become part of the trace attribute, such that trace could also contain these as either values or span events, as needed, since when these metrics are being produced, they would highly likely be correlated with the task's execution - thus makes sense to have them existing during the task's duration. My concern is that for some monitoring tools, this may introduce a high cardinality (as each individual task runs can be defined as independent sources for some tools), so we might want to have this turned on / off as part of the configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
