omkar-foss commented on issue #40975: URL: https://github.com/apache/airflow/issues/40975#issuecomment-2248091950
I'd like to help out. Feel free to call me out wherever required. The debug snapshot idea sounds very useful @potiuk. It may give a canonical view of the user's environment. I suppose Jaeger provides a similar tool called [Anonymizer](https://www.jaegertracing.io/docs/1.58/tools/#anonymizer), which generates a shareable json of a trace - probably same one that you were referring to in your mail. Either we can have our own version of the debug snapshot, or can think of using this tool with Jaeger since it supports the existing OTEL [metrics](https://github.com/apache/airflow/blob/main/airflow/metrics/otel_logger.py) and [traces](https://github.com/apache/airflow/blob/main/airflow/traces/otel_tracer.py). Regarding logging, currently all logs seem to be using the `info` severity level. We can make them more readable by using other severity levels (`error`, `warning`, `critical`, `debug`) and color-coding as per severity. And of course yes, by reducing verbosity where possible. Might be a bit far-fetched, but we can also provide capability to push the logs to Logstash, Fluentd etc. for custom analysis or alerting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
