potiuk commented on issue #12771: URL: https://github.com/apache/airflow/issues/12771#issuecomment-1383856897
> Hi folks! Just curious: is there a way we could auto instrument Airflow through the use of cluster policies? For example, what if we created a task policy that a wraps the callback and execute methods of a task? The wrapper could perform actions similar to the [Celery](https://github.com/uber-common/opentracing-python-instrumentation/blob/master/opentracing_instrumentation/client_hooks/celery.py) example above. You could use cluster policies, yes. But also you have (in Airlfow 2.5.0) listeners that you could use for that purpose. https://airflow.apache.org/docs/apache-airflow/stable/listeners.html - they were designed for a different purpose, but I believe you can go a long way by utilising these two mechanisms if you think the "standard" approach of Airlfow will not be good enough for you. Airlow in it's "2" line (now at 2.5) is more and more "platform" that you can extend in various ways and we are approaching it in the way that you can write your own extensions by plugging in your code with well-defined APIs in the way that will allow you to customize it heavily, while relying on the fact that you will be able to upgrade Airflow without losing your customizations. We have a few documentation updates coming in 2.5.1 specifically targetting to make it clearer what you CAN and what you CAN't modify in this way. Both cluster policies and listeners are "public interface" of Airlow and you can utilize them for that purpose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
