potiuk commented on issue #20774: URL: https://github.com/apache/airflow/issues/20774#issuecomment-1011957388
Cool! I think what also will need to be checked (demoed) if all of them are working. I expect some problems similar to the one we had with gunicorn because forks are used in a few places: 1) when we run tasks, by default we are forking the tasks - so some of the processes (Especially when we run dags) might not really report all the instrumentaiton events - this might be checked by running airflow with example dags and running those dags and seeing if some of those (logging, sqlalchemy possibly) tasks when executing a DAG will also work. ``./breeze start-airflow --backend postgres --load-example-dags` will load all the example DAGS and you wll be able to run the dags and see the open-telemetry events they will generate. next week we can have a 1-1 session and we can walk-thtrough them together 2) Also Celery tests might be tricky - this one is a bit more difficult to test and for that I think you would have to take a closer look at the https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html - because this is the easiest way to run the "Celery Executor" in airflow. For that however you will have to modify the "quick-start" docker compose and add jaeger there (which is also an interesting part) - but we can also talk about it next week :). I think also we need to figure out how we can "selectively" enable/disable the instrumentations. Problem with having all the instrumentations enabled is that they - potentially - impact performance of Airflow. And I can imagine some users might want to be able to enable only some of the instrumentations. For now I understand it's just "installing" the package that enables the instrumentation but I think we need to figure out if it is possible to enable/disable certain instrumentations without installing/uninstalliung a package - some configuration option ? I am not sure - maybe it is already part of the OTEL variables? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
