xBis7 commented on PR #61897: URL: https://github.com/apache/airflow/pull/61897#issuecomment-3943745508
@nickstenning I've gone over the example. I'm a bit skeptical about the ID generation but possibly there won't be any issues if it's just for the dag_run span. If we decide to go through with this, I've already added a `context_carrier` column in the dag_run table for storing the context and retrieving it later for new sub-spans. We don't have to create new columns. https://github.com/astronomer/airflow/blob/add-spans-for-task-execution/airflow-core/src/airflow/models/dagrun.py#L223-L226 Based on previous discussions with @ashb, the integration tests in `test_otel` https://github.com/astronomer/airflow/blob/add-spans-for-task-execution/airflow-core/tests/integration/otel/test_otel.py are all possible scenarios that we need to account for. We need to make sure that we know what is happening with the spans and that the results are intuitive and user friendly. The scenarios are * Same scheduler for the entire dag_run (straight-forward) * Scheduler changes in the middle of the dag_run execution * the initial scheduler is still alive and healthy * Scheduler exits gracefully in the middle of the dag_run execution * the shutdown hooks run * Scheduler exits forcefully in the middle of the dag_run execution * something happened and the process was killed, no cleanup took place Let's assume that we create a span using a context manager, which is what this PR does. If the process that started it, exits, then the context manager will automatically end the span. What happens visually in that case? Everything created afterwards will be out of place. The audience (users) won't make much sense of it. So, we either recreate the span with a synthetic one or somehow understand that the parent process changed and create a new one and make it the new parent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
