xBis7 commented on PR #61897: URL: https://github.com/apache/airflow/pull/61897#issuecomment-3921349486
This is adding spans for airflow internal operations which don't provide any value to the users. The users need to be able to create sub-spans from under tasks so that they can track how long each step took, and optimize it or refactor it. For example, with the approach in `main`, we can know that a task took 10 minutes and out of these 10 minutes, 8 were spent in an individual step. And then we can start wondering how we can make it better. As each step finishes, the span gets exported and then we also know which part of the task has already been completed and where we are in the execution. I ran a basic test dag. This is from `main` <img width="2064" height="598" alt="image" src="https://github.com/user-attachments/assets/70fcde2b-815b-42d0-89cf-37f9769ab7e2"; /> This is with this patch, <img width="2042" height="712" alt="image" src="https://github.com/user-attachments/assets/b5a3a3b4-0031-47ac-bc55-67b24a26c752"; /> We can't create any spans from under tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
