mobuchowski commented on PR #27113: URL: https://github.com/apache/airflow/pull/27113#issuecomment-1305596298
@uranusjr we have some problems with how `task instance state listener` works. One is that the Airflow object we get from `SqlAlchemy` event mechanism can be different from the one where it's executed. For example https://github.com/apache/airflow/pull/27167 https://github.com/apache/airflow/pull/21157 Another problem is that in `task instance state listener` we're getting notified about state change in `after_flush` - which is the only `SqlAlchemy` event that has the database changes we need and occurs after actual execution. The problem is that `database transaction` haven't yet finished, so we need to run our code in a separate thread. The worker then can exit by `os._exit()` so we can't introduce any more sophisticated mechanism to manage that. One of the problems here is this Snowflake issue: https://github.com/OpenLineage/OpenLineage/issues/1226 One solution for threading issues in `task instance state listener` on Airflow side would be to split the listener to listen to two events: first would get actual changes and cache them locally, second would notify plugin about those changes after database transaction ended. I think it's more risky solution for scheduler, since in contrast to worker process, where there is only one task instance run at one time, scheduler could cache a lot more data. And still, this would not solve the first issue about differing objects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
