uranusjr commented on issue #22404: URL: https://github.com/apache/airflow/issues/22404#issuecomment-1075245303
Finalisers in Python are somewhat known to not always work, unfortunately, which is why context manager is a thing. Since Python objects can contain non-transparent reference cycles, it is not guaranteed when things are finalised, or even at all. In this particular situation, my guess is that the exception raised in the function is causing a reference cycle. An exception instance holds the frame object of the current function, which references the local variables. But Airflow likely catches the exception somewhere (for logging, for example), which prevents the exception from being freed, which in turn holds on the `TemporaryDirectory` object. This is a wild guess and can be very far off, but my main point is that relying on scoping to clean things up is unreliable by design (of the Python langauge), and it shouldn’t be too surprising when it does not work, unfortunately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
