m1racoli edited a comment on issue #22404:
URL: https://github.com/apache/airflow/issues/22404#issuecomment-1075163309
Yeah, I know that when using a context manager the problem doesn't occur, as
the context manager explicitly cleans up the directory upon completion.
And yes, running
```python
import os
import tempfile
tmpdir = tempfile.TemporaryDirectory()
print(f"directory {tmpdir.name} created")
assert os.path.exists(tmpdir.name)
raise Exception("error!")
```
directly in Python cleans up the directory (done by the finalizer as @potiuk
said).
So now I wonder what's different in Airflow. My judgement is based on the
assumption that each task is executed in it's own process and therefore also
it's own Python process. Thus a failing task's Python process should clean up
the directory too (unless Airflow messes with the finalizers). Maybe the child
Python process knows it's a child of a parent process and doesn't invoke the
finalizer (pure speculation). 🤔
> The original one does not (i.e. the folder remains after Exception is
thrown and even after scheduler interpreter is stopped
This ☝🏼 makes the issue even more interesting.
I'm not an expert of the Python runtime and I did not verify (yet) if this
also occurs when running in a sub-process. I initially was hoping that someone
on Airflow's side knows what's going on. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
