Hi Alex,
Maybe I'm stating the obvious but if any executors catches Airflow exceptions
to do cleanup before exiting they can still do that with SystemExit. E.g.:
class AirflowSystemExit(SystemExit):
pass
try:
raise AirflowSystemExit
except AirflowSystemExit:
print('cleanup for airflow throwing system exit')
finally:
print('cleanup for any type of exit')
In fact if people use either:
try:
...
except:
...
Or
try:
...
finally:
...
Their code will still run on this type of exit, but in the case of 1) this can
at least be put down to poor python code and case 2) any code in a finally
statement shouldn't be written so it can cause harm on exiting.
Damian
-----Original Message-----
From: Alex Guziel [mailto:alex.guz...@airbnb.com.INVALID]
Sent: Wednesday, October 02, 2019 4:03 PM
To: dev@airflow.apache.org
Subject: Re: How to manage Airflow SIGTERM excetion catches?
Task_copy.on_kill() should probably be killing the underlying process, but
I think it's fuzzy where the exception gets thrown. I think the intention
is for the exception to get caught in that same block, so the cleanup can
happen, but this is not the case since it is thrown in the main thread. I
think the behavior depends a little bit on which executor you use. We
probably need to think about the right behavior here that works for a
variety of executors. SystemExit works for things that don't use
subprocess, but I'm not sure what the behavior for cleanup is when running
subprocess.
===============================================================================
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
