Actually, reading the docs, the handler throws it in the main thread. In
that case we should definitely change it to subclass SystemExit, or just
use System.exit
On Wed, Oct 2, 2019 at 12:53 PM Alex Guziel <alex.guz...@airbnb.com> wrote:
> It's been a while since I've looked at this code, but the exception thrown
> there is thrown from a place where it should not be able to be caught by
> your operator code, so the issue may be somewhere else.
>
> On Wed, Oct 2, 2019 at 12:41 PM Shaw, Damian P. <
> damian.sha...@credit-suisse.com> wrote:
>
>> Today I had an issue where my operator caught the SIGTERM exception that
>> Airflow throws. It seems to me that Airflow shouldn't be throwing a SIGTERM
>> exception that subclasses Exception, particular as in arbitrary Python
>> functions this might happen commonly.
>>
>> The code in my operator looked like this:
>> try:
>> optimistic_function()
>> except Exception:
>> self.log.exception('Optimistic function failed: ')
>>
>> When I terminated the task instance early in the Airflow GUI the
>> exception was caught and the task carried on as normal:
>> [2019-10-02 15:10:39,153] {my_operators.py:415} ERROR - Something went
>> wrong with checksum validation:
>> Traceback (most recent call last):
>> ...
>> File ".../python3.7/site-packages/airflow/models/__init__.py", line
>> 1418, in signal_handler
>> raise AirflowException("Task received SIGTERM signal")
>> airflow.exceptions.AirflowException: Task received SIGTERM signal
>> [2019-10-02 15:10:39,177] {my_operators.py:428} INFO - Carrying on with
>> the operator like everything is good
>>
>> The above is from Airflow 1.10.3, on master branch the signal_handler
>> code is located here:
>> https://github.com/apache/airflow/blob/d719e1fd6705a93a0dfefef4b46478ade5e006ea/airflow/models/taskinstance.py#L898
>>
>> And the exception it throws is located here:
>> https://github.com/apache/airflow/blob/d719e1fd6705a93a0dfefef4b46478ade5e006ea/airflow/exceptions.py#L25
>>
>> It seems to me that if Airflow is throwing an exception with the intent
>> to kill the Python process it should be subclassing "SystemExit" not
>> "Exception":
>> https://docs.python.org/3/library/exceptions.html#exception-hierarchy .
>> In the mean time I will rewrite my code for special handling when receiving
>> AirflowException.
>>
>> Would everyone agree in general this is a bug or am I missing some use
>> case?
>>
>> Damian
>>
>>
>>
>>
>> ===============================================================================
>>
>> Please access the attached hyperlink for an important electronic
>> communications disclaimer:
>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>> ===============================================================================
>>
>>
>
