Bharath Palaksha created AIRFLOW-4527:
-----------------------------------------
Summary: Execution time out doesn't work after MySql Lost
connection while refreshing from db
Key: AIRFLOW-4527
URL: https://issues.apache.org/jira/browse/AIRFLOW-4527
Project: Apache Airflow
Issue Type: Bug
Reporter: Bharath Palaksha
{{I have setup airflow with mysql as metastore. When there is a network issue
and task fails with an exception, airflow tries to refresh status from db and
gets a connection error - This results in task getting stuck in running and
execution time out doesn't apply for this. }}
{{There is no retry for mysql connection error and it never fails the task.}}
Scheduler heartbeat got an exception: (MySQLdb._exceptions.OperationalError)
(2013, "Lost connection to MySQL server at 'reading authorization packet',
system error: 104") (Background on this error at: http://sqlalche.me/e/e3q8)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
Traceback (most recent call last):
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/bin/airflow", line 32, in <module>
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
args.func(args)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in
wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
return f(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 526, in
run
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
_run(args, dag, ti)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 445, in
_run
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
pool=args.pool,
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in
wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
return func(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1692, in
_run_raw_task
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
self.refresh_from_db()
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in
wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
return func(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1218, in
refresh_from_db
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data ti
= qry.first()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
