[
https://issues.apache.org/jira/browse/AIRFLOW-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Palaksha updated AIRFLOW-4527:
--------------------------------------
Summary: Connection error while calling refreshfromdb() makes the task
stuck in running state (was: Execution time out doesn't work after MySql Lost
connection while refreshing from db)
> Connection error while calling refreshfromdb() makes the task stuck in
> running state
> ------------------------------------------------------------------------------------
>
> Key: AIRFLOW-4527
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4527
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Bharath Palaksha
> Priority: Major
>
> {{I have setup airflow with mysql as metastore. When there is a network issue
> and task fails with an exception, airflow tries to refresh status from db and
> gets a connection error - This results in task getting stuck in running and
> execution time out doesn't apply for this. }}
> {{There is no retry for mysql connection error and it never fails the task.}}
>
> Scheduler heartbeat got an exception: (MySQLdb._exceptions.OperationalError)
> (2013, "Lost connection to MySQL server at 'reading authorization packet',
> system error: 104") (Background on this error at: http://sqlalche.me/e/e3q8)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> Traceback (most recent call last):
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/bin/airflow", line 32, in <module>
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> args.func(args)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74,
> in wrapper
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> return f(*args, **kwargs)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 526,
> in run
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> _run(args, dag, ti)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 445,
> in _run
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> pool=args.pool,
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73,
> in wrapper
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> return func(*args, **kwargs)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1692,
> in _run_raw_task
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> self.refresh_from_db()
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73,
> in wrapper
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> return func(*args, **kwargs)
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1218,
> in refresh_from_db
> {base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data
> ti = qry.first()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
