Tanuj Gupta created AIRFLOW-3131:
------------------------------------
Summary: Start Date is not honoured in case of task is failed due
to Mysql connection failure
Key: AIRFLOW-3131
URL: https://issues.apache.org/jira/browse/AIRFLOW-3131
Project: Apache Airflow
Issue Type: Bug
Components: cli, db
Affects Versions: 1.9.0
Reporter: Tanuj Gupta
Sometimes, when we trigger some DAG then task's start sate is seen as null in
database. In this we have observed that "airflow run" command fails due to
mysql connection failure and in turn task state is set as FAILED but start date
is not updated in the database.
Traceback (most recent call last): File "/usr/src/venv/bin/airflow", line 27,
in <module> args.func(args) File
"/usr/src/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 387,
in run run_job.run() File
"/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 193, in
run id_ = self.id File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
line 242, in __get__ return self.impl.get(instance_state(instance), dict_) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
line 594, in get value = state._load_expired(state, passive) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line
608, in _load_expired self.manager.deferred_scalar_loader(self, toload) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 876, in load_scalar_attributes only_load_props=attribute_names) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 188, in load_on_ident identity_token=identity_token File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
line 250, in load_on_pk_identity return q.one() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2947, in one ret = self.one_or_none() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2917, in one_or_none ret = list(self) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2988, in __iter__ return self._execute_and_instances(context) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
3009, in _execute_and_instances close_with_result=True) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
3018, in _get_bind_args **kw File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
3000, in _connection_from_session conn = self.session.connection(**kw) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 1035, in connection execution_options=execution_options) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 1040, in _connection_for_bind engine, execution_options) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 409, in _connection_for_bind conn = bind.contextual_connect() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2123, in contextual_connect self._wrap_pool_connect(self.pool.connect,
None), File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2162, in _wrap_pool_connect e, dialect, self) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1476, in _handle_dbapi_exception_noconnection exc_info File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb,
cause=cause) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2158, in _wrap_pool_connect return fn() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403,
in connect return _ConnectionFairy._checkout(self) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 791,
in _checkout fairy = _ConnectionRecord.checkout(pool) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532,
in checkout rec = pool._do_get() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
1287, in _do_get return self._create_connection() File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350,
in _create_connection return _ConnectionRecord(self) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477,
in __init__ self.__connect(first_connect_check=True) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 674,
in __connect connection = pool._invoke_creator(self) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",
line 106, in connect return dialect.connect(*cargs, **cparams) File
"/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 412, in connect return self.dbapi.connect(*cargs, **cparams) File
"/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 86,
in Connect return Connection(*args, **kwargs) File
"/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line
204, in __init__ super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2003,
"Can't connect to MySQL server on 'orchmysql2ykgeqni.mysql.database.azure.com'
(110)") (Background on this error at: [http://sqlalche.me/e/e3q8)]
[2018-09-22 00:22:15,461] \{jobs.py:1425} ERROR - Executor reports task
instance %s finished (%s) although the task says its %s. Was the task killed
externally?
[2018-09-22 00:22:15,475] \{models.py:1595} ERROR - Executor reports task
instance %s finished (%s) although the task says its %s. Was the task killed
externally?
[2018-09-22 00:22:15,475] \{jobs.py:1435} ERROR - Cannot load the dag bag to
handle failure for <TaskInstance:
f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.ACPComputeGateway_0_score
2018-09-22 00:00:00 [queued]>. Setting task to FAILED without callbacks or
retries. Do you have enough resources?
[2018-09-22 00:22:13,408] \{local_executor.py:50} ERROR - Failed to execute
task Command 'exec bash -c 'airflow run
f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz
ACPComputeGateway_0_score 2018-09-22T00:00:00 --local -sd
/usr/local/airflow/dags/F75D1CCA5B17165C0A495C1F/f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.py''
returned non-zero exit status 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
