[
https://issues.apache.org/jira/browse/AIRFLOW-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aizhamal Nurmamat kyzy updated AIRFLOW-3131:
--------------------------------------------
Component/s: (was: db)
(was: cli)
database
> Start Date is not honoured in case of task is failed due to Mysql connection
> failure
> ------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3131
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
> Affects Versions: 1.9.0
> Reporter: Tanuj Gupta
> Priority: Minor
>
> Sometimes, when we trigger some DAG then task's start sate is seen as null in
> database. In this we have observed that "airflow run" command fails due to
> mysql connection failure and in turn task state is set as FAILED but start
> date is not updated in the database.
>
> Traceback (most recent call last): File "/usr/src/venv/bin/airflow", line 27,
> in <module> args.func(args) File
> "/usr/src/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line
> 387, in run run_job.run() File
> "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 193,
> in run id_ = self.id File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
> line 242, in __get__ return self.impl.get(instance_state(instance), dict_)
> File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py",
> line 594, in get value = state._load_expired(state, passive) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 608, in _load_expired self.manager.deferred_scalar_loader(self, toload)
> File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 876, in load_scalar_attributes only_load_props=attribute_names) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 188, in load_on_ident identity_token=identity_token File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py",
> line 250, in load_on_pk_identity return q.one() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2947, in one ret = self.one_or_none() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2917, in one_or_none ret = list(self) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2988, in __iter__ return self._execute_and_instances(context) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3009, in _execute_and_instances close_with_result=True) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3018, in _get_bind_args **kw File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3000, in _connection_from_session conn = self.session.connection(**kw)
> File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 1035, in connection execution_options=execution_options) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 1040, in _connection_for_bind engine, execution_options) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 409, in _connection_for_bind conn = bind.contextual_connect() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 2123, in contextual_connect self._wrap_pool_connect(self.pool.connect,
> None), File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 2162, in _wrap_pool_connect e, dialect, self) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1476, in _handle_dbapi_exception_noconnection exc_info File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
> line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb,
> cause=cause) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 2158, in _wrap_pool_connect return fn() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 403, in connect return _ConnectionFairy._checkout(self) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 791, in _checkout fairy = _ConnectionRecord.checkout(pool) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 532, in checkout rec = pool._do_get() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 1287, in _do_get return self._create_connection() File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 350, in _create_connection return _ConnectionRecord(self) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 477, in __init__ self.__connect(first_connect_check=True) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
> 674, in __connect connection = pool._invoke_creator(self) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",
> line 106, in connect return dialect.connect(*cargs, **cparams) File
> "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
> line 412, in connect return self.dbapi.connect(*cargs, **cparams) File
> "/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line
> 86, in Connect return Connection(*args, **kwargs) File
> "/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py",
> line 204, in __init__ super(Connection, self).__init__(*args, **kwargs2)
> sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2003,
> "Can't connect to MySQL server on
> 'orchmysql2ykgeqni.mysql.database.azure.com' (110)") (Background on this
> error at: [http://sqlalche.me/e/e3q8)]
>
> [2018-09-22 00:22:15,461] \{jobs.py:1425} ERROR - Executor reports task
> instance %s finished (%s) although the task says its %s. Was the task killed
> externally?
> [2018-09-22 00:22:15,475] \{models.py:1595} ERROR - Executor reports task
> instance %s finished (%s) although the task says its %s. Was the task killed
> externally?
> [2018-09-22 00:22:15,475] \{jobs.py:1435} ERROR - Cannot load the dag bag to
> handle failure for <TaskInstance:
> f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.ACPComputeGateway_0_score
> 2018-09-22 00:00:00 [queued]>. Setting task to FAILED without callbacks or
> retries. Do you have enough resources?
>
> [2018-09-22 00:22:13,408] \{local_executor.py:50} ERROR - Failed to execute
> task Command 'exec bash -c 'airflow run
> f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz
> ACPComputeGateway_0_score 2018-09-22T00:00:00 --local -sd
> /usr/local/airflow/dags/F75D1CCA5B17165C0A495C1F/f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.py''
> returned non-zero exit status 1.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
