Antoine Grouazel created AIRFLOW-1347:
-----------------------------------------
Summary: sqlalchemy.orm.exc.FlushError: Can't update table
task_instance using NULL for primary key
Key: AIRFLOW-1347
URL: https://issues.apache.org/jira/browse/AIRFLOW-1347
Project: Apache Airflow
Issue Type: Bug
Components: DagRun, db
Affects Versions: 1.8.0
Environment: ubuntu 12.04
mysql 5.7.18
Reporter: Antoine Grouazel
Description of the issue: the scheduler cannot update the mysql db via
SqlAlchemy and the tasks are queued and not rescheduled. It affects all our
DAGs but not all the tasks nor all the time. Manually clear a task having such
issue can fix the problem for the given task instance.
There is a kind of random behavior...
Upgrading mysql 5.7.17 to 5.7.18 does not fix the issue.
Issue is neitheir fixed in the lastest version 1.8.2rc1 .
Don't know if there is a link with
https://issues.apache.org/jira/browse/AIRFLOW-977.
{{[2017-06-24 00:31:10,272] {models.py:167} INFO - Filling up the DagBag from
/home3/homedir7/perso/satwave/airflow/dags/sentinel1_copernicus_download.py
[2017-06-24 00:31:10,427] {base_task_runner.py:112} INFO - Running: ['bash',
'-c', u'airflow run Sentinel1_copernicus_download latest_only
2017-06-23T00:31:00 --job_id 293 --raw -sd
DAGS_FOLDER/sentinel1_copernicus_download.py']
[2017-06-24 00:31:11,449] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:11,449] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-06-24 00:31:11,668] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:11,668] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/Grammar.txt
[2017-06-24 00:31:11,724] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:11,723] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-06-24 00:31:12,067] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:12,066] {models.py:167} INFO - Filling up the DagBag from
/home3/homedir7/perso/satwave/airflow/dags/sentinel1_copernicus_download.py
[2017-06-24 00:31:12,184] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:12,184] {models.py:1126} INFO - Dependencies all met for <TaskInstance:
Sentinel1_copernicus_download.latest_only 2017-06-23 00:31:00 [queued]>
[2017-06-24 00:31:12,188] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:12,188] {models.py:1126} INFO - Dependencies all met for <TaskInstance:
Sentinel1_copernicus_download.latest_only 2017-06-23 00:31:00 [queued]>
[2017-06-24 00:31:12,188] {base_task_runner.py:95} INFO - Subtask: [2017-06-24
00:31:12,188] {models.py:1318} INFO -
[2017-06-24 00:31:12,188] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-06-24 00:31:12,188] {base_task_runner.py:95} INFO - Subtask: Starting
attempt 1 of 3
[2017-06-24 00:31:12,189] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-06-24 00:31:12,189] {base_task_runner.py:95} INFO - Subtask:
[2017-06-24 00:31:12,228] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-06-24 00:31:12,228] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/bin/airflow", line 28, in <module>
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask:
args.func(args)
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 422, in run
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask:
pool=args.pool,
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in
wrapper
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: result =
func(*args, **kwargs)
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1328, in run
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask:
session.commit()
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 801,
in commit
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask:
self.transaction.commit()
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392,
in commit
[2017-06-24 00:31:12,229] {base_task_runner.py:95} INFO - Subtask:
self._prepare_impl()
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372,
in _prepare_impl
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask:
self.session.flush()
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019,
in flush
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask:
self._flush(objects)
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137,
in _flush
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask:
transaction.rollback(_capture_exception=True)
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line
60, in __exit__
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask:
compat.reraise(exc_type, exc_value, exc_tb)
[2017-06-24 00:31:12,230] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2101,
in _flush
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask:
flush_context.execute()
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line
373, in execute
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask:
rec.execute(self)
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line
532, in execute
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: uow
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line
170, in save_obj
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: mapper,
table, update)
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line
634, in _emit_update_statements
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: lambda
rec: (
[2017-06-24 00:31:12,231] {base_task_runner.py:95} INFO - Subtask: File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line
531, in _collect_update_commands
[2017-06-24 00:31:12,232] {base_task_runner.py:95} INFO - Subtask: "key
value on column %s" % (table, col))
[2017-06-24 00:31:12,232] {base_task_runner.py:95} INFO - Subtask:
sqlalchemy.orm.exc.FlushError: Can't update table task_instance using NULL for
primary key value on column task_instance.execution_date
[2017-06-24 00:31:12,311] {jobs.py:2083} INFO - Task exited with return code 1}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
