kaxil edited a comment on issue #13099:
URL: https://github.com/apache/airflow/issues/13099#issuecomment-767163927
I was not able to reproduce it with Postgres 9.6 and 12, haven't tried it
with 13.1
**EDIT**: I was able to reproduce it with 2.0.0 and Sqlite
```
❯ airflow scheduler
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-01-26 12:46:09,634] {scheduler_job.py:1241} INFO - Starting the
scheduler
[2021-01-26 12:46:09,634] {scheduler_job.py:1246} INFO - Processing each
file at most -1 times
[2021-01-26 12:46:09,639] {dag_processing.py:250} INFO - Launched
DagFileProcessorManager with pid: 4646
[2021-01-26 12:46:09,640] {scheduler_job.py:1751} INFO - Resetting orphaned
tasks for active dag runs
[2021-01-26 12:46:09,645] {settings.py:52} INFO - Configured default
timezone Timezone('UTC')
[2021-01-26 12:46:09,657] {scheduler_job.py:1805} INFO - Reset the following
3 orphaned TaskInstances:
<TaskInstance: scenario1_case2_01.tasks__1_of_10 2021-01-25
00:00:00+00:00 [queued]>
<TaskInstance: scenario4_case2_1_40.tasks__1_of_10 2021-01-25
00:00:00+00:00 [queued]>
<TaskInstance: scenario1_case2_02.tasks__2_of_10 2021-01-25
00:00:00+00:00 [scheduled]>
[2021-01-26 12:46:09,661] {dag_processing.py:518} WARNING - Because we
cannot use more than 1 thread (parsing_processes = 2 ) when using sqlite. So we
set parallelism to 1.
[2021-01-26 12:46:09,845] {plugins_manager.py:231} ERROR - UNEXPECTED COMMIT
- THIS WILL BREAK HA LOCKS!
Traceback (most recent call last):
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/plugins_manager.py",
line 222, in load_plugins_from_plugin_directory
loader.exec_module(mod)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File "/Users/kaxilnaik/airflow/plugins/test.py", line 3, in <module>
print(Variable.get('test'))
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/models/variable.py",
line 123, in get
var_val = Variable.get_variable_from_secrets(key=key)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/models/variable.py",
line 181, in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/session.py",
line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/Users/kaxilnaik/opt/anaconda3/lib/python3.7/contextlib.py", line
119, in __exit__
next(self.gen)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/session.py",
line 32, in create_session
session.commit()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 1042, in commit
self.transaction.commit()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 504, in commit
self._prepare_impl()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 472, in _prepare_impl
self.session.dispatch.before_commit(self.session)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/event/attr.py",
line 322, in __call__
fn(*args, **kw)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/sqlalchemy.py",
line 217, in _validate_commit
raise RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!")
RuntimeError: UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!
[2021-01-26 12:46:09,853] {plugins_manager.py:232} ERROR - Failed to import
plugin /Users/kaxilnaik/airflow/plugins/test.py
... followed by
[2021-01-26 13:07:54,518] {scheduler_job.py:1293} ERROR - Exception when
executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1277, in _execute_context
cursor, statement, parameters, context
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 593, in do_execute
cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: dag_run.dag_id,
dag_run.run_id
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1275, in _execute
self._run_scheduler_loop()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1377, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1474, in _do_scheduling
self._create_dag_runs(query.all(), session)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1582, in _create_dag_runs
creating_job_id=self.id,
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/session.py",
line 62, in wrapper
return func(*args, **kwargs)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/airflow/models/dag.py",
line 1781, in create_dagrun
session.flush()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2523, in flush
self._flush(objects)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2664, in _flush
transaction.rollback(_capture_exception=True)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py",
line 69, in __exit__
exc_value, with_traceback=exc_tb,
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py",
line 2624, in _flush
flush_context.execute()
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 422, in execute
rec.execute(self)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 589, in execute
uow,
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py",
line 245, in save_obj
insert,
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py",
line 1136, in _emit_insert_statements
statement, params
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1011, in execute
return meth(self, multiparams, params)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/elements.py",
line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1130, in _execute_clauseelement
distilled_params,
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1317, in _execute_context
e, statement, parameters, cursor, context
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1277, in _execute_context
cursor, statement, parameters, context
File
"/Users/kaxilnaik/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint
failed: dag_run.dag_id, dag_run.run_id
[SQL: INSERT INTO dag_run (dag_id, execution_date, start_date, end_date,
state, run_id, creating_job_id, external_trigger, run_type, conf,
last_scheduling_decision, dag_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: ('scenario1_case2_01', '2021-01-25 00:00:00.000000',
'2021-01-26 13:07:54.516957', None, 'running',
'scheduled__2021-01-25T00:00:00+00:00', 1, 0, <DagRunType.SCHEDULED:
'scheduled'>, <memory at 0x7fc388665530>, None,
'09f9507aca47e7751a40cc98e546ac0d')]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
