ephraimbuddy commented on PR #38743:
URL: https://github.com/apache/airflow/pull/38743#issuecomment-2037064921
> Are you sure when you reproduced it your Trigger table was empty? looking
at the fix, the whole for loop would be skipped if it was empty. Maybe you run
it with `start-airflow` and trigger managed to create somethign there ? Or
maybe you had old data ? Breeze by default keeps the data in the specially kept
volumes until you run breeze with `--db-reset` or run `breeze down` so there
could be some dangling data there.
I reconfirmed that it was empty. I reproduced it. The error happens on the
query itself not on the decryption. The session.query(Triggers) causes the task
instance table to be loaded and because ORM already has task_display_name but
this no longer exist in db after downgrade, the error occurs. Below is the
stacktrace:
```log
INFO [alembic.runtime.migration] Running downgrade 1fd565369930 ->
88344c1d9134, Add rendered_map_index to TaskInstance.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
self.dialect.do_execute(
File
"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line
736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column task_instance_1.task_display_name
does not exist
LINE 1: ...tance_1.try_number AS task_instance_1_try_number, task_insta...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/airflow/airflow/__main__.py", line 58, in main
args.func(args)
File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/cli.py", line 115, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line
55, in wrapped_function
return func(*args, **kwargs)
File "/opt/airflow/airflow/cli/commands/db_command.py", line 182, in
downgrade
db.downgrade(to_revision=to_revision, from_revision=from_revision,
show_sql_only=args.show_sql_only)
File "/opt/airflow/airflow/utils/session.py", line 79, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/airflow/airflow/utils/db.py", line 1752, in downgrade
decrypt_trigger_kwargs(session=session)
File "/opt/airflow/airflow/utils/db.py", line 996, in
decrypt_trigger_kwargs
for trigger in session.query(Trigger):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py",
line 2901, in __iter__
result = self._iter()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py",
line 2916, in _iter
result = self.session.execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py",
line 1717, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py",
line 334, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1577, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1953, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 2134, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
self.dialect.do_execute(
File
"/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line
736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column
task_instance_1.task_display_name does not exist
LINE 1: ...tance_1.try_number AS task_instance_1_try_number, task_insta...
^
[SQL: SELECT trigger.kwargs AS trigger_kwargs, trigger.id AS trigger_id,
trigger.classpath AS trigger_classpath, trigger.created_date AS
trigger_created_date, trigger.triggerer_id AS trigger_triggerer_id,
task_instance_1.try_number AS task_instance_1_try_number,
task_instance_1.task_display_name AS task_instance_1_task_display_name,
dag_run_1.state AS dag_run_1_state, dag_run_1.id AS dag_run_1_id,
dag_run_1.dag_id AS dag_run_1_dag_id, dag_run_1.queued_at AS
dag_run_1_queued_at, dag_run_1.execution_date AS dag_run_1_execution_date,
dag_run_1.start_date AS dag_run_1_start_date, dag_run_1.end_date AS
dag_run_1_end_date, dag_run_1.run_id AS dag_run_1_run_id,
dag_run_1.creating_job_id AS dag_run_1_creating_job_id,
dag_run_1.external_trigger AS dag_run_1_external_trigger, dag_run_1.run_type AS
dag_run_1_run_type, dag_run_1.conf AS dag_run_1_conf,
dag_run_1.data_interval_start AS dag_run_1_data_interval_start,
dag_run_1.data_interval_end AS dag_run_1_data_interval_end, dag_run_1.last_sch
eduling_decision AS dag_run_1_last_scheduling_decision, dag_run_1.dag_hash AS
dag_run_1_dag_hash, dag_run_1.log_template_id AS dag_run_1_log_template_id,
dag_run_1.updated_at AS dag_run_1_updated_at, dag_run_1.clear_number AS
dag_run_1_clear_number, task_instance_1.task_id AS task_instance_1_task_id,
task_instance_1.dag_id AS task_instance_1_dag_id, task_instance_1.run_id AS
task_instance_1_run_id, task_instance_1.map_index AS task_instance_1_map_index,
task_instance_1.start_date AS task_instance_1_start_date,
task_instance_1.end_date AS task_instance_1_end_date, task_instance_1.duration
AS task_instance_1_duration, task_instance_1.state AS task_instance_1_state,
task_instance_1.max_tries AS task_instance_1_max_tries,
task_instance_1.hostname AS task_instance_1_hostname, task_instance_1.unixname
AS task_instance_1_unixname, task_instance_1.job_id AS task_instance_1_job_id,
task_instance_1.pool AS task_instance_1_pool, task_instance_1.pool_slots AS
task_instance_1_pool_slots, task_in
stance_1.queue AS task_instance_1_queue, task_instance_1.priority_weight AS
task_instance_1_priority_weight, task_instance_1.operator AS
task_instance_1_operator, task_instance_1.custom_operator_name AS
task_instance_1_custom_operator_name, task_instance_1.queued_dttm AS
task_instance_1_queued_dttm, task_instance_1.queued_by_job_id AS
task_instance_1_queued_by_job_id, task_instance_1.pid AS task_instance_1_pid,
task_instance_1.executor AS task_instance_1_executor,
task_instance_1.executor_config AS task_instance_1_executor_config,
task_instance_1.updated_at AS task_instance_1_updated_at,
task_instance_1.rendered_map_index AS task_instance_1_rendered_map_index,
task_instance_1.external_executor_id AS task_instance_1_external_executor_id,
task_instance_1.trigger_id AS task_instance_1_trigger_id,
task_instance_1.trigger_timeout AS task_instance_1_trigger_timeout,
task_instance_1.next_method AS task_instance_1_next_method,
task_instance_1.next_kwargs AS task_instance_1_next_kwargs
FROM trigger LEFT OUTER JOIN (task_instance AS task_instance_1 JOIN dag_run
AS dag_run_1 ON dag_run_1.dag_id = task_instance_1.dag_id AND dag_run_1.run_id
= task_instance_1.run_id) ON trigger.id = task_instance_1.trigger_id]
(Background on this error at: https://sqlalche.me/e/14/f405)
```
This can only be reproduced when you downgrade half-way to 2.8.4, that's why
we missed it. In the CI, we downgrade from latest to 2.0.0 then backup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
