ciancolo opened a new issue, #56451:
URL: https://github.com/apache/airflow/issues/56451
### Apache Airflow version
3.1.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
After upgrading Airflow from version 3.0.6 to 3.1.0, we started encountering
an issue with the Dag Processor service.
During the DAG serialization phase, the Dag Processor raises an exception
and then restarts itself.
Below is an excerpt from the logs:
`Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
[2025-10-07T10:09:21.642749Z] {dag_processor_job_runner.py:63} ERROR -
Exception when executing DagProcessorJob
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: Traceback
(most recent call last):
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/jobs/dag_processor_job_runner.py",
line 61, in _execute
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self.processor.run()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 272, in run
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
self._run_parsing_loop()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 361, in _run_parsing_loop
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._collect_results()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/session.py",
line 100, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, session=session, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 827, in _collect_results
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._file_stats[file] = process_parse_results(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 1155, in process_parse_results
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
update_dag_parsing_results_in_db(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 372, in update_dag_parsing_results_in_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: for
attempt in run_with_db_retries(logger=log):
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 445, in __iter__
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: do =
self.iter(retry_state=retry_state)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 378, in iter
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: result
= action(retry_state)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 400, in <lambda>
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._add_action_func(lambda rs: rs.outcome.result())
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
self.__get_result()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in
__get_result
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: raise
self._exception
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 382, in update_dag_parsing_results_in_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
SerializedDAG.bulk_write_to_db(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/session.py",
line 98, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/serialization/serialized_objects.py",
line 2868, in bulk_write_to_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
dag_op.update_dags(orm_dags, parse_duration, session=session)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 530, in update_dags
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
last_automated_data_interval = get_run_data_interval(dag.timetable,
last_automated_run)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/models/dag.py",
line 143, in get_run_data_interval
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
infer_automated_data_interval(timetable, run.logical_date)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/models/dag.py",
line 121, in infer_automated_data_interval
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: raise
ValueError(f"Not a valid timetable: {timetable!r}")
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: ValueError:
Not a valid timetable: <airflow.timetables.trigger.CronTriggerTimetable object
at 0x7f2b67e9f940>
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
[2025-10-07T10:09:21.653271Z] {supervisor.py:709} INFO - Process exited
pid=2721376 exit_code=<Negsignal.SIGTERM: -15> signal_sent=SIGTERM
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
[2025-10-07T10:09:21.661775Z] {process_utils.py:285} INFO - Waiting up to 5
seconds for processes to exit...
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: Traceback
(most recent call last):
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/bin/airflow", line 8, in <module>
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
sys.exit(main())
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/__main__.py",
line 55, in main
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
args.func(args)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/cli/cli_config.py",
line 49, in command
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/cli.py",
line 114, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
f(*args, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py",
line 54, in wrapped_function
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/cli/commands/dag_processor_command.py",
line 53, in dag_processor
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
run_command_with_daemon_option(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py",
line 86, in run_command_with_daemon_option
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
callback()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/cli/commands/dag_processor_command.py",
line 56, in <lambda>
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
callback=lambda: run_job(job=job_runner.job,
execute_callable=job_runner._execute),
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/session.py",
line 100, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, session=session, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/jobs/job.py",
line 368, in run_job
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
execute_job(job, execute_callable=execute_callable)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/jobs/job.py",
line 397, in execute_job
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: ret =
execute_callable()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/jobs/dag_processor_job_runner.py",
line 61, in _execute
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self.processor.run()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 272, in run
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
self._run_parsing_loop()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 361, in _run_parsing_loop
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._collect_results()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/session.py",
line 100, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, session=session, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 827, in _collect_results
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._file_stats[file] = process_parse_results(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/manager.py",
line 1155, in process_parse_results
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
update_dag_parsing_results_in_db(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 372, in update_dag_parsing_results_in_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: for
attempt in run_with_db_retries(logger=log):
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 445, in __iter__
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: do =
self.iter(retry_state=retry_state)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 378, in iter
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: result
= action(retry_state)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/tenacity/__init__.py",
line 400, in <lambda>
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
self._add_action_func(lambda rs: rs.outcome.result())
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
self.__get_result()
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in
__get_result
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: raise
self._exception
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 382, in update_dag_parsing_results_in_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
SerializedDAG.bulk_write_to_db(
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/utils/session.py",
line 98, in wrapper
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
func(*args, **kwargs)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/serialization/serialized_objects.py",
line 2868, in bulk_write_to_db
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
dag_op.update_dags(orm_dags, parse_duration, session=session)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/dag_processing/collection.py",
line 530, in update_dags
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]:
last_automated_data_interval = get_run_data_interval(dag.timetable,
last_automated_run)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/models/dag.py",
line 143, in get_run_data_interval
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: return
infer_automated_data_interval(timetable, run.logical_date)
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: File
"/opt/tenaris/venvs/airflow/lib/python3.10/site-packages/airflow/models/dag.py",
line 121, in infer_automated_data_interval
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: raise
ValueError(f"Not a valid timetable: {timetable!r}")
Oct 07 12:09:21 daleulbd11.dalmine.techint.net airflow[2721133]: ValueError:
Not a valid timetable: <airflow.timetables.trigger.CronTriggerTimetable object
at 0x7f2b67e9f940>
Oct 07 12:09:22 daleulbd11.dalmine.techint.net systemd[1]:
airflow-dag-processor.service: Main process exited, code=exited,
status=1/FAILURE
Oct 07 12:09:22 daleulbd11.dalmine.techint.net systemd[1]:
airflow-dag-processor.service: Failed with result 'exit-code'.
Oct 07 12:09:22 daleulbd11.dalmine.techint.net systemd[1]:
airflow-dag-processor.service: Consumed 32.023s CPU time.
Oct 07 12:09:27 daleulbd11.dalmine.techint.net systemd[1]:
airflow-dag-processor.service: Scheduled restart job, restart counter is at
15533.
Oct 07 12:09:27 daleulbd11.dalmine.techint.net systemd[1]: Stopped Airflow
Dag processor daemon.
Oct 07 12:09:27 daleulbd11.dalmine.techint.net systemd[1]:
airflow-dag-processor.service: Consumed 32.023s CPU time.`
Despite this error, the Airflow instance itself works correctly — all DAGs
appear in the UI and execute as expected.
No changes were made to any DAG configurations.
`
default_args = {
'owner': 'airflow',
'email': ['myemail'],
'email_on_failure': True,
'email_on_retry': False,
'retries': 2,
'retry_delay': timedelta(minutes=10),
'execution_timeout':timedelta(minutes=50),
'weight_rule': 'upstream'
}
dag = DAG(
dag_id="dag_name",
schedule="0 2 * * *",
start_date=pendulum.datetime(2022, 8, 28, tz="Europe/Rome"),
catchup=False,
max_active_tasks=7,
default_args=default_args)`
### What you think should happen instead?
_No response_
### How to reproduce
I’m not sure what triggered this behavior. I can provide my Airflow
configuration if it helps reproduce the issue.
### Operating System
Ubuntu 22.04.5 LTS
### Versions of Apache Airflow Providers
`
apache-airflow 3.1.0
apache-airflow-core 3.1.0
apache-airflow-providers-apache-hdfs 4.10.3
apache-airflow-providers-apache-spark 5.3.2
apache-airflow-providers-apache-sqoop 3.2.0
apache-airflow-providers-common-compat 1.7.4
apache-airflow-providers-common-io 1.6.3
apache-airflow-providers-common-sql 1.28.1
apache-airflow-providers-elasticsearch 6.3.3
apache-airflow-providers-fab 2.4.3
apache-airflow-providers-http 5.3.4
apache-airflow-providers-jdbc 5.2.3
apache-airflow-providers-postgres 6.3.0
apache-airflow-providers-salesforce 5.11.3
apache-airflow-providers-smtp 2.2.1
apache-airflow-providers-ssh 4.1.4
apache-airflow-providers-standard 1.8.0
apache-airflow-providers-tableau 5.2.0
apache-airflow-task-sdk 1.1.0
`
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
