geocomm-descue opened a new issue, #59117:
URL: https://github.com/apache/airflow/issues/59117
### Apache Airflow version
3.1.3
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
We have a couple of custom subclasses of BaseTrigger. When testing a new
one, I had a couple of print statements inside the definition to `async def
run` on the trigger class. When an operator defered to the triggerer with that
trigger, the Triggerer immediatly dies with the following error.
```
[2025-12-05T21:29:15.305663Z] {triggerer_job_runner.py:173} ERROR -
Exception when executing TriggerRunnerSupervisor.run
Traceback (most recent call last):
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/triggerer_job_runner.py",
line 170, in _execute
self.trigger_runner.run()
~~~~~~~~~~~~~~~~~~~~~~~^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/triggerer_job_runner.py",
line 529, in run
self._service_subprocess(1)
~~~~~~~~~~~~~~~~~~~~~~~~^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 772, in _service_subprocess
need_more = socket_handler(key.fileobj)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1679, in cb
gen.send(line)
~~~~~~~~^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1777, in forward_to_log
log.log(level, msg, logger=logger)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_native.py",
line 197, in log
return self._proxy_to_logger(
~~~~~~~~~~~~~~~~~~~~~^
name, _maybe_interpolate(event, args), **kw
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_base.py",
line 222, in _proxy_to_logger
args, kw = self._process_event(method_name, event, event_kw)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_base.py",
line 173, in _process_event
event_dict = proc(self._logger, method_name, event_dict)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/_shared/logging/percent_formatter.py",
line 152, in __call__
sio.write(self._fmt % params)
~~~~~~~~~~^~~~~~~~
TypeError: %d format: a real number is required, not NoneType
[2025-12-05T21:29:15.306829Z] {triggerer_job_runner.py:176} INFO - Waiting
for triggers to clean up
[2025-12-05T21:29:15.310214Z] {supervisor.py:709} INFO - Process exited
pid=62478 exit_code=-2 signal_sent=SIGINT
[2025-12-05T21:29:15.310309Z] {triggerer_job_runner.py:181} INFO - Exited
trigger loop
Traceback (most recent call last):
File "/workspace/main.py", line 40, in <module>
main()
~~~~^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/click/core.py",
line 1442, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/click/core.py",
line 1363, in main
rv = self.invoke(ctx)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/click/core.py",
line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/click/core.py",
line 794, in invoke
return callback(*args, **kwargs)
File "/workspace/main.py", line 34, in main
sys.exit(main())
~~~~^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/__main__.py",
line 55, in main
args.func(args)
~~~~~~~~~^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/cli/cli_config.py",
line 49, in command
return func(*args, **kwargs)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/utils/cli.py",
line 114, in wrapper
return f(*args, **kwargs)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/utils/providers_configuration_loader.py",
line 54, in wrapped_function
return func(*args, **kwargs)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/cli/commands/triggerer_command.py",
line 69, in triggerer
run_command_with_daemon_option(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
args=args,
^^^^^^^^^^
...<2 lines>...
should_setup_logging=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/cli/commands/daemon_utils.py",
line 86, in run_command_with_daemon_option
callback()
~~~~~~~~^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/cli/commands/triggerer_command.py",
line 72, in <lambda>
callback=lambda: triggerer_run(args.skip_serve_logs, args.capacity,
triggerer_heartrate),
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/cli/commands/triggerer_command.py",
line 55, in triggerer_run
run_job(job=triggerer_job_runner.job,
execute_callable=triggerer_job_runner._execute)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/utils/session.py",
line 100, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/job.py",
line 368, in run_job
return execute_job(job, execute_callable=execute_callable)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/job.py",
line 397, in execute_job
ret = execute_callable()
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/triggerer_job_runner.py",
line 170, in _execute
self.trigger_runner.run()
~~~~~~~~~~~~~~~~~~~~~~~^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/jobs/triggerer_job_runner.py",
line 529, in run
self._service_subprocess(1)
~~~~~~~~~~~~~~~~~~~~~~~~^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 772, in _service_subprocess
need_more = socket_handler(key.fileobj)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1679, in cb
gen.send(line)
~~~~~~~~^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1777, in forward_to_log
log.log(level, msg, logger=logger)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_native.py",
line 197, in log
return self._proxy_to_logger(
~~~~~~~~~~~~~~~~~~~~~^
name, _maybe_interpolate(event, args), **kw
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_base.py",
line 222, in _proxy_to_logger
args, kw = self._process_event(method_name, event, event_kw)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/structlog/_base.py",
line 173, in _process_event
event_dict = proc(self._logger, method_name, event_dict)
File
"/home/vscode/.pyenv/versions/3.13.7/lib/python3.13/site-packages/airflow/_shared/logging/percent_formatter.py",
line 152, in __call__
sio.write(self._fmt % params)
~~~~~~~~~~^~~~~~~~
TypeError: %d format: a real number is required, not NoneType
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [62450]
```
Any attempt to restart the triggerer fails and crashes with the same error
due to the dagrun still being active and the operator still in a defered state.
The triggerer only restarts succesfully if the defered task times out or is
killed .
I was able to reproduce the error by TimeDeltaTrigger and adding a single
print statement to an otherwise unmodified trigger. With the print statement
there, the triggerer crashes. without it, it works normally. I understand that
printing is synchronous and generally not recomended inside an async function,
but I would not expect it to bring down the whole triggerer.
### What you think should happen instead?
I would think that print statement, while maybe not recomended in async
programing, wouldn't crash the triggerer.
### How to reproduce
Add a print statement inside the call to `async def run` in any trigger.
### Operating System
Ubuntu 22.04.5 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.16.0
apache-airflow-providers-celery==3.13.0
apache-airflow-providers-cncf-kubernetes==10.9.0
apache-airflow-providers-common-compat==1.8.0
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-sql==1.28.2
apache-airflow-providers-git==0.0.9
apache-airflow-providers-http==5.4.0
apache-airflow-providers-sendgrid==4.1.4
apache-airflow-providers-smtp==2.3.1
apache-airflow-providers-standard==1.9.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
We see this behavior both in our deployed instances and in `airflow
standalone`
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
