ecodina commented on issue #61916:
URL: https://github.com/apache/airflow/issues/61916#issuecomment-3907497818
For us, this issue was critical. The triggerer was being restarted every 2
hours.
After a thorough investigation, I let Codex 5.3 investigate itself the
problem and it came up with a patch, and it worked!
I have applied this patch through a plugin. If anyone else has this issue,
you can apply it as well. It should also help fix the bug in Airflow.
```python
from __future__ import annotations
from contextlib import suppress
def patch_triggerer_log_file_closure() -> None:
"""Patch Airflow triggerer logging to close file descriptors explicitly.
Airflow's TriggerLoggingFactory currently relies on object finalization
to release
the open file handle used for per-trigger logs. This patch makes closure
explicit
when a trigger logger is removed from the cache.
"""
try:
from airflow.jobs.triggerer_job_runner import TriggerLoggingFactory
except Exception:
return
if getattr(TriggerLoggingFactory, "_fd_patch_applied", False):
return
original_upload_to_remote = TriggerLoggingFactory.upload_to_remote
def upload_to_remote_and_close(self) -> None:
try:
original_upload_to_remote(self)
finally:
bound_logger = getattr(self, "bound_logger", None)
raw_logger = getattr(bound_logger, "_logger", None)
file_handle = getattr(raw_logger, "_file", None)
with suppress(Exception):
if file_handle and not file_handle.closed:
file_handle.flush()
file_handle.close()
with suppress(Exception):
delattr(self, "bound_logger")
TriggerLoggingFactory.upload_to_remote = upload_to_remote_and_close
TriggerLoggingFactory._fd_patch_applied = True
```
I'd prefer not to open the PR myself, since I know this patch works for us
but I can't guarantee it will work for everyone. Maybe someone more expert in
logging / triggerer could validate it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
