Krishna-skk opened a new issue, #51143:
URL: https://github.com/apache/airflow/issues/51143
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.4
### What happened?
**Environment:**
* **Airflow version**: 2.10.4
* **Python version**: 3.10
* **Deployment**: ECS Fargate (Dockerized Airflow)
* **Remote Logging**: Enabled using Amazon S3
---
#Problem Description
I've configured Airflow to use **remote logging with S3**, and logs are
successfully uploaded to my S3 bucket. However, **all task logs—regardless of
DAG, task, or attempt—are written to a single file named**:
logfile_2024_03_03.log
This is unexpected, as I have set the `log_filename_template` in
`airflow.cfg` to follow Airflow’s structured directory format, yet it is being
ignored.
#Current Relevant Config in `airflow.cfg`
remote_logging = True
remote_base_log_folder = s3://quicflow-airflow-logs-bucket/airflow-logs
remote_log_conn_id = aws_default
delete_local_logs = True
log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id
}}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index
}}/{% endif %}attempt={{ try_number }}.log
log_processor_filename_template = {{ filename }}
#What I’ve Tried
* Checked `airflow.cfg` thoroughly — **there is no `base_log_folder`
setting** (Airflow should fall back to the default `/opt/airflow/logs`).
* I **do not have any custom `logging_config_class`**.
* Removed `.log` suffix from `log_processor_filename_template` — no effect.
* Checked S3 bucket — **only one file** `logfile_2024_03_03.log` keeps
getting overwritten by every new task run.
* I’m not setting `AIRFLOW__LOGGING__BASE_LOG_FOLDER` as an environment
variable.
#Errors/Warnings Observed
I also see this warning in the logs:
FileNotFoundError: [Errno 2] No such file or directory:
'/opt/airflow/logs/logfile_2024_03_03.log'
### What you think should happen instead?
_No response_
### How to reproduce
#Questions
1. Why is Airflow not honoring the `log_filename_template` setting?
2. Where is `logfile_2024_03_03.log` even coming from? I’ve never specified
this anywhere.
3. Could it be falling back to a default handler or using `StreamLogHandler`
due to a misconfigured `log_processor_filename_template`?
Any help in resolving this issue would be greatly appreciated! Let me know
if you need me to share the full `airflow.cfg`.
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
I have dockerized the airflow with version 2.10.4 and deployed the airflow
on Aws ECS as fargate containers
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
