KanaSukita opened a new issue, #41171:
URL: https://github.com/apache/airflow/issues/41171
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.2
### What happened?
I have configured the airflow.cfg file and make sure the the Celery worker
logs are stored in both the worker instance and the Amazon S3.
The logs in S3 are stored in
s3://${BUCKET_NAME}/airflow_logs/dag_id=xxxx/run_id=scheduled__2024-07-30T03:20:00+00:00/task_id=xxxx/，and
logs saved in the worker instance with path
${AIRFLOW_HOME}/logs/dag_id=xxxx/run_id=scheduled__2024-07-30T03:20:00+00:00/task_id=xxxx/
When I check the Airflow UI for task retrieval, it cannot retrieve the logs
from S3, but it can retrieve the logs from the worker instance. Logs are as
below:
```
ip-10-121-129-159.eu-west-1.compute.internal
*** No logs found on s3 for ti=<TaskInstance:
logstash_alice_etl.convert_euw1 scheduled__2024-07-30T03:20:00+00:00 [success]>
*** Found logs served from host
http://ip-10-121-129-159.eu-west-1.compute.internal:8793/log/dag_id=xxxx/run_id=scheduled__2024-07-30T03:20:00+00:00/task_id=xxxx/attempt=4.log
[2024-08-01, 07:29:55 UTC] {local_task_job_runner.py:120} ▶ Pre task
execution logs
[2024-08-01, 07:30:03 UTC] {subprocess.py:63} INFO - Tmp dir root location:
/tmp
...
```
### What you think should happen instead?
The logs should be found in S3 instead of the worker instance
### How to reproduce
configure the [logging] parts in airflow.cfg with the below modifications:
base_log_folder = /opt/apache/airflow/logs
remote_logging = True
remote_log_conn_id = aws_airflow_euw1 # Just an IAM
connection to S3 in eu-west-1
remote_base_log_folder = s3://${BUCKET_NAME}/airflow_logs
### Operating System
Amazon Linux 2023
### Versions of Apache Airflow Providers
airflow 2.9.2
### Deployment
Virtualenv installation
### Deployment details
Deploy in Amazon Linux 2023 with default python environment.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
