pkesso opened a new issue, #39876: URL: https://github.com/apache/airflow/issues/39876
### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? I have airflow 2.9.1 deployed on kubernetes with official helm chart. Remote logging to s3 (minio) is enabled. Recently I switched workers from statefulset to deployment (persistence.enabled: false). In this configuration log groomer sidecar is disabled, and I noticed that logs volume size is growing. Using command `find /opt/airflow/logs -type f` I found some log files left in /opt/airflow/logs on worker pods - logs from different DAGs and different tasks. Some task logs were left more often than other, but generally I cant see any pattern. All of these logs were uploaded to s3 successfully. There was no any useful log entries on log level INFO. logging section of config file: ``` [logging] colored_console_log = False delete_local_logs = True logging_level = INFO remote_base_log_folder = s3://airflow-prod-logs/ remote_log_conn_id = minio2 remote_logging = True ``` ### What you think should happen instead? Logs volume is an EmptyDir, and its overflowing would drain node's ephemeral storage. So, log cleanup is extremely important for long-living containers, even for "non-persistent". ### How to reproduce Deploy an airflow 2.9.1 installation via official helm chart using options `worker.persistence.enabled: false` ``` [logging] colored_console_log = False delete_local_logs = True logging_level = INFO remote_base_log_folder = s3://some-bucker/ remote_log_conn_id = some-s3-connection remote_logging = True ``` Run some DAGs via celery executor Check s3 for logs arrived Check /opt/airflow/logs in worker pods for logs left: `find /opt/airflow/logs -type f` Wait some time to see /opt/airflow/logs size growth. ### Operating System kubernetes 1.27.2 ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==6.2.0 ### Deployment Official Apache Airflow Helm Chart ### Deployment details airflow 2.9.1 minio/minio:RELEASE.2024-05-10T01-41-38Z ### Anything else? Log groomer sidecar probably should be available even if logs.persistence is disabled. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
