GitHub user bun-paul-kwon created a discussion: How to disable component file logging (scheduler, worker) and log only to stdout/stderr?
Hello Airflow community, We are currently using AWS MWAA (Celery Executor) and considering migrating to Airflow 3.0.2 on Kubernetes (AWS EKS). - Image: apache/airflow:3.0.2 - Helm chart: [https://airflow.apache.org airflow 1.18.0](https://airflow.apache.org/docs/helm-chart/stable/index.html) - [values.yaml](https://github.com/apache/airflow/blob/helm-chart/1.18.0/chart/values.yaml) Our **task logs** are configured to use remote logging to S3, and this is working well. Our configuration(`values.yaml`) for this is: ```yaml executor: "CeleryExecutor" logging: logging_level: "INFO" delete_local_logs: "True" remote_logging: "True" remote_base_log_folder: s3://<MY_AWS_S3_BUCKET> remote_log_conn_id: <MY_AIRFLOW_CONNECTION_NAME> scheduler: logGroomerSidecar: false dagProcessor: logGroomerSidecar: false workers: logGroomerSidecar: false # ... ``` Our question is about the logs for **Airflow components (e.g., scheduler, dag-processor, worker)**, not the task logs. We've observed the following behavior in the Helm chart: 1. The `apiserver` logs only to `stdout`/`stderr` (which is our desired behavior). 2. The `scheduler`, `dag-processor` and `worker` pods log to both `stdout`/`stderr` AND local files (in `/opt/airflow/logs`). 3. Because of this, the `logGroomerSidecar` is needed to clean up these local files. Our goal is to **disable local file logging** for all components (scheduler, worker, dag-processor) and have them **log only to** `stdout`/`stderr`. We use Datadog autodiscovery, which scrapes these `stdout`/`stderr` streams for logging. The local file logs are redundant for us, and we would like to disable the `logGroomerSidecar` container entirely. Is there a recommended way to configure this using the Helm chart `values.yaml`? Thank you for your guidance! GitHub link: https://github.com/apache/airflow/discussions/57535 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
