GitHub user lesliejjytbw edited a discussion: CeleryExecutor tasks flooded by repeated ‘Secrets backends loaded for worker’ INFO logs
## Env Python: 3.11 Airflow: 3.0 Executor: CeleryExecutor Broker: Redis Concurrency: 16 ## Issue Hi, all. It seems like [#49205](https://github.com/apache/airflow/pull/49205) change made all the task logs get dominated by the exact logs for our celery executors. ``` [2025-05-09, 15:03:48] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:03:48] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:05:00] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:05:00] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:05:41] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:05:41] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:06:21] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:06:21] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:03] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:03] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:08:06] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:08:06] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:09:15] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:09:15] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:11] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:11] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:20] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:20] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:33] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" [2025-05-09, 15:10:33] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor" ``` There appear to have at least a few lines to tens of these chunks in all task logs. Currently I just monkey patched to not log at all: ``` from airflow.sdk.execution_time import supervisor as sup_mod from airflow.configuration import ensure_secrets_loaded as _cfg_load from airflow.secrets import DEFAULT_SECRETS_SEARCH_PATH_WORKERS def ensure_secrets_backend_loaded(): return _cfg_load(default_backends=DEFAULT_SECRETS_SEARCH_PATH_WORKERS) sup_mod.ensure_secrets_backend_loaded = ensure_secrets_backend_loaded ``` Is there a more ‘official’ way (a config flag, built-in log filter, or plugin hook) I can use to suppress these repeated INFO lines on CeleryExecutor, without monkey-patching the Task-SDK code? GitHub link: https://github.com/apache/airflow/discussions/50472 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
