nailo2c opened a new pull request, #60384: URL: https://github.com/apache/airflow/pull/60384
closes: #56340 # Why The user configured a custom logging handler in Airflow by following this [doc](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html). It works in a direct Python test, but it does not work when setting `logging_config_class`. # How It lacks the `load_logging_config` step in the `configure_logging` function; adding it resolves this issue. https://github.com/apache/airflow/blob/67c95c3f80a07ac42e4c39c3aac6c31cb36417f8/task-sdk/src/airflow/sdk/log.py#L96 # What ## Step 1: Setup Here are the settings I used to reproduce this issue locally: <details> <summary><code>files/airflow-breeze-config/environment_variables.env</code> <b>(Click to expand)</b></summary> ```console AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG PYTHONPATH=/files/config${PYTHONPATH:+:$PYTHONPATH} ``` </details> <details> <summary><code>files/config/clickhouse_logging.py</code> <b>(Click to expand)</b></summary> ```python import json import logging import os from pathlib import Path from threading import Lock from typing import Any DEFAULT_LOG_PATH = os.path.join( os.environ.get("AIRFLOW_HOME", "/opt/airflow"), "logs", "clickhouse_handler.log", ) class ClickHouseHTTPHandler(logging.Handler): def __init__(self, log_path: str | None = None) -> None: super().__init__() self.log_path = log_path or DEFAULT_LOG_PATH Path(self.log_path).parent.mkdir(parents=True, exist_ok=True) self._lock = Lock() def emit(self, record: logging.LogRecord) -> None: try: payload: dict[str, Any] = { "ts": record.created, "logger": record.name, "level": record.levelname, "message": self.format(record), } line = json.dumps(payload, ensure_ascii=True) with self._lock: with open(self.log_path, "a", encoding="utf-8") as handle: handle.write(line + "\n") except Exception: self.handleError(record) ``` </details> <details> <summary><code>files/config/log_config.py</code> <b>(Click to expand)</b></summary> ```python import os from copy import deepcopy from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG) REMOTE_TASK_LOG = None # to prevent triggerer and dag-processor crash DEFAULT_REMOTE_CONN_ID = None # to prevent triggerer and dag-processor crash LOGGING_CONFIG["handlers"]["clickhouse"] = { "class": "clickhouse_logging.ClickHouseHTTPHandler", "log_path": os.environ.get( "AIRFLOW__CLICKHOUSE_LOG_PATH", os.path.join(os.environ.get("AIRFLOW_HOME", "/opt/airflow"), "logs", "clickhouse_handler.log"), ), "level": "INFO", } task_handlers = list(LOGGING_CONFIG["loggers"]["airflow.task"]["handlers"]) if "clickhouse" not in task_handlers: task_handlers.append("clickhouse") LOGGING_CONFIG["loggers"]["airflow.task"]["handlers"] = task_handlers ``` </details> <details> <summary><code>files/dags/repro_logging_config_class.py</code> <b>(Click to expand)</b></summary> ```python import logging import pendulum from airflow import DAG from airflow.operators.python import PythonOperator def _describe_handlers(logger: logging.Logger) -> list[str]: descriptions = [] for handler in logger.handlers: name = getattr(handler, "name", None) level = logging.getLevelName(handler.level) descriptions.append( f"{handler.__class__.__module__}.{handler.__class__.__name__}(name={name}, level={level})" ) return descriptions def emit_logs() -> None: logger = logging.getLogger("airflow.task") logger.info("airflow.task handlers: %s", _describe_handlers(logger)) logger.info("airflow.task propagate=%s parent=%s", logger.propagate, getattr(logger.parent, "name", None)) logger.info("stdlib logger message from repro task") print("stdout message from repro task") with DAG( dag_id="repro_logging_config_class", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), schedule=None, catchup=False, tags=["repro"], ) as dag: PythonOperator(task_id="emit_logs", python_callable=emit_logs) ``` </details> ## Step 2: Before the fix - Reproducing the issue First, make sure the Airflow config has the correct value: ```console [Breeze:3.10.19] root@e09e71f55eb8:/opt/airflow/logs$ airflow config get-value logging logging_config_class log_config.LOGGING_CONFIG ``` I ran this command in the container, and it wrote a line to `logs/clickhouse_handler.log` as expected: ```console PYTHONPATH=/files/config python - <<'PY' import logging from clickhouse_logging import ClickHouseHTTPHandler logger = logging.getLogger("airflow.task") logger.setLevel(logging.INFO) logger.addHandler(ClickHouseHTTPHandler()) logger.info("direct python handler test") PY ``` `logs/clickhouse_handler.log` ```console {"ts": 1768007319.4109492, "logger": "airflow.task", "level": "INFO", "message": "direct python handler test"} ``` However, when I triggered the DAG, nothing was written to the log file, and the UI logs showed that the handler had not been loaded from the config. <img width="1913" height="505" alt="截圖 2026-01-10 下午9 24 50" src="https://github.com/user-attachments/assets/e0294fef-6b38-4309-ad45-1ca1f85e327c"; /> ## Step 3: After the fix After the fix, it works as expected and the config is loaded: <img width="1906" height="512" alt="截圖 2026-01-10 下午9 25 00" src="https://github.com/user-attachments/assets/223923c8-a7ac-4693-90bc-c90b77f4e9cc"; /> It also wrote logs to `logs/clickhouse_handler.log`: ```console {"ts": 1768084859.17813, "logger": "airflow.task", "level": "INFO", "message": "airflow.task handlers: ['airflow.utils.log.file_task_handler.FileTaskHandler(name=task, level=NOTSET)', 'clickhouse_logging.ClickHouseHTTPHandler(name=clickhouse, level=INFO)']"} {"ts": 1768084859.1806972, "logger": "airflow.task", "level": "INFO", "message": "airflow.task propagate=True parent=airflow"} {"ts": 1768084859.1817467, "logger": "airflow.task", "level": "INFO", "message": "stdlib logger message from repro task"} ``` ## Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: gpt-5.2-codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
