jason810496 commented on code in PR #67104:
URL: https://github.com/apache/airflow/pull/67104#discussion_r3427843411
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst:
##########
@@ -102,6 +103,56 @@ See :doc:`../modules_management` for details on how Python
and Airflow manage mo
You can override the way both standard logs of the components and "task"
logs are handled.
+Custom logging configs and remote logging
+-----------------------------------------
+
+When ``[logging] remote_logging = True`` and you point ``logging_config_class``
+at your own module, define two module-level attributes alongside your
+``LOGGING_CONFIG`` dict:
+
+* ``REMOTE_TASK_LOG`` — an instance of
+ :class:`~airflow.logging.remote.RemoteLogIO` (or
+ :class:`~airflow.logging.remote.RemoteLogStreamIO`) that uploads task logs
+ and reads them back for the UI.
+* ``DEFAULT_REMOTE_CONN_ID`` — default Airflow connection id used when
+ ``[logging] remote_log_conn_id`` is unset.
+
+If ``REMOTE_TASK_LOG`` is missing, Airflow emits one ``WARNING`` at startup
+and the UI cannot read task logs back from the remote backend.
+
+ .. code-block:: python
+
+ # ~/airflow/config/log_config.py
+ from airflow.logging.remote import RemoteLogIO
+
+
+ class MyRemoteLogIO:
+ @property
+ def processors(self):
+ return ()
+
+ def upload(self, path, ti): ... # upload local log file at ``path``
to your backend
+
+ def read(self, relative_path, ti): ... # return (source_info,
log_messages) for the UI
+
+
+ REMOTE_TASK_LOG: RemoteLogIO | None = MyRemoteLogIO()
+ DEFAULT_REMOTE_CONN_ID: str | None = "my_remote_conn"
+
+ LOGGING_CONFIG = {
+ "version": 1,
+ "disable_existing_loggers": False,
+ # ... your formatters / handlers / loggers / root ...
+ }
Review Comment:
IIUC, the dict is still used. In the default case (`logging_config_class`
unset) it's imported by its dotted path in `_get_logging_config()`:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/airflow-core/src/airflow/logging_config.py#L65-L74
passed as `stdlib_config` into `configure_logging`:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/airflow-core/src/airflow/logging_config.py#L178-L194
spread into the dictConfig payload via `config = {**stdlib_config}`:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/shared/logging/src/airflow_shared/logging/structlog.py#L631-L636
and applied at `logging.config.dictConfig(config)`:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/shared/logging/src/airflow_shared/logging/structlog.py#L684
Including the `FileTaskHandler` and logger tree all reach `dictConfig`. That
handler is what writes task logs:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L394
What structlog *does* override is console formatting: it injects its own
`structlog` formatter + `default` handler and strips per-handler
`stream`/`formatter`:
https://github.com/apache/airflow/blob/9849cf9b851298e5d7bede1b0fe3e54c7260b3dc/shared/logging/src/airflow_shared/logging/structlog.py#L648-L668
So the formatter details are replaced, but the config dict itself isn't
bypassed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
