mik-laj opened a new issue #10593:
URL: https://github.com/apache/airflow/issues/10593
Hello,
The configuration of the Airflow logger is not easy, as in most
applications. We have many loggers, some loggers generate multiple streams and
our architecture is heterogeneous, so we end up with multiple log streams. This
is very confusing for many users, and the lack of documentation doesn't make it
any easier.
We have following loggers
- ``root logger``
- ``airflow.processor``
- ``airflow.task``
- ``flask_appbuilder``
- ``airflow.processor_manager``
``airflow.task`` creates a new log file for each pair of task instances and
try numbers. These entries are accessible via the Web UI.
``airflow.processor`` creates a new log file for each DAG File.
``flask_appbuilder`` is only used to filter logs records because the FAB is
too verbose, so in most cases, we don't have to configure it specially.
Webserver writes logs to ``root logger``.
Worker writes logs to ``airflow.processor`` and ``root logger``
Scheduler writes logs to ``airflow.processor``,
``airflow.processor_manager`` and ``root logger``
I would be happy if there was a detailed description for the default
configuration and the methods on how to influence these configurations were
described. Currently, the documentation only describes configurations for
``airflow.task``. I wonder if I have the correct configuration for
``airflow.processor``. in my environment. The community only supports writing
logs from this logger to a files for each DAG file, but this is not a useful
solution in an environment with volatile volumes like Kubernetes.
Best regards,
Kamil BreguĊa
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
