jedcunningham commented on code in PR #27758: URL: https://github.com/apache/airflow/pull/27758#discussion_r1095023715
########## docs/apache-airflow/administration-and-deployment/logging-monitoring/logging-tasks.rst: ########## @@ -122,15 +122,33 @@ When using remote logging, you can configure Airflow to show a link to an extern Some external systems require specific configuration in Airflow for redirection to work but others do not. -Serving logs from workers -------------------------- +Serving logs from workers and triggerer +--------------------------------------- -Most task handlers send logs upon completion of a task. In order to view logs in real time, Airflow automatically starts an HTTP server to serve the logs in the following cases: +Most task handlers send logs upon completion of a task. In order to view logs in real time, Airflow starts an HTTP server to serve the logs in the following cases: - If ``SequentialExecutor`` or ``LocalExecutor`` is used, then when ``airflow scheduler`` is running. - If ``CeleryExecutor`` is used, then when ``airflow worker`` is running. -The server is running on the port specified by ``worker_log_server_port`` option in ``[logging]`` section. By default, it is ``8793``. +In triggerer, logs are served unless the service is started with option ``--skip-serve-logs``. + +The server is running on the port specified by ``worker_log_server_port`` option in ``[logging]`` section, and option ``triggerer_log_server_port`` for triggerer. Defaults are 8793 and 8794, respectively. Communication between the webserver and the worker is signed with the key specified by ``secret_key`` option in ``[webserver]`` section. You must ensure that the key matches so that communication can take place without problems. We are using `Gunicorn <https://gunicorn.org/>`__ as a WSGI server. Its configuration options can be overridden with the ``GUNICORN_CMD_ARGS`` env variable. For details, see `Gunicorn settings <https://docs.gunicorn.org/en/latest/settings.html#settings>`__. + +Implementing a custom file task handler +--------------------------------------- + +.. note:: This is an advanced topic and most users should be able to just use an existing handler from :doc:`apache-airflow-providers:core-extensions/logging`. + +In our providers we have a healthy variety of options with all the major cloud providers. But should you need to implement logging with a different service, and should you then decide to implement a custom FileTaskHandler, there are a few settings to be aware of, particularly in the context of trigger logging. + +Triggers require a shift in the way that logging is set up. In contrast with tasks, many triggers run in the same process, and with triggers, since they run in asyncio, we have to be mindful of not introducing blocking calls through the logging handler. And because of the variation in handler behavior (some write to file, some upload to blob storage, some send messages over network as they arrive, some do so in thread), we need to have some way to let triggerer know how to use them. + +To accomplish this we have a few attributes that may be set either on the handler, either instance or the class. Inheritance is not respected for these parameters, because subclasses of FileTaskHandler may differ from it in the relevant characteristics. These params are described below: Review Comment: ```suggestion To accomplish this we have a few attributes that may be set on the handler instance or class. Inheritance is not respected for these parameters, because subclasses of FileTaskHandler may differ from it in the relevant characteristics. These params are described below: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
