Owen-CH-Leung commented on code in PR #44973: URL: https://github.com/apache/airflow/pull/44973#discussion_r1888457249
########## docs/apache-airflow-providers-elasticsearch/logging/index.rst: ########## @@ -22,6 +22,8 @@ Writing logs to Elasticsearch Airflow can be configured to read task logs from Elasticsearch and optionally write logs to stdout in standard or json format. These logs can later be collected and forwarded to the Elasticsearch cluster using tools like fluentd, logstash or others. +Airflow also supports writing log to Elasticsearch directly without requiring additional software like filebeat and logstash. To enable this feature, set ``write_to_es`` and ``json_format`` to ``True`` and ``write_stdout`` to ``False`` in ``airflow.cfg``. Review Comment: Indeed, while in this PR I have added feature to automatically write task logs to ES, I've also added codes to consider the `delete_local_logs` defined in airflow config. https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-local-logs If users set both `write_to_es` and `delete_local_logs` to true, airflow will attempt to remove local task logs upon successfully writing task logs to ES. This is inspired by the `S3TaskHandler` which shares the same feature. https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py#L100-L102 I've revised the doc to flag this out. Actually I'm not sure if we should consider the `delete_local_logs` config to remove local task logs. Maybe it's safer to not remove local copies so in case writing to es fails, users can always re-enable file-beat to ship logs. But Incorporating the delete_local_logs config aligns this feature more closely with the behavior of other handlers and provides a more comprehensive solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
