pkgit36 commented on issue #50349:
URL: https://github.com/apache/airflow/issues/50349#issuecomment-3096699826
Hello, We are also experiencing issues sending Airflow logs to Elasticsearch
(ELK).
Environment:
Airflow version: 2.10.5
ELK provider version: 6.3.1
[elasticsearch]
host = <elasticsearch-url>
write_stdout = False
write_to_es = True
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}
target_index = <index>
end_of_log_mark = end_of_log
[elasticsearch_configs]
verify_certs = False
basic_auth = <user>:<password>
timeout = 30
max_retries = 3
retry_timeout = True
[logging]
remote_logging = True
logging_level = DEBUG
json_format = True
Airflow is able to connect to Elasticsearch, as confirmed by the following
log from the webserver:
InsecureRequestWarning: Unverified HTTPS request is being made to host
'<elasticsearch-url>'.
INFO - POST https://<elasticsearch-url>/_all/_count [status:200
duration:0.082s]
However, when checking the task logs in the Airflow UI, we see the following
error:
*** Log <log_id> not found in Elasticsearch. If your task started recently,
please wait a moment and reload this page. Otherwise, the logs for this task
instance may have been removed.
It appears that logs are not being written to the specified index.
Any resolution for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
