benjamindinh-loreal opened a new issue, #44724: URL: https://github.com/apache/airflow/issues/44724
### Apache Airflow version 2.10.3 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Hello all, hope you doing well. While trying to send logs to elasticsearch directly threw the elastic adapter (inside airflow conf), it does not work. --> today we pass threw file share (azure) mounted as PV inside K8S, then logstash pipeline, but it costs a lots per years ... I think the adapter is broken, even when trying to test a connection directly inside the webserver, we have an error : `'ESConnection' object has no attribute 'close'` Then when trying to send logs to elasticsearch, it does not try to send logs and it cannot connect to the elasticsearch when trying to get back logs. Anyway, while trying to look for logs inside a DAG we also have this error : `elasticsearch.AuthenticationException: AuthenticationException(401, 'security_exception', 'missing authentication credentials for REST request [/airflow-logs-*/_count]') airflow-webserver-1 | 172.18.0.1 - - [06/Dec/2024:11:26:46 +0000] "GET /api/v1/dags/debug_airflow_to_elastic/dagRuns/manual__2024-12-06T11:26:37.042642+00:00/taskInstances/print_debug_message/logs/1?full_content=false HTTP/1.1" 500 1588 "http://localhost:8080/dags/debug_airflow_to_elastic/grid?dag_run_id=manual__2024-12-06T11%3A26%3A37.042642%2B00%3A00&task_id=print_debug_message&tab=logs"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"` But the I tried with an api key, user password and even both but cannot get rid of it. I think there is a bug around it or did we do something wrong. Thousands thanks ! Benjamin ### What you think should happen instead? _No response_ ### How to reproduce Just add a connection to elasticsearch, try to connect to it. Then add remote logging inside conf. It does not try to send logs and it cannot connect to the elasticsearch. ### Operating System Kubernetes and docker compose. (both d'ont work) ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details ` [core] sensitive_var_conn_names = key,login,secret,pass,auth hide_sensitive_var_conn_fields = True # for splitter of knumh, the result was over 7k max_map_length = 16396 expose_config = non-sensitive-only load_examples = False test_connection = Enabled [webserver] show_trigger_form_if_no_params = True allow_testing_connections = Enabled [logging] remote_logging = True remote_log_conn_id = elasticsearch_default logging_level = INFO [elasticsearch] host = ************************************ write_stdout = True json_format = True index_patterns = airflow-logs-* [elasticsearch_config] verify_certs=False ` ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
