Owen-CH-Leung commented on issue #44724: URL: https://github.com/apache/airflow/issues/44724#issuecomment-2548412080
From your error log, it seems that the elasticsearch cluster has a security setup to prevent unauthorised access from your k8s cluster. The `AuthenticationException` is a clear indication. I'd advise to start by testing connectivity outside of Airflow to narrow down the root cause. For example, try running a standalone Python script inside the same Kubernetes cluster that hosts your Airflow environment. In that script, use the official [elasticsearch-py] (https://github.com/elastic/elasticsearch-py) client library to connect to your Elasticsearch cluster and try to do sth like `es_client.ping()`. Make sure to experiment with SSL-related parameters such as ssl_verify and ca_certs until you can reliably connect. Once you've confirmed that your Python script can successfully interact with Elasticsearch, you can mirror those working configurations in your airflow.cfg (e.g., adjusting the Elasticsearch configuration sections) and restart Airflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
