cesar-vermeulen opened a new issue, #34099:
URL: https://github.com/apache/airflow/issues/34099
### Apache Airflow version
2.7.0
### What happened
When upgrading apache-airflow-providers-elasticsearch to the newest provider
(5.0.1), Airflow is unable to spin up. The scheudler and airflow-migrations
both crash on following error:
....................
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/logging/config.py", line 573, in configure
handler = self.configure_handler(handlers[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/logging/config.py", line 758, in
configure_handler
result = factory(**kwargs)
^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py",
line 110, in __init__
self.client = elasticsearch.Elasticsearch(host, **es_kwargs) # type:
ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Elasticsearch.__init__() got an unexpected keyword argument
'use_ssl'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/__init__.py", line
68, in <module>
settings.initialize()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line
524, in initialize
LOGGING_CLASS_PATH = configure_logging()
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py",
line 74, in configure_logging
raise e
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py",
line 69, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.11/logging/config.py", line 823, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.11/logging/config.py", line 580, in configure
raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'task'
Exception ignored in atexit callback: <function shutdown at 0x7fb5255f22a0>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/logging/__init__.py", line 2193, in
shutdown
h.close()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py",
line 396, in close
if not self.mark_end_on_close or getattr(self, "ctx_task_deferred",
None):
^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ElasticsearchTaskHandler' object has no attribute
'mark_end_on_close'
2023-09-05T06:04:58.154163010Z
### What you think should happen instead
Deploy Airflow without any crashing schedulers
### How to reproduce
I'm currently running airflow on version 2.7.0 on AKS. I'm trying to upgrade
elasticsearch to package 5.0.1.
Following config is present for elasticsearch:
[elasticsearch]
frontend =
https://*redacted*.kb.westeurope.azure.elastic-cloud.com/app/discover#/?_a=(columns:!(message),filters:!(),index:'6551970b-aa72-4e2a-b255-d296a6fcdc09',interval:auto,query:(language:kuery,query:'log_id:"{log_id}"'),sort:!(log.offset,asc))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1m,to:now))
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}
[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==7.4.2
apache-airflow-providers-docker==3.7.4
apache-airflow-providers-microsoft-azure==4.3.0
apache-airflow-providers-elasticsearch==5.0.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
