amindadgar opened a new issue, #38064:
URL: https://github.com/apache/airflow/issues/38064
### Apache Airflow version
2.8.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
There's a conflict between the newest version llama-index sqlalchemy
dependency version and airflow sqlalchemy. When trying to setup airflow using
docker, the airflow-init service would raise the following error
```
airflow-dags-airflow-init-1 | ....................
airflow-dags-airflow-init-1 | ERROR! Maximum number of retries (20)
reached.
airflow-dags-airflow-init-1 |
airflow-dags-airflow-init-1 | Last check result:
airflow-dags-airflow-init-1 | $ airflow db check
airflow-dags-airflow-init-1 | Traceback (most recent call last):
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-dags-airflow-init-1 | from airflow.__main__ import main
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/__init__.py", line
68, in <module>
airflow-dags-airflow-init-1 | settings.initialize()
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line
546, in initialize
airflow-dags-airflow-init-1 | configure_orm()
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line
242, in configure_orm
airflow-dags-airflow-init-1 | engine =
create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args,
future=True)
airflow-dags-airflow-init-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1 | File "<string>", line 2, in
create_engine
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py",
line 281, in warned
airflow-dags-airflow-init-1 | return fn(*args, **kwargs) # type:
ignore[no-any-return]
airflow-dags-airflow-init-1 | ^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/create.py",
line 612, in create_engine
airflow-dags-airflow-init-1 | dialect = dialect_cls(**dialect_args)
airflow-dags-airflow-init-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
line 653, in __init__
airflow-dags-airflow-init-1 | self.executemany_mode =
parse_user_argument_for_enum(
airflow-dags-airflow-init-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1 | File
"/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py",
line 1734, in parse_user_argument_for_enum
airflow-dags-airflow-init-1 | raise exc.ArgumentError(f"Invalid
value for '{name}': {arg!r}")
airflow-dags-airflow-init-1 | sqlalchemy.exc.ArgumentError: Invalid
value for 'executemany_mode': 'values'
airflow-dags-airflow-init-1 |
airflow-dags-airflow-init-1 exited with code 1
service "airflow-init" didn't complete successfully: exit 1
```
I checked the sqlalchemy library version and I see it is listed as version
2.0.28 and checking the `executemany_mode` value in apache airflow setting.py
codes I can see it is set to `values` which is not valid for the newer version
of sqlalchemy.
My Question is how to set `executemany_mode` in docker when trying to
initialize airflow, so we won't face this error? or is there another option
that we could do to fix this error?
_Note:_ I've tried lowering the sqlalchemy library version but when trying
to install dependencies I'm facing another error
```
The conflict is caused by:
The user requested SQLAlchemy==1.4.49
sqlalchemy[asyncio] 2.0.28 depends on sqlalchemy 2.0.28 (from
https://files.pythonhosted.org/packages/0b/50/2b8277511bf6744bc8c98114b61cf96002e6144fe7df5413c47e095de2b2/SQLAlchemy-2.0.28-cp310-cp310-macosx_11_0_arm64.whl
(from https://pypi.org/simple/sqlalchemy/) (requires-python:>=3.7))
The user requested SQLAlchemy==1.4.49
sqlalchemy[asyncio] 2.0.27 depends on sqlalchemy 2.0.27 (from
https://files.pythonhosted.org/packages/ed/16/b51a2647e399f51b21a9b707b30a4286678a4259bdc24487e33de92eabf2/SQLAlchemy-2.0.27-cp310-cp310-macosx_11_0_arm64.whl
(from https://pypi.org/simple/sqlalchemy/) (requires-python:>=3.7))
The user requested SQLAlchemy==1.4.49
sqlalchemy[asyncio] 2.0.26 depends on sqlalchemy 2.0.26 (from
https://files.pythonhosted.org/packages/b1/cf/b25482e8d225a7cf180649f3be2ad48e9ccf3cdea2bda14e8c244e0b4e16/SQLAlchemy-2.0.26-cp310-cp310-macosx_11_0_arm64.whl
(from https://pypi.org/simple/sqlalchemy/) (requires-python:>=3.7))
The user requested SQLAlchemy==1.4.49
sqlalchemy[asyncio] 2.0.25 depends on sqlalchemy 2.0.25 (from
https://files.pythonhosted.org/packages/79/c8/13b106217110e84e9282cdbd548f3d33e33365586cf18f85b930ad17e47c/SQLAlchemy-2.0.25-cp310-cp310-macosx_11_0_arm64.whl
(from https://pypi.org/simple/sqlalchemy/) (requires-python:>=3.7))
```
### What you think should happen instead?
I know why this happened and it is because of the old sqlalchemy version set
in airflow. The error was expected to happen but is there a way to solve it
without a huge number of changes?
### How to reproduce
Please head to the
https://github.com/TogetherCrew/airflow-dags/tree/feat/upgrade-llama-index-dependency
and clone the code. Create `.env.airflow` and `.env.airflow.init`, Finally run
the `docker compose -f docker-compose.yaml up --build`
### Operating System
MacOS Sonoma 14 and Ubuntu 2022
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
