kaxil opened a new pull request, #77166:
URL: https://github.com/apache/airflow/pull/77166

   An explicit `[celery] result_backend` with a driverless URL such as 
`db+postgresql://user:pass@host/db` goes to Celery unchanged, and Celery passes 
it to SQLAlchemy's `create_engine`. SQLAlchemy 2.0 picks psycopg2 for a 
driverless `postgresql://` URL, so when psycopg2 is not installed the scheduler 
and every Celery worker crash with `ModuleNotFoundError: No module named 
'psycopg2'`, even though psycopg (v3) is installed and the metadata database 
already uses it.
   
   The derived case already handles this. When `result_backend` is not set, the 
provider builds it from `sql_alchemy_conn` and turns a driverless 
`postgresql://` into `postgresql+psycopg://`, or `postgresql+psycopg2://` 
without psycopg and SQLAlchemy 2.0 
([default_celery.py](https://github.com/apache/airflow/blob/e306ae6529f2e1e725b69300c2620580ff2d532f/providers/celery/src/airflow/providers/celery/executors/default_celery.py#L156-L167),
 from #69526). Core does the same for a driverless `[database] 
sql_alchemy_conn` 
([configuration.py](https://github.com/apache/airflow/blob/e306ae6529f2e1e725b69300c2620580ff2d532f/airflow-core/src/airflow/configuration.py#L376-L393)).
 The explicit setting never got that treatment, so the same URL works as 
`sql_alchemy_conn` and crashes as `result_backend`. The Helm chart writes its 
result backend as `db+<protocol>://` from `data.resultBackendConnection` into 
`AIRFLOW__CELERY__RESULT_BACKEND` 
([template](https://github.com/apache/airflow/blob/e306ae65
 
29f2e1e725b69300c2620580ff2d532f/chart/templates/secrets/result-backend-connection-secret.yaml#L49)),
 and its example uses `protocol: postgresql`, so chart deployments without 
psycopg2 hit this too.
   
   The fix moves the existing rewrite into one helper that both paths go 
through. A URL that names a driver is left alone. The rewrite runs before the 
engine options are built, so an explicit driverless URL now also gets the 
`prepare_threshold=None` default for psycopg from #74841.
   
   **With both drivers installed, an explicit driverless result backend moves 
from psycopg2 to psycopg.** That matches what the metadata database and the 
derived result backend already use; set `db+postgresql+psycopg2://` to keep 
psycopg2. The #69526 changelog note said an explicit `result_backend` was 
unaffected, so this adds a note for the next release, and the option's 
description now says how the driver is chosen. On Airflow 2.11 with SQLAlchemy 
1.4 nothing changes: the rewrite produces `+psycopg2`, which SQLAlchemy 1.4 
picked anyway.
   
   Run with CeleryExecutor on Postgres, Python 3.12, SQLAlchemy 2.0.57, celery 
5.6.3, psycopg 3.3.8, psycopg2 uninstalled, and `[celery] result_backend = 
db+postgresql://...`:
   
   | | main | this PR |
   |---|---|---|
   | Result backend engine | `ModuleNotFoundError` | driver `psycopg`, `select 
1` works |
   | Celery worker | exits at startup | `celery@... ready.` |
   | Scheduler | crashes once the executor touches the backend | runs, no 
tracebacks |
   | Dag run with 4 tasks | never completes, 0 tasks run | success, 4 tasks 
succeeded |
   
   Scheduler on main (it also crashed from `executor.heartbeat()` in another 
run):
   
   ```
   File "airflow/providers/celery/executors/celery_executor.py", line 348, in 
try_adopt_task_instances
   File "celery/backends/database/session.py", line 56, in get_engine
   File "sqlalchemy/engine/create.py", line 617, in create_engine
   File "sqlalchemy/dialects/postgresql/psycopg2.py", line 697, in import_dbapi
   ModuleNotFoundError: No module named 'psycopg2'
   ```
   
   The `explicit-driverless` test cases fail on main with `'psycopg2' != 
'psycopg'` and pass here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to