moomindani commented on issue #71525:
URL: https://github.com/apache/airflow/issues/71525#issuecomment-5302280351
Thanks for the thorough write-up, and for coming back with your own
follow-up analysis — that saved a lot of digging.
Your conclusion is right in substance: this is a core-level bug, it is fixed
in core, and the Databricks provider does not need a change. One correction to
the version boundary, though, because it matters a lot for anyone hitting this:
**the fix is already in 3.1.0**, not only in >= 3.2.1. The affected window is
the 3.0.x line.
## Verified on a real workspace
I ran the same deferrable `DatabricksRunNowOperator` against a real
Databricks job on each core version, with the **provider pinned at 7.18.1 in
every run** (official `apache/airflow` images, `airflow standalone`, connection
stored in the metadata DB so the triggerer has to fetch it through the
Execution API):
| core | result | trigger log |
|---|---|---|
| 3.0.6 | task **failed** on the first poll | `RuntimeError: You cannot use
AsyncToSync in the same thread as an async event loop`, frame-for-frame
identical to your traceback (`triggerer_job_runner.py send` ->
`asgiref/sync.py`), then `Trigger exited without sending an event. Dependent
tasks will be failed.` |
| 3.1.0 | task **success** | polled `life_cycle_state: RUNNING` twice,
`Trigger fired event`, `trigger completed` |
| 3.1.1 | task **success** | same |
| 3.2.2 | task **success** | same |
Since the provider was identical across all four runs, the difference is
entirely core-side — which also confirms your point that no provider upgrade
can help someone on 3.0.x.
## Why 3.1.0 already works
The fix from #55799 landed in `task-sdk`'s `Connection.get()` (not in the
secrets backend, which is what makes it easy to miss when reading the diff):
```python
except RuntimeError as e:
if str(e).startswith("You cannot use AsyncToSync in the same thread as
an async event loop"):
import greenback
task = asyncio.current_task()
if greenback.has_portal(task):
warnings.warn("You should not use sync calls here -- use `await
Conn.async_get` instead", stacklevel=2)
return greenback.await_(cls.async_get(conn_id))
```
That is exactly the path the Databricks hook takes: the 3.1.0 and 3.1.1
trigger logs contain that `You should not use sync calls here` warning, emitted
from `BaseHook.get_connection`, immediately before the successful poll. So
`BaseDatabricksHook.databricks_conn` staying a synchronous `cached_property` is
fine from 3.1.0 onward — core absorbs it.
For completeness, the mechanism moved twice after that, which is why the
code looks different depending on which version you read:
- 3.1.1 - 3.2.1: the same greenback fallback also present in
`ExecutionAPISecretsBackend.get_connection` (the #57154 line of work).
- 3.2.2 and later, including `main`: the bridge moved one layer down into
`TriggerCommsDecoder.send()` (#66412), so it now covers **every** sync SDK call
made from a trigger's event-loop thread, not just connection fetches.
## Suggested resolution
- This can be closed as fixed in core, with **Airflow >= 3.1.0** as the
requirement for deferrable Databricks operators. On MWAA 3.0.6 the only fix is
a core upgrade; there is no provider-side workaround.
- #55568 and #63775 were correctly closed — a Databricks-specific async
`databricks_conn` is not needed, and adding one now would duplicate what core
does for every provider.
- The one gap I do see is discoverability: the provider declares
`apache-airflow>=2.11.0`, so it installs happily on 3.0.x where deferrable mode
cannot work, and the resulting failure (`AsyncToSync` deep in `asgiref`) gives
the user nothing to act on. A one-line note in the provider's deferrable docs
stating that deferrable operators need core >= 3.1.0 would have answered this
issue before it was filed. Happy to raise that as a small docs PR if
maintainers think it is worth having.
---
Drafted-by: Claude Code (Opus 5); reviewed by @moomindani before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
