kaxil opened a new pull request, #73991:
URL: https://github.com/apache/airflow/pull/73991
## Summary
The Common AI provider required Airflow 3.0, so an Airflow 2.11 deployment
could not install it at all. This lowers the floor to `apache-airflow>=2.11.0`,
the floor `common.compat`, `standard` and `common.sql` already carry. When the
community provider floor moves to 3.1, this provider moves with it.
On 2.11 the operators, decorators, hooks and toolsets run as they do on
Airflow 3.0. Features that need a newer core already have version gates, which
this PR leaves in place: approval gates and HITL review (3.1), and retry
policies, tool-approval pause and resume, and the task state store (3.3).
Approval gates and HITL review fail at Dag parse time with a message naming the
version. `LLMRetryPolicy` fails at import. An approval-required tool fails the
task, as it already does on 3.0 to 3.2. Durable execution falls back to
`durable_cache_path` as on 3.0 to 3.2.
How this was checked:
| Check | Result |
| -- | -- |
| Provider unit tests on Airflow 2.11.1, the `Compat 2.11.1` job's path |
2313 passed, 0 failed |
| Provider unit tests on Airflow 3 | 2890 passed |
| `common.compat` tests on 2.11.1 / Airflow 3 | 243 / 278 passed |
| One Dag under the scheduler in the `apache/airflow:2.11.2` image, against
a real model | All 14 task instances in the expected state |
The Dag covers `LLMOperator`, `@task.llm`, `AgentOperator` with a function
toolset and with `SQLToolset`, `@task.agent` structured output consumed
downstream, `LLMBranchOperator`, `LLMSQLQueryOperator`, `durable=True` and a
mapped `LLMOperator`.
## Design rationale
- **Most of the provider already went through `common.compat.sdk`.** The
gaps were a few direct Airflow 3 imports and APIs:
- `SET_DURING_EXECUTION` in the decorators
- `BaseHook.get_hook(hook_params=...)`
- `TaskInstance.id`
- `ObjectStoragePath` from `airflow.sdk`
- `structlog`, which the provider imported but never declared; Airflow 3
brings it in through the Task SDK and Airflow 2 does not.
- **`common.compat` changes behaviour on Airflow 2, deliberately.**
- `get_current_context` now resolves to the standard provider's Airflow 2
fallback before core's. Outside a task it raises `RuntimeError`, as Airflow 3
does, instead of core's `AirflowException`. Without the standard provider it
still falls back to core. No caller in this repository catches
`AirflowException` from the compat name; `SandboxToolset` catches
`RuntimeError`, so on Airflow 2 its `attach_to` outside a task failed instead
of falling back to `owner=`.
- `SET_DURING_EXECUTION` is new in compat. On Airflow 2 it is an
`ArgNotSet` whose `repr` matches Airflow 3's sentinel. With a bare `NOTSET`,
Airflow 2 stores the decorator's `prompt` template field as an object address,
which differs per process and changes the serialized Dag's hash.
- The `common-compat` line in the Common AI `pyproject.toml` is marked `#
use next version`, since the decorators need the new export.
- **`PydanticAIHook.get_hook` mirrors Airflow 3's `BaseHook.get_hook` body**
(`get_connection(conn_id).get_hook(hook_params=...)`). Overriding the
classmethod keeps the operators, and the tests that patch `get_hook`,
unchanged. Calling `get_connection().get_hook()` from the operators instead
would have bypassed those patches.
- **The agent's per-attempt run key** (pydantic-ai `run_id`, the `run_id`
XCom, `gen_ai.agent.call.id`) falls back to
`dag_id/run_id/task_id/map_index/try_number` where the task instance has no
`id`. Spans leave out `airflow.task_instance.id` there, because a composite is
not a task-instance id.
- **Logging on Airflow 2.** Nothing configures structlog on Airflow 2, so
the provider's module loggers printed every level, `debug` included, into task
logs. `get_task_logger()` wraps the `airflow.task` stdlib logger on Airflow 2,
so the task log's level and handlers apply and the record names the caller's
line. It does not change the process-wide structlog configuration. On Airflow 3
it returns the same logger as before.
- **`AgentOperator` declares the HITL review extra link only on 3.1+.** On
Airflow 2 the webserver logs an error for the unregistered link class on every
Dag load, and the link can never render there.
- **CI:** `common.ai` is removed from `remove-providers` in the `Compat
2.11.1` row, so its tests run in the existing job. Test files that imported
`airflow.sdk` directly now import through `common.compat.sdk`. Two tests are
gated to Airflow 3:
- Airflow 2 reports a missing constructor argument as `AirflowException`.
- Airflow 2's `MappedOperator` resolves an expansion through the metadata
database and a task session, which a unit test with a dict context cannot drive.
## Gotchas
- Install Airflow with its constraints file, then add the provider without
it: the 2.11 constraints pin `common-compat` and `common-sql` below this
provider's floors. Airflow 2.11.0 installed without constraints can pick up
`universal-pathlib` 0.3, which its `ObjectStoragePath` rejects; 2.11.1 and
later cap it. The installation page now says this.
- The `skills` and `git` extras need `apache-airflow-providers-git`, which
requires Airflow 3.
- Before Airflow 3.3, a structured output reaches downstream tasks as a
`dict`, the same as on 3.0 to 3.2. The connection form's Model field needs 3.2;
on older cores the model goes in Extra.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
