GitHub user kaxil added a comment to the discussion: Proposal: Databricks Unity AI Gateway model class for apache-airflow-providers-common-ai
Thanks for the detailed writeup, @aman210122. **Path A, gated on a pydantic-ai release (not just the merge).** Quick correction on the upstream status: pydantic/pydantic-ai#4036 is no longer in draft. It's open, ~3,400 lines, 42 commits, 63 review comments, assigned to @DouweM, last activity Apr 6. So the "blocked on upstream timing" framing for Path B is weaker than the proposal suggests. The work is happening, the bottleneck is review on that PR. Why Path A: 1. common.ai's other model hooks (Azure, Bedrock, Vertex) all wrap upstream `pydantic-ai` Models rather than fork them. A local subclass of `OpenAIChatModel` here would duplicate a 3.4k-line PR that's still under active review, and we'd inherit any course corrections that review surfaces. 2. The provider is still 0.x. Adding a class we plan to delete is a deprecation chore we don't need. Why "wait for the release," not just the merge: The `databricks` extra has to pin `pydantic-ai >= X.Y.Z` against a real PyPI version. Until upstream cuts a release containing `DatabricksModel`, the hook's `from pydantic_ai.models.databricks import DatabricksModel` has nothing to resolve against, and pinning a git ref in a provider extra isn't something we do. So the highest-leverage place to push this forward is review on pydantic/pydantic-ai#4036 itself, not a fork PR here. Once that lands and ships in a pydantic-ai release, the common.ai wrapper is roughly a 30-line hook plus a connection-type stub plus docs. On the other questions: - **Home:** `common.ai`, consistent with Bedrock and Vertex precedent. - **Connection name:** `pydanticai-databricks` plus a `databricks` pip extra, matching the existing pattern. - **CI:** mock against `pydantic-ai` test doubles, same as today. No live workspace. - **Surface selection (GA vs Beta):** connection extra field. Two hook subclasses is too much surface for a flag. The AI Gateway feature toggles, embeddings, agent-specific bits are correctly out of v1 scope. **Concretely: nothing to open on apache/airflow right now.** Let's watch pydantic/pydantic-ai#4036 land and get into a pydantic-ai release, and then the common.ai PR is small and mechanical. Happy to review it when that point comes. GitHub link: https://github.com/apache/airflow/discussions/67581#discussioncomment-17069317 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
