moomindani commented on issue #39448: URL: https://github.com/apache/airflow/issues/39448#issuecomment-4862263598
Investigated this at a maintainer's request. Summary: the reported problem is fixed on current versions — validated today against a live Databricks SQL warehouse, on current `main`, through the Airflow 3 task-sdk XCom serialization path (`airflow.sdk.bases.xcom.BaseXCom.serialize_value`): - Multi-row / multi-type SELECT (int, double, string, boolean, date, timestamp) via both `DatabricksSqlHook.run` and `DatabricksSqlOperator.execute`: results are converted to plain namedtuples and JSON-serialize cleanly through XCom - Single-row, empty-result, and multi-statement variants also pass Why it is fixed: - Provider side: `DatabricksSqlHook._make_common_data_structure` converts `databricks.sql.types.Row` into namedtuples before results leave the hook - Core side: Airflow 3 removed the pickle XCom path entirely, and serde handles namedtuples as tuples, so the dynamic-class identity problem from the 2.x pickle path is gone (same conclusion as #59103, closed) One adjacent gap the validation surfaced: `hook.run(..., handler=fetch_one_handler)` on an empty result set raised `TypeError: Expected Sequence[Row] or Row, but got <class 'NoneType'>` because `_make_common_data_structure` did not accept the `None` that DBAPI `fetchone()` returns. That is a separate, much smaller bug — fix in #69230. Recommend closing this issue as resolved. Users on Airflow 2.x hitting the pickle variant of this should upgrade to Airflow 3 with a current databricks provider. --- Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
