jscheffl opened a new issue, #59103:
URL: https://github.com/apache/airflow/issues/59103
### Apache Airflow version
2.11.0
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
Using DatabricksSqlOperator in Airflow 2.11.0 raises an exception when
executing and pickling results to XCom as result.
Dag code:
```
dbx_task = DatabricksSqlOperator(
task_id="dbx_query",
sql="SELECT execution_environment FROM
silver.digital_testing.reports",
)
```
Exception:
```
[2025-12-03, 17:36:03 CET] {taskinstance.py:3348} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 798, in _execute_task
task_instance.xcom_push(key=XCOM_RETURN_KEY, value=xcom_value,
session=session_or_null)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py",
line 94, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 3680, in xcom_push
XCom.set(
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/api_internal/internal_api_call.py",
line 166, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py",
line 94, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/xcom.py",
line 250, in set
value = cls.serialize_value(
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/xcom.py",
line 687, in serialize_value
return pickle.dumps(value)
^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: Can't pickle <class
'airflow.providers.databricks.hooks.databricks_sql.Row'>: it's not the same
object as airflow.providers.databricks.hooks.databricks_sql.Row
```
I attempted multiple up- and downgrades including
apache-airflow-providers-databricks==[6.13.0|7.0.0|7.6.0|7.8.0]
(Also noting that 6.13.0 was noted "working" in bug report
https://github.com/apache/airflow/issues/39448)
As of Slack discussion () I also attempted to make a workaround via
```
def row_to_dict(cursor):
columns = [col[0] for col in cursor.description]
return [dict(zip(columns, row)) for row in cursor.fetchall()]
dbx_task = DatabricksSqlOperator(
task_id="dbx_query",
sql="SELECT execution_environment FROM silver.digital_testing.reports",
handler=row_to_dict
)
```
...but this failed in another exception before reaching the custom handler
code:
````
[2025-12-03, 18:38:38 CET] {taskinstance.py:3348} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 776, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 742, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py",
line 424, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/common/sql/operators/sql.py",
line 312, in execute
output = hook.run(
^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_sql.py",
line 308, in run
result = self._make_common_data_structure(raw_result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_sql.py",
line 334, in _make_common_data_structure
rows_fields = tuple(rows[0].__fields__)
^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute '__fields__'
```
Another attempt was to test SAME provider in Airflwo 3.1.3 and there it is
working. But have the immediate demand to have DBX Query Operator running in
Airflow 2.11. Any ideas welcome
### What you think should happen instead?
Results should be in XCom, no error
### How to reproduce
User Airflow 2.11.0 Docker and install
apache-airflow-providers-databricks==7.8.0 and run a Dag with the code above.
### Operating System
Kubernetes with Docker from Docker-Hub + Provider
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-kafka==1.10.0
apache-airflow-providers-celery==3.12.1
apache-airflow-providers-cncf-kubernetes==10.10.0
apache-airflow-providers-databricks==7.6.0
apache-airflow-providers-docker==4.4.1
apache-airflow-providers-edge3==1.6.0rc1
apache-airflow-providers-ftp==3.13.1
apache-airflow-providers-http==5.3.2
apache-airflow-providers-imap==3.9.1
apache-airflow-providers-sqlite==4.1.1
apache-airflow-providers-standard==1.4.1
apache-airflow-providers-microsoft-azure==12.8.1
apache-airflow-providers-common-compat==1.9.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
Any hints welcome as workaround, if a workaround is needed I am also happy
to patch or contribute back a fix with hints how to resolve.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
