Stormhand commented on issue #31080:
URL: https://github.com/apache/airflow/issues/31080#issuecomment-1578243626
Unfortunately it happened again. This time i need **do_xcom_push**:
```
[2023-06-06, 08:52:24 UTC] {sql.py:375} INFO - Running statement: SELECT
cast(max(id) as STRING) FROM prod.unified.sessions, parameters: None
[2023-06-06, 08:52:25 UTC] {taskinstance.py:1824} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py",
line 73, in wrapper
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py",
line 2354, in xcom_push
XCom.set(
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py",
line 73, in wrapper
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/models/xcom.py",
line 237, in set
value = cls.serialize_value(
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/models/xcom.py",
line 632, in serialize_value
return json.dumps(value, cls=XComEncoder).encode("UTF-8")
File "/usr/local/lib/python3.10/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/json.py", line
102, in encode
o = self.default(o)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/json.py", line
91, in default
return serialize(o)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/serialization/serde.py",
line 144, in serialize
return encode(classname, version, serialize(data, depth + 1))
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/serialization/serde.py",
line 123, in serialize
return [serialize(d, depth + 1) for d in o]
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/serialization/serde.py",
line 123, in <listcomp>
return [serialize(d, depth + 1) for d in o]
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/serialization/serde.py",
line 132, in serialize
qn = qualname(o)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py",
line 47, in qualname
return f"{o.__module__}.{o.__name__}"
File
"/home/airflow/.local/lib/python3.10/site-packages/databricks/sql/types.py",
line 161, in __getattr__
raise AttributeError(item)
AttributeError: __name__. Did you mean: '__ne__'?
```
This is how i use it:
```
get_max_id_task = DatabricksSqlOperator(
databricks_conn_id=databricks_sql_conn_id,
sql_endpoint_name='sql_endpoint',
task_id='get_max_id',
sql="SELECT cast(max(id) as STRING) FROM prod.unified.sessions",
do_xcom_push=True
)
```
Databricks providers, i use the latest:
```
apache-airflow-providers-common-sql==1.5.1
databricks-sql-connector==2.5.2
apache-airflow-providers-databricks==4.2.0
```
Airflow is 2.6.1/python 3.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
