phanikumv commented on issue #31753:
URL: https://github.com/apache/airflow/issues/31753#issuecomment-1580789960
Yes @Stormhand I was able to reproduce this issue. I am getting this error
on 2.6.1 with the below DAG
cc @jedcunningham @potiuk
```
from datetime import datetime
from airflow import DAG
from airflow.providers.databricks.operators.databricks_sql import
DatabricksSqlOperator
with DAG(
dag_id="example_databricks_sql",
schedule_interval=None,
start_date=datetime(2022, 1, 1),
tags=["example", "databricks"],
catchup=False,
) as submit_dag:
get_max_id_task = DatabricksSqlOperator(
databricks_conn_id="databricks_default",
sql_endpoint_name='astro-sdk-test',
task_id='get_max_id',
sql="SELECT cast(max(col2) as STRING) FROM test_table2",
do_xcom_push=True
)
```
```
*** Found local files:
*** *
/usr/local/airflow/logs/dag_id=example_databricks_sql/run_id=manual__2023-06-07T13:08:10.011124+00:00/task_id=get_max_id/attempt=1.log
[2023-06-07, 13:08:10 UTC] {taskinstance.py:1103} INFO - Dependencies all
met for dep_context=non-requeueable deps ti=<TaskInstance:
example_databricks_sql.get_max_id manual__2023-06-07T13:08:10.011124+00:00
[queued]>
[2023-06-07, 13:08:10 UTC] {taskinstance.py:1103} INFO - Dependencies all
met for dep_context=requeueable deps ti=<TaskInstance:
example_databricks_sql.get_max_id manual__2023-06-07T13:08:10.011124+00:00
[queued]>
[2023-06-07, 13:08:10 UTC] {taskinstance.py:1308} INFO - Starting attempt 1
of 1
[2023-06-07, 13:08:10 UTC] {taskinstance.py:1327} INFO - Executing
<Task(DatabricksSqlOperator): get_max_id> on 2023-06-07 13:08:10.011124+00:00
[2023-06-07, 13:08:10 UTC] {standard_task_runner.py:57} INFO - Started
process 852 to run task
[2023-06-07, 13:08:10 UTC] {standard_task_runner.py:84} INFO - Running:
['***', 'tasks', 'run', 'example_databricks_sql', 'get_max_id',
'manual__2023-06-07T13:08:10.011124+00:00', '--job-id', '7', '--raw',
'--subdir', 'DAGS_FOLDER/example_databricks_sql.py', '--cfg-path',
'/tmp/tmpiivo9oy_']
[2023-06-07, 13:08:10 UTC] {standard_task_runner.py:85} INFO - Job 7:
Subtask get_max_id
[2023-06-07, 13:08:10 UTC] {task_command.py:410} INFO - Running
<TaskInstance: example_databricks_sql.get_max_id
manual__2023-06-07T13:08:10.011124+00:00 [running]> on host 7bc0a25b3ba8
[2023-06-07, 13:08:10 UTC] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='example_databricks_sql'
AIRFLOW_CTX_TASK_ID='get_max_id'
AIRFLOW_CTX_EXECUTION_DATE='2023-06-07T13:08:10.011124+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-06-07T13:08:10.011124+00:00'
[2023-06-07, 13:08:10 UTC] {sql.py:265} INFO - Executing: SELECT
cast(max(col2) as STRING) FROM test_table2
[2023-06-07, 13:08:10 UTC] {base.py:73} INFO - Using connection ID
'databricks_default' for task execution.
[2023-06-07, 13:08:11 UTC] {databricks_base.py:424} INFO - Using token auth.
For security reasons, please set token in Password field instead of extra
[2023-06-07, 13:08:11 UTC] {databricks_base.py:424} INFO - Using token auth.
For security reasons, please set token in Password field instead of extra
[2023-06-07, 13:08:13 UTC] {client.py:193} INFO - Successfully opened
session b'\x01\xee\x054[\xb4\x19i\x9f:m\xcd];[W'
[2023-06-07, 13:08:13 UTC] {sql.py:375} INFO - Running statement: SELECT
cast(max(col2) as STRING) FROM test_table2, parameters: None
[2023-06-07, 13:08:15 UTC] {taskinstance.py:1824} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/airflow/utils/session.py",
line 73, in wrapper
return func(*args, **kwargs)
File
"/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line
2354, in xcom_push
XCom.set(
File "/usr/local/lib/python3.10/site-packages/airflow/utils/session.py",
line 73, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/airflow/models/xcom.py",
line 237, in set
value = cls.serialize_value(
File "/usr/local/lib/python3.10/site-packages/airflow/models/xcom.py",
line 632, in serialize_value
return json.dumps(value, cls=XComEncoder).encode("UTF-8")
File "/usr/local/lib/python3.10/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/local/lib/python3.10/site-packages/airflow/utils/json.py", line
102, in encode
o = self.default(o)
File "/usr/local/lib/python3.10/site-packages/airflow/utils/json.py", line
91, in default
return serialize(o)
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
144, in serialize
return encode(classname, version, serialize(data, depth + 1))
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
123, in serialize
return [serialize(d, depth + 1) for d in o]
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
123, in <listcomp>
return [serialize(d, depth + 1) for d in o]
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
123, in serialize
return [serialize(d, depth + 1) for d in o]
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
123, in <listcomp>
return [serialize(d, depth + 1) for d in o]
File
"/usr/local/lib/python3.10/site-packages/airflow/serialization/serde.py", line
132, in serialize
qn = qualname(o)
File
"/usr/local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line
47, in qualname
return f"{o.__module__}.{o.__name__}"
File "/usr/local/lib/python3.10/site-packages/databricks/sql/types.py",
line 161, in __getattr__
raise AttributeError(item)
AttributeError: __name__. Did you mean: '__ne__'?
[2023-06-07, 13:08:15 UTC] {taskinstance.py:1345} INFO - Marking task as
FAILED. dag_id=example_databricks_sql, task_id=get_max_id,
execution_date=20230607T130810, start_date=20230607T130810,
end_date=20230607T130815
[2023-06-07, 13:08:15 UTC] {standard_task_runner.py:104} ERROR - Failed to
execute job 7 for task get_max_id (__name__; 852)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
