sumeshpremraj opened a new issue, #32399: URL: https://github.com/apache/airflow/issues/32399
### What do you see as an issue? I wrote a DAG which uses BigQueryGetDatasetOperator, following [this doc](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html#get-dataset-details). The DAG fails with what seems like a JSON serialize error. I'm not sure if the documentation is outdated, or there is a bug in the operator itself. ### Solving the problem _No response_ ### Anything else Logs: ``` [2023-07-06, 08:31:23 UTC] {taskinstance.py:1327} INFO - Executing <Task(BigQueryGetDatasetOperator): test-bq> on 2023-07-06 08:31:15.362389+00:00 [2023-07-06, 08:31:23 UTC] {standard_task_runner.py:57} INFO - Started process 53 to run task [2023-07-06, 08:31:23 UTC] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', '***_tests', 'test-bq', 'manual__2023-07-06T08:31:15.362389+00:00', '--job-id', '31', '--raw', '--subdir', 'DAGS_FOLDER/code/tests.py', '--cfg-path', '/tmp/tmp74oqsglt'] [2023-07-06, 08:31:23 UTC] {standard_task_runner.py:85} INFO - Job 31: Subtask test-bq [2023-07-06, 08:31:23 UTC] {task_command.py:410} INFO - Running <TaskInstance: airflow_tests.test-bq manual__2023-07-06T08:31:15.362389+00:00 [running]> on host 9c1ce2a5c3b0 [2023-07-06, 08:31:23 UTC] {taskinstance.py:1545} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='***_tests' AIRFLOW_CTX_TASK_ID='test-bq' AIRFLOW_CTX_EXECUTION_DATE='2023-07-06T08:31:15.362389+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-07-06T08:31:15.362389+00:00' [2023-07-06, 08:31:23 UTC] {crypto.py:83} WARNING - empty cryptography key - values will not be stored encrypted. [2023-07-06, 08:31:23 UTC] {base.py:73} INFO - Using connection ID 'google_cloud_default' for task execution. [2023-07-06, 08:31:23 UTC] {bigquery.py:1963} INFO - Start getting dataset: *** [2023-07-06, 08:31:23 UTC] {credentials_provider.py:323} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook. [2023-07-06, 08:31:25 UTC] {bigquery.py:1090} INFO - Dataset Resource: Dataset(DatasetReference('**', '**')) [2023-07-06, 08:31:25 UTC] {xcom.py:634} ERROR - Object of type Dataset is not JSON serializable. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your *** config or make sure to decorate your object with attr. [2023-07-06, 08:31:25 UTC] {taskinstance.py:1824} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/json.py", line 91, in default return serialize(o) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/serialization/serde.py", line 171, in serialize raise TypeError(f"cannot serialize object of type {cls}") TypeError: cannot serialize object of type <class 'google.cloud.bigquery.dataset.Dataset'> During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 73, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2354, in xcom_push XCom.set( File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 73, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/xcom.py", line 237, in set value = cls.serialize_value( File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/xcom.py", line 632, in serialize_value return json.dumps(value, cls=XComEncoder).encode("UTF-8") File "/usr/local/lib/python3.9/json/__init__.py", line 234, in dumps return cls( File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/json.py", line 104, in encode return super().encode(o) File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/json.py", line 93, in default return super().default(o) File "/usr/local/lib/python3.9/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Dataset is not JSON serializable [2023-07-06, 08:31:25 UTC] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=***_tests, task_id=test-bq, execution_date=20230706T083115, start_date=20230706T083123, end_date=20230706T083125 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
