ramonvermeulen commented on PR #46911: URL: https://github.com/apache/airflow/pull/46911#issuecomment-2731964728
Managed to get rid of the above error, and again pulled in the latest upstream. However, I notice there is an error when the web app tries to fetch the extra link and I have the feeling it might be related to recent changes: https://github.com/apache/airflow/blob/df584d3c8f397dbfec33f383292a9ca7c05dcc07/task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py#L48 in https://github.com/apache/airflow/commit/fe5a2ea7a0b57901bb6239d666b875f6c71cd7e8 Line causing the error https://github.com/apache/airflow/blob/0010596c56f7f8b9607e8d98622fb7c1b7edff71/airflow/models/xcom.py#L338 <details> <summary>Stacktrace</summary> ``` Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 29, in __call__ await responder(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 126, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 46, in __call__ await self.app(scope, receive, self.send_with_compression) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 178, in __call__ recv_stream.close() File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.9/site-packages/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 175, in __call__ response = await self.dispatch_func(request, call_next) File "/opt/airflow/airflow/api_fastapi/core_api/middleware.py", line 28, in dispatch response = await call_next(request) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 153, in call_next raise app_exc File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 140, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 73, in app response = await f(request) File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 214, in run_endpoint_function return await run_in_threadpool(dependant.call, **values) File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool return await anyio.to_thread.run_sync(func) File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 967, in run result = context.run(func, *args) File "/opt/airflow/airflow/api_fastapi/core_api/routes/public/extra_links.py", line 85, in get_extra_links all_extra_links = {link_name: link_url or None for link_name, link_url in sorted(all_extra_link_pairs)} File "/opt/airflow/airflow/api_fastapi/core_api/routes/public/extra_links.py", line 83, in <genexpr> (link_name, task.get_extra_links(ti, link_name)) for link_name in task.extra_links File "/opt/airflow/airflow/serialization/serialized_objects.py", line 1220, in get_extra_links return link.get_link(self.unmap(None), ti_key=ti.key) File "/opt/airflow/airflow/serialization/serialized_objects.py", line 2036, in get_link return XComModel.deserialize_value(value) File "/opt/airflow/airflow/models/xcom.py", line 338, in deserialize_value return json.loads(result.value, cls=XComDecoder) File "/usr/local/lib/python3.9/json/__init__.py", line 359, in loads return cls(**kw).decode(s) File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` </details> Error has to do with the `f"_link_{self.__class__.__name__}"` being retrieved from XCom, however not being json serializable (e.g. it is just a string with the URI and not valid json). I tested existing google operators on the latest main such as `BigQueryCreateEmptyTableOperator` with `BigQueryTableLink` and I get exactly the same error so I have the feeling it is not related to my specific link implementation. To give an example, this is how the XCom looks like for the CloudRunExecuteJobOperator:  And for the BigQueryCreateTableOperator that I tested: <img width="1120" alt="2" src="https://github.com/user-attachments/assets/a4a6c4b7-8ca8-4595-8b97-5bb8ba852079"; /> Both seem to contain a valid link, however the API server throws a 500 with the jsondecode error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
