hussein-awala opened a new pull request, #77198: URL: https://github.com/apache/airflow/pull/77198
## Why The HITL review plugin reads the review session and the agent outputs straight from the XCom table (`row.value`). A custom XCom backend that keeps values outside the metadata database stores only a reference there. For example, with `XComObjectStorageBackend` from `common.io` and `xcom_objectstorage_threshold = 0`, every worker-written review XCom is a path such as `s3://.../xcom-store/...` or `file:///...`. The plugin then fails to parse the session (`AgentSessionData.model_validate` on a string), so the review tab and the `find`, `feedback`, `approve` and `reject` endpoints fail. Workers are unaffected because they read through the backend. ## How - `_read_xcom` and `_read_xcom_by_prefix` pass each row to the configured backend's `deserialize_value`, which is how workers read these values. `_read_xcom_by_prefix` now selects whole XCom rows instead of the key and value columns. The Airflow 3.4 region scoping (`region_id`, `include_node_regions=False`) is unchanged. - `XCom` comes from `common.compat.sdk` (`airflow.sdk.execution_time.xcom`, which resolves `[core] xcom_backend`), available on every Airflow version the plugin loads on (3.1+) and in the minimum `common.compat` the provider requires. - With the default backend nothing changes: the stored values are plain JSON (strings and `model_dump(mode="json")` dicts), which `BaseXCom.deserialize_value` returns unchanged. The existing tests, including the JSON-looking string outputs covered after #70070, still pass. - Plugin writes stay native. The plugin's own entries (human action, feedback, updated session) are written to the database as before, and `XComObjectStorageBackend.deserialize_value` returns a value that is not a reference unchanged, both on the worker and in the plugin. Writing through the backend would also need serialization and write access to the backend's storage from the API server, and it is not needed to fix the bug. - Docs: the HITL review page now says the plugin reads review XComs through the configured backend, so such a backend must also be installed and configured on the API server with access to its storage. One sentence in the observability page that said the API server never runs a backend's `deserialize_value` now refers to the public XCom endpoint only. Note: this runs the configured backend's `deserialize_value` on the API server for these review XComs only. That is needed because the plugin has to parse the session, not only display it. The public XCom endpoint keeps returning what is stored in the database. ## Tests New `TestReferenceStoringXComBackend` in `test_hitl_review.py` patches the plugin's `XCom` with a stub `BaseXCom` subclass that resolves references from an in-memory store, like object-storage backends do. The session and the latest output are stored as references, an earlier output and human feedback are stored natively. It covers `find` and the `feedback`, `approve` and `reject` actions, whose responses also read back the session the plugin has just rewritten natively. All four tests fail on `main` with the `AgentSessionData` validation error. - `uv run --project providers/common/ai pytest providers/common/ai/tests/unit/common/ai/plugins/` (main sources): 113 passed, 1 skipped - `breeze shell --use-airflow-version 3.3.2 --mount-sources providers-and-tests`, `test_hitl_review.py` and `test_hitl_review_compat.py`: 78 passed, 22 skipped (3.4-only tests); this covers the pre-3.4 query path - Same on `--use-airflow-version 3.1.8`, the oldest version the plugin loads on: 68 passed, 32 skipped - `uv run --project providers/common/ai pytest providers/common/ai/tests/unit`: 5019 passed, 45 skipped, 2 failed. The 2 failures are `sandbox/test_base.py` list-directory tests that need GNU `find -printf`, which the macOS host lacks; they are unrelated to this change - `prek run --stage pre-commit --files <changed files>`: all passed - `prek run --stage manual --skip compile-ui-assets-dev --skip view-skill-eval --skip run-skill-eval-codex --from-ref upstream/main` (includes `mypy-providers`): all passed - `breeze build-docs common.ai`: successful Not run locally: the full provider suites `breeze verify` lists for dependent providers (amazon, google, standard and others); CI covers them. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
