ad1happy2go opened a new pull request, #19411:
URL: https://github.com/apache/hudi/pull/19411
### Describe the issue this Pull Request addresses
Part of the agentic-lakehouse umbrella #19256. The v1 gateway (#19265)
serves its guarded lakehouse tools exclusively through Trino — but many Hudi
deployments already run Spark and have no Trino in the picture, and Spark reads
everything Spark can read: MOR snapshot queries and Hudi 1.x tables carrying
the unstructured types (`BLOB`, `VECTOR`, `VARIANT`) that the in-repo Trino
connector does not serve yet. This PR makes the lakehouse engine pluggable:
`GATEWAY_ENGINE=spark` serves the identical tool surface through a Spark Thrift
Server (HiveServer2 protocol), so any Spark cluster with the Hudi bundle
becomes an agent-queryable lakehouse with one env var.
### Summary and Changelog
One selectable backend behind the existing tool contract; the chat API, UI,
MCP server, guardrails, and agent behavior are identical across engines — only
the SQL dialect and the endpoint reported by `/v1/info` change.
**`tools/spark_client.py` — SparkThriftClient (PyHive)**
- Async facade mirroring `TrinoClient`: each query runs the sync client in a
dedicated bounded worker pool under a hard timeout, with best-effort
server-side `cursor.cancel()` on expiry; `ping()` is cached so readiness probes
do not open a fresh Thrift session every period.
- PyHive is an optional dependency (`pip install
'hudi-agent-gateway[spark]'`, pure-sasl — no C toolchain); a missing dependency
degrades to a clear `/ready` detail instead of a crash, and the client records
`last_error` so readiness failures carry the underlying cause (connection
refused vs missing dependency vs timeout).
- HiveServer2 auth modes: `none` (SASL PLAIN, the Spark Thrift default),
`nosasl`, and `ldap` (config validation fails fast if the password is missing);
qualified result column names (`tbl.col`) are normalized to bare names so
results look identical across engines.
**`tools/spark_tools.py` — the same three tools, Spark shape**
- `query_lakehouse` validates and re-renders model SQL through the shared
AST guardrails in the `spark` sqlglot dialect (single statement, SELECT-only,
row cap injected as a real `LIMIT`); backtick identifiers parse and render
correctly.
- Spark has no `information_schema`: `list_tables` uses `SHOW TABLES IN`,
`describe_table` uses `DESCRIBE TABLE`, with the same strict identifier
validation (names arrive over HTTP/MCP and never reach SQL unvalidated) and the
same error-as-payload contract (`{"error", "hint"}`) so the agent
self-corrects; HiveServer2's Java-stack-trace error renderings are reduced to
the first meaningful line.
**Shared plumbing**
- New `tools/common.py` hosts the engine-agnostic pieces (`QueryResult`,
result shaping with row/byte truncation notices, identifier validation) with
`LakehouseQueryError`/`LakehouseTimeoutError` base classes both engines' errors
subclass; the Trino modules re-export their existing names, so nothing
downstream changes.
- `enforce_guardrails` gains a `dialect` parameter (default `trino`), hints
name the dialect the model should write.
- Engine-aware surfaces: `/ready` reports the active engine's check (with
cause detail for spark), `/v1/info` gains `engine` and engine-neutral
`sql_url`/`web_ui_url`, the chat UI connect panel shows "Spark SQL (JDBC)" with
the `jdbc:hive2://` URL and hides the web-UI row when the engine has none, and
the agent system prompt speaks the active dialect (`database.table` vs
`catalog.schema.table`).
- Packaging and deploy: the gateway Docker image installs the `spark` extra
so one image serves both engines; the Helm chart gains an `engine` value with
conditional `GATEWAY_SPARK_*` env (helm lint and template verified for both
engines).
**Tests** — full mirror of the Trino suite plus client-level coverage
- Offline (fake Spark client, no network): tool shaping/truncation,
guardrail rejections with spark-dialect hints, `SHOW TABLES`/`DESCRIBE` SQL
shapes, identifier-injection rejection, not-found mapping from both empty
results and server errors, tool-surface parity across engines, client
timeout→cancel mapping, ping caching, `last_error` lifecycle, concurrency,
config validation, engine-aware readiness/info.
- Live (gated by `GATEWAY_IT_SPARK_HOST`, mirroring the Trino live tests):
list/describe/query against a real Spark Thrift Server serving the same `trips`
dataset the local-dev example writes, plus a live guardrail-rejection check.
No code copied from other projects.
### Impact
New optional engine + configs: `GATEWAY_ENGINE` and
`GATEWAY_SPARK_{HOST,PORT,DATABASE,USER,AUTH,PASSWORD}` (documented in the
module README's config table, with a quickstart section). New pip extra
`spark`. One intentional API rename while the module is days old: `/v1/info`'s
`trino_url`/`trino_ui_url` become engine-neutral `sql_url`/`web_ui_url` (plus a
new `engine` field); the shipped UI is updated in the same PR. No existing Hudi
APIs, configs, storage format, or bundles change.
### Risk Level
low — additive code paths; the Trino backend is untouched apart from
importing shared helpers it re-exports. Verified by the full offline suite (121
passing; ruff and mypy clean), the new gated live integration tests passing 4/4
against a real Spark Thrift Server serving Hudi 1.x tables, and the existing
Trino live integration tests re-run green against the local-dev stack (no
regression). Additionally validated end to end: agent round-trips through
`GATEWAY_ENGINE=spark` with both a local Ollama model and a frontier model
answered aggregation and filtered-lookup questions correctly, and the same
gateway tools queried a Hudi 1.x unstructured table (BLOB struct columns,
`size(embedding)` over a `VECTOR(768)` column) — the concrete capability the
Trino connector cannot serve today.
### Documentation Update
Module README updated in this PR (engine row in the config table, "Using a
Spark Thrift Server instead of Trino" quickstart, live integration-test
instructions). Website docs for the gateway are tracked with the umbrella
#19256.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
