vinothchandar opened a new pull request, #19265: URL: https://github.com/apache/hudi/pull/19265
### Describe the issue this Pull Request addresses Closes #19257 (initial implementation and developer experience), the first sub-issue of the agentic-lakehouse umbrella #19256: give Hudi an optional, self-hostable serving layer so every Hudi table becomes something you can talk to — an agent that inspects the catalog, writes guarded SQL, runs it through Trino, and answers grounded in real query results. ### Summary and Changelog Two new top-level, fully additive modules — nothing in the existing build or runtime changes. **`hudi-agent-gateway/`** — a Python (FastAPI + LangGraph) service, one process serving: - Agent loop over guarded lakehouse tools: `query_lakehouse` (single read-only SELECT, AST-validated via sqlglot, LIMIT injected/capped, time-bounded), `list_tables`, `describe_table` - `POST /v1/chat` with sessions and SSE streaming; `/v1/models`, `/v1/tools`, `/v1/info`, `/health`, `/ready` - MCP server mounted at `/mcp` so external agents (e.g. Claude) can operate on the lakehouse with the same guarded tools - First-party chat UI at `/ui/` (zero third-party JS/CSS; ASF-header-clean) with model picker and a Connect panel showing Trino/inference/MCP endpoints - Pluggable inference: Ollama (default), Anthropic, OpenAI, or any OpenAI-compatible URL (covers vLLM); provider-aware live model discovery - 81 offline tests (no live LLM/Trino needed), ruff + mypy clean; gated live-stack integration tests **`hudi-lakehouse/`** — deployment for the above: - `charts/hudi-trino`: Trino 472 with the repo-built Hudi connector; thrift/Glue metastore and S3/MinIO via values - `charts/hudi-agent-gateway`: the gateway, with a standard Secret convention (`envFrom`, `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`) and probes - `charts/vllm`: optional in-cluster inference (ungated open model default) - `local-dev/`: minikube environment (MinIO, Derby HMS, ASF spark-kubernetes-operator) + a copy-me SparkApplication example that writes a Hudi table and hive-syncs it; `scripts/quickstart.sh` for one-command bring-up - CI: a path-filtered GitHub Actions workflow runs the gateway test suite (Python 3.11/3.12) and helm lint/template Also: one line added to the root `pom.xml` rat excludes (`**/py.typed`). No code copied from other projects; UI and markdown renderer are original. ### Impact No changes to existing Hudi APIs, configs, storage format, or bundles. New optional modules only; Maven reactor untouched (the new modules are not wired into it). New user-facing surface: the gateway HTTP API/UI and Helm values documented in the module READMEs. ### Risk Level low — additive code paths only. Verified end-to-end on minikube: Spark (ASF operator) writes a partitioned Hudi COW table to MinIO with hive-sync; Trino serves it through the in-repo connector; the agent answers analytical questions grounded in query results (validated against a ~60M-row NYC TLC dataset); MCP session exercised from Claude; SSE/UI verified in-browser. The gateway test suite runs offline in the new CI workflow. ### Documentation Update Module READMEs cover install, values, providers, endpoints, and local-dev quickstart. Website docs for the new surface can follow as the modules stabilize (tracked under #19256). ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
