GitHub user nsivabalan created a discussion: RFC-117: Agent-Ready Packaged 
Lakehouse for Apache Hudi

I'd like to start a discussion on **RFC-117: Agent-Ready Packaged Lakehouse for 
Apache Hudi**.

- RFC (draft PR): https://github.com/apache/hudi/pull/20575
- RFC number claim: https://github.com/apache/hudi/pull/20574
- Related: Spark query and write serving endpoint for hudi-lakehouse (#19843)

## Why

Increasingly, the one asking a lakehouse questions is an LLM agent. A user asks 
Claude or ChatGPT "what were gross bookings per city last week?" and expects a 
correct answer computed from their own Hudi tables. Hudi already provides the 
bottom half of a database for the lake: transactional storage, indexes, table 
services, catalog sync and engine integrations. The part that turns a question 
into an answer the user can trust is left for each user to assemble.

Three things make this worth solving now:

- **Pointing an LLM at schemas isn't reliable.** Published text-to-SQL 
benchmarks on realistic schemas report roughly 16–40% accuracy (BIRD, Spider 
2.0, Sequeda et al.). Business meaning has to come from approved semantic 
models, not from the model's guesses.
- **Commercial platforms already offer this.** Snowflake Cortex Analyst, 
Databricks Genie and BigQuery conversational analytics all do, but each is tied 
to its vendor. The open lakehouse community has no equivalent.
- **The pieces are in the repo but not packaged.** We have the agent gateway 
(#19265), Trino and Spark SQL Connect endpoints (#19265, #20243, #20329), BI 
integrations (#20467) and quickstart validation (#20462). Getting to a first 
answer still means assembling everything and building images with a JDK and 
Maven.

## What the RFC proposes

A Hudi lakehouse delivered as one installable unit, fronted by a **gateway** 
that agents connect to over MCP:

- **Spark SQL Connect** is the default query engine. Trino stays supported and 
can be swapped in.
- **Data catalog:** Hive Metastore for v1, behind a catalog adapter interface. 
Apache Gravitino is the leading candidate when we add governance. DataHub and 
OpenMetadata are optional discovery adapters.
- **Semantic layer:** consumes pre-built models only, so the same question 
always produces the same SQL. The shortlist is MetricFlow and Cube Core, with 
Wren and Boring Semantic Layer worth prototyping.
- **Installation:** one command with Docker as the only prerequisite, plus 
local Kubernetes and production Helm tiers, all using the same published images.
- **Later:** governance; Spark job submission as a stretch goal.

## Where feedback would help most

1. **Semantic layer:** MetricFlow (embedded in the gateway) or Cube (a separate 
service that BI tools can share)? Does anyone already run either with Hudi?
2. **Catalog:** is HMS for v1, with Gravitino considered when governance 
arrives, the right call for your deployments? Which catalogs do you run Hudi 
with today?
3. **Packaging:** what's the right way, under ASF policy, to publish images and 
a Helm repository?
4. **Default LLM for the first run:** bring your own API key, a local Ollama, 
or a bundled local model?
5. **Scope:** anything missing, or anything you'd cut, for a first release?

The full list of open questions is at the end of the RFC. Comments are welcome 
here or inline on the PR. Thanks!

GitHub link: https://github.com/apache/hudi/discussions/20602

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to