GitHub user nsivabalan created a discussion: RFC-117: Agent-Ready Packaged Lakehouse for Apache Hudi
I'd like to start a discussion on **RFC-117: Agent-Ready Packaged Lakehouse for Apache Hudi**. - RFC (draft PR): https://github.com/apache/hudi/pull/20575 - RFC number claim: https://github.com/apache/hudi/pull/20574 - Related: Spark query and write serving endpoint for hudi-lakehouse (#19843) ## Why Increasingly, the one asking a lakehouse questions is an LLM agent. A user asks Claude or ChatGPT "what were gross bookings per city last week?" and expects a correct answer computed from their own Hudi tables. Hudi already provides the bottom half of a database for the lake: transactional storage, indexes, table services, catalog sync and engine integrations. The part that turns a question into an answer the user can trust is left for each user to assemble. Three things make this worth solving now: - **Pointing an LLM at schemas isn't reliable.** Published text-to-SQL benchmarks on realistic schemas report roughly 16–40% accuracy (BIRD, Spider 2.0, Sequeda et al.). Business meaning has to come from approved semantic models, not from the model's guesses. - **Commercial platforms already offer this.** Snowflake Cortex Analyst, Databricks Genie and BigQuery conversational analytics all do, but each is tied to its vendor. The open lakehouse community has no equivalent. - **The pieces are in the repo but not packaged.** We have the agent gateway (#19265), Trino and Spark SQL Connect endpoints (#19265, #20243, #20329), BI integrations (#20467) and quickstart validation (#20462). Getting to a first answer still means assembling everything and building images with a JDK and Maven. ## What the RFC proposes A Hudi lakehouse delivered as one installable unit, fronted by a **gateway** that agents connect to over MCP: - **Spark SQL Connect** is the default query engine. Trino stays supported and can be swapped in. - **Data catalog:** Hive Metastore for v1, behind a catalog adapter interface. Apache Gravitino is the leading candidate when we add governance. DataHub and OpenMetadata are optional discovery adapters. - **Semantic layer:** consumes pre-built models only, so the same question always produces the same SQL. The shortlist is MetricFlow and Cube Core, with Wren and Boring Semantic Layer worth prototyping. - **Installation:** one command with Docker as the only prerequisite, plus local Kubernetes and production Helm tiers, all using the same published images. - **Later:** governance; Spark job submission as a stretch goal. ## Where feedback would help most 1. **Semantic layer:** MetricFlow (embedded in the gateway) or Cube (a separate service that BI tools can share)? Does anyone already run either with Hudi? 2. **Catalog:** is HMS for v1, with Gravitino considered when governance arrives, the right call for your deployments? Which catalogs do you run Hudi with today? 3. **Packaging:** what's the right way, under ASF policy, to publish images and a Helm repository? 4. **Default LLM for the first run:** bring your own API key, a local Ollama, or a bundled local model? 5. **Scope:** anything missing, or anything you'd cut, for a first release? The full list of open questions is at the end of the RFC. Comments are welcome here or inline on the PR. Thanks! GitHub link: https://github.com/apache/hudi/discussions/20602 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
