cshuo opened a new issue, #20620:
URL: https://github.com/apache/hudi/issues/20620
### Task Description
Implement the Docker Compose packaging portion of RFC-117 so users can run a
local Hudi lakehouse without installing a host JDK, Maven, Kubernetes, or Helm.
Package the existing agent gateway and Spark Connect endpoint with Hive
Metastore, MinIO, and sample data. Preserve the current catalog and query path:
Chat / MCP → Gateway → Spark Connect → HMS-backed HoodieCatalog → Hudi data
in MinIO
This task retains the existing read-only gateway tools and does not
introduce a catalog adapter or semantic layer.
### Scope
- Add a Compose stack containing:
- MinIO for Hudi storage and a one-shot bucket initialization job.
- Hive Metastore with persistent Derby metadata.
- Spark Connect running in local mode with HoodieCatalog.
- A one-shot Python Connect client to seed and verify the trips/drivers
tables.
- The agent gateway exposing the chat UI, HTTP API, and MCP endpoint.
- An optional Ollama service for local inference.
- Add an installation script with Docker/Compose and resource checks,
startup ordering, readiness checks, diagnostics, and lifecycle commands.
- Support host Ollama, containerized Ollama, and configured API providers.
- Provide an MCP-only installation mode that does not require a gateway LLM
or API key.
- Keep the base Compose file image-based and provide a separate opt-in
source-build override.
- Bind published ports to localhost and persist table data, metastore
metadata, and downloaded models in named volumes.
- Document service roles, configuration, client connections, and data
lifecycle.
### User Experience
From a repository checkout:
```bash
./hudi-lakehouse/docker/lakehouse.sh --build --mcp-only
```
For built-in chat using containerized Ollama:
```bash
./hudi-lakehouse/docker/lakehouse.sh --build --local-llm
```
Default endpoints:
- Chat UI: http://localhost:8000/ui/
- MCP: http://localhost:8000/mcp/
- Spark Connect: sc://localhost:15002
- MinIO console: http://localhost:9001
The initial prerequisite is Docker Engine/Desktop with Compose 2.27.2 or
newer. Source builds run inside Docker.
### Acceptance Criteria
- [ ] A clean installation starts successfully and creates readable Hudi
sample tables.
- [ ] MCP table discovery, schema inspection, and SQL queries work through
Spark Connect.
- [ ] A configured local LLM can answer a sample question using the SQL tool.
- [ ] Missing local models and unavailable providers produce actionable
readiness failures.
- [ ] Repeated installation preserves existing sample rows without
introducing duplicates.
- [ ] Container recreation without reseeding preserves tables, partitions,
and data.
- [ ] HMS can recover a missing initialization marker by validating the
existing schema; invalid databases are preserved and startup fails.
- [ ] Cached Ollama models are reused on subsequent installations.
- [ ] Automated regression tests and an opt-in Docker acceptance workflow
cover the installation lifecycle.
### Boundaries and Follow-ups
This issue delivers the checkout-based Compose installation and contributor
build path. Default application image tags are local build tags.
Publishing release images, hosting versioned downloads, and providing the
public `hudi.apache.org/lakehouse.sh` bootstrap require separate
release-infrastructure work tracked by the umbrella issue.
Catalog abstractions, semantic models, optional BI/Trino profiles, and
Kubernetes installation changes are outside this task.
### Related Issues
Parent feature issue: #20619
RFC PR: https://github.com/apache/hudi/pull/20575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]