GitHub user cshuo added a comment to the discussion: Spark query and write serving endpoint for hudi-lakehouse
Thanks, Siva, for the detailed review and suggestions. Following today’s meeting, I’ve outlined the v1 scope, validation completed so far, and remaining work below. 1. Scope Focus v1 on interactive SQL for querying and writing Hudi tables. General-purpose DataFrame/DataSource workloads and Spark job submission are out of scope. Direct clients can write; the agent gateway remains read-only. 2. Compute model Follow a Trino-like deployment model: a shared Spark application with configurable workers and optional autoscaling. For v1, we’ll rely on Spark’s existing scheduling, isolation, and recovery behavior, without introducing separate workload pools or custom recovery mechanisms. The current chart runs in local mode; cluster-backed execution still needs validation. 3. Validation completed Local tests cover Superset/Zeppelin SQL reads and INSERT/UPDATE/DELETE through ADBC, Codex/MCP queries through the gateway’s official PySpark Connect client, and multi-turn Gateway Chat UI queries. The Helm quickstart also reads externally written Hudi data, with results matching Trino. 4. Further validation and delivery Remaining checks include cluster-backed execution, long-lived-session metadata freshness, cross-engine compatibility, and MERGE/table-service coverage. Following your feedback, we also need to verify cancellation and timeout/retry behavior, including cancellation propagation to Spark and retries after a lost write response. We’ll prioritize an incremental SQL-serving delivery, document its limits and outstanding validation gaps, and use real usage feedback to guide follow-up work. I’ll address implementation-specific feedback in the PR. GitHub link: https://github.com/apache/hudi/discussions/19843#discussioncomment-19116487 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
