GitHub user nsivabalan added a comment to the discussion: Spark query and write serving endpoint for hudi-lakehouse
Thanks, Shuo, for driving this forward. Spark Connect + ADBC looks like a reasonable starting point. I have a few questions around the broader requirements and longer-term direction. I'm not suggesting we address all of these in v1, but it would be useful to align on the intended scope and validate a few foundational assumptions early, so we don't run into architectural constraints in subsequent phases. ### 1. Areas where alignment would help **1. Scope of the Spark endpoint (v1 and long term)** Is the intent primarily an interactive SQL/DataFrame endpoint, including Spark DataSource reads and writes, or do we eventually envision supporting general-purpose Spark application submission (`spark-submit`) and long-running workloads as well? **2. Compute and resource management (v1 and long term)** Are we expecting multiple callers to share one Spark driver and executor pool? Should callers eventually be able to request independent compute resources (executor counts, memory, etc.)? Worth understanding whether a shared Spark application is the long-term model or just the initial implementation. **3. Workload isolation and resiliency (v1 fundamentals)** What failure isolation are we targeting across concurrent callers? Executor failures should generally be recoverable, but a shared driver failure would impact all active jobs. Is reconnecting and retrying acceptable, or do we envision stronger recovery guarantees over time? **4. Operation lifecycle and retry semantics (long term)** Do we expect synchronous, best-effort execution, or eventually asynchronous operations with status tracking, cancellation, and recovery? This is particularly relevant for writes where a client might lose connectivity after the Hudi commit succeeds. **5. Security and multi-tenancy (long term)** Are we initially targeting trusted clients within a cluster, or do we expect multi-tenant usage with distinct identities, authorization, and resource isolation? We should understand the eventual trust boundary even if v1 is deliberately limited. **6. Resource governance and observability (v1 fundamentals)** How do we manage concurrent operations, runaway queries, resource contention, and cancellation? It would also help to have a way to correlate client requests with Spark jobs and Hudi commits. **7. Extensibility and compatibility (long term)** How do we envision supporting custom UDFs, DataSources, dependencies, and different Spark/Hudi versions? Worth ensuring the initial packaging doesn't constrain future upgrades or workload-specific runtimes. **8. Streaming and long-running jobs (long term)** Should Structured Streaming jobs and long-running ingestion/backfill workloads be supported through the same endpoint, or should they eventually have a separate execution lifecycle? **9. Catalog and metadata consistency (v1 validation)** Can long-lived sessions reliably observe table changes made by external Spark/Flink writers, including schema evolution and metadata updates? **10. Hudi writer configuration (minor clarification)** My assumption is that individual callers will be responsible for supplying the appropriate Hudi writer configurations, including concurrency control and lock provider settings where applicable. The endpoint itself would not enforce multi-writer coordination across sessions. Is that the intended responsibility model? I don't think we need to solve everything upfront. The main thing is to distinguish what we want for v1, what we expect to support eventually, and which architectural assumptions are worth validating now. ### 2. Additional context and possible validations **1. Scope: SQL/DataFrame endpoint vs. general-purpose Spark execution** Spark Connect supports remote SQL and DataFrame operations, including DataSource reads/writes, but it is not a replacement for `spark-submit`. The latter creates independent Spark applications with their own drivers and executors. If the initial goal is interactive SQL and DataFrame access, Spark Connect is a natural fit. If we eventually want general-purpose job submission, we may need a separate execution or orchestration layer. The question is whether we should preserve that extensibility from the beginning, rather than treating one Spark Connect server as the eventual execution abstraction. **2. Compute and resource management** With a shared Spark Connect server, multiple sessions execute within the same Spark application. They share a driver and executor pool. Callers can tune certain session-level Spark properties, but they cannot independently request different executor memory or resource allocations within that shared application. For example, a small interactive query and a large Hudi backfill could have very different resource requirements. It would be useful to validate a cluster-backed Spark Connect deployment with concurrent workloads and understand the limitations of fair scheduling, dynamic allocation, and workload prioritization. Longer term, we may need multiple Spark application pools or independent applications for different workload classes. We don't have to implement that now, but should understand whether the architecture can evolve in that direction. **3. Failure isolation and resiliency** There are different failure boundaries to consider: - A single SQL/DataFrame operation fails: other jobs should generally continue. - An executor fails or OOMs: Spark can retry affected tasks, but executors are shared, so multiple jobs may be affected. - The shared driver fails: all running jobs are interrupted, and clients must reconnect. A particularly important question is whether one problematic workload can destabilize the shared driver, affecting every connected client. A small fault-injection PoC would help: run concurrent workloads, induce executor and driver failures, and observe the impact on unrelated operations. I'm not suggesting we provide complete job-level isolation in v1. But we should know the actual failure boundaries and decide what is acceptable. **4. Operation lifecycle and recovery** Consider a Hudi write that successfully commits but whose response is lost because the client disconnects or the driver crashes. >From the client's perspective, the operation outcome may be unknown. Blindly >retrying could be unsafe, depending on the write semantics. This raises questions around operation identity, status tracking, and the ability to determine whether a write committed successfully. Spark Connect does not inherently provide a durable job execution service. If that's something we eventually need, particularly for agents or automated workloads, we should consider how it might fit into the architecture. A useful validation would be to inject failures around Hudi commit completion and document what a caller can reliably recover or determine. **5. Security and multi-tenancy** A shared endpoint may execute operations using a common service identity with access to the underlying object storage and catalog. Authenticating individual clients doesn't automatically provide per-user authorization or execution isolation. If we eventually support multiple tenants, we need to understand how client identity, credentials, authorization, and potentially custom code execution interact with the shared Spark driver. For v1, restricting the endpoint to trusted internal clients may be perfectly reasonable. The important part is defining that boundary explicitly so we don't inadvertently assume stronger isolation. **6. Resource governance and observability** Even if executors are shared, we need reasonable controls around concurrent execution. Potential considerations include active session limits, operation concurrency, query timeouts, cancellation, result-size limits, and fairness between short queries and large workloads. We should also be able to correlate an incoming request with the Spark job/stages it triggers and, for writes, the corresponding Hudi transaction. A mixed-workload test with concurrent short queries and larger writes could help expose driver pressure, starvation, cancellation issues, and resource contention. **7. Extensibility and compatibility** With Spark Connect, dependencies and execution largely live on the server side. That raises questions when different clients need custom UDFs, DataSources, or conflicting library versions. There is also the upgrade problem: Spark, Hudi, Spark Connect clients, and ADBC drivers may evolve at different cadences. We should decide what compatibility guarantees we intend to provide, and whether custom dependencies require separate runtime environments. A small compatibility matrix and a PoC with custom UDFs/DataSources would help establish the boundaries. **8. Streaming and long-running workloads** Structured Streaming applications and large backfills have different lifecycle requirements from interactive SQL. They may need independent checkpoints, resource allocations, restart policies, and execution that survives client disconnections. Even though Spark Connect supports streaming APIs, that doesn't necessarily make a shared interactive endpoint the right long-term execution environment for production streaming pipelines. It would be useful to clarify whether these workloads are explicitly out of scope, eventually supported within the same service, or expected to use a separate job-submission mechanism. **9. Catalog and metadata consistency** Since we expect the lakehouse to support external Spark/Flink writers, long-lived Spark Connect sessions must interact correctly with metadata and schema changes made outside the endpoint. We should validate scenarios such as an external writer adding commits, schema evolution, table recreation, and metadata refresh across sessions. The goal is to identify whether explicit refresh or cache-invalidation mechanisms are required and document the expected behavior. **10. Hudi writer configuration** I assume the endpoint exposes standard Spark/Hudi write functionality, with each caller responsible for supplying the appropriate writer properties. For instance, if a caller intends to use OCC, configuring the required concurrency and lock-provider settings would remain that caller's responsibility. Incorrect configurations would be a caller-side issue rather than something the endpoint is expected to detect or prevent. Just wanted to confirm that this is the expected model, rather than introducing any centralized writer-configuration enforcement in the Spark endpoint. ### Suggested near-term validations Rather than expanding v1 scope significantly, I think a handful of focused experiments would give us confidence in the foundational choices: 1. **Cluster-backed execution:** Multiple concurrent SQL/DataFrame workloads, resource contention, scheduling, and cancellation. 2. **Failure injection:** Executor/driver failures during concurrent operations, including Hudi writes and ambiguous commit outcomes. 3. **Isolation boundaries:** Validate the behavior of concurrent sessions, shared resources, and the intended client trust model. 4. **Extensibility:** Custom UDFs/DataSources, dependency compatibility, and runtime upgrades. 5. **Metadata freshness:** Long-lived sessions observing table changes from external writers. These don't all need to be completed before v1. The idea is to agree on which assumptions carry the greatest long-term risk and validate those early. **Finally, none of the above is intended to block the current PR or my review.** I'll continue reviewing the patches Shuo has put together in parallel while we align on the broader requirements and long-term direction. We can treat any additional work as follow-ups where appropriate. GitHub link: https://github.com/apache/hudi/discussions/19843#discussioncomment-19104010 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
