+1, this builds on RTM's community-reviewed shuffle infrastructure instead of having to deal with introducing the extra complexity of adding that to Spark entirely separately.
On 2026-11-12 01:34 UTC cheng xu via dev wrote: > Hi Spark community, > > I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM) > execution infrastructure to eligible distributed batch SQL and DataFrame > queries. > > Agentic applications increasingly use SQL for retrieval, feature > generation, ranking, and analytical tool calls. For these interactive > workloads, materialized shuffle boundaries can add latency by requiring > downstream stages to wait for upstream shuffle output. The proposal allows > downstream stages to consume intermediate data as it arrives, overlapping > producer computation, cross-executor transfer, and consumer computation > while avoiding unnecessary shuffle materialization. > > SPIP document: > https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le > > > JIRA: https://issues.apache.org/jira/browse/SPARK-61454 > > The proposed approach builds on RTM’s existing connected-stage execution > and distributed transport, with three main additions: > > - Planner selection of eligible batch shuffle boundaries, with explicit > opt-in and materialized execution retained for unsupported plans and > exchanges. > - Consumer-first admission with resident reader capacity, bounded > producer concurrency, and aggregate buffer budgets to preserve progress > under bounded resources. > - Batch-specific lifecycle handling to isolate execution attempts and > release transport and scheduler state on completion, cancellation, or > failure. > > The initial scope targets shorter, read-only queries for which restarting > the whole query after a transient failure is acceptable. Materialized > shuffle remains the default. AQE integration and automatic group retry are > follow-up work, not part of the initial milestone. > > Early prototype results include a 1.64x speedup on distributed TPC-H > scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. > These are workload-specific results, not a general performance claim. The > broader goal is to reduce p95 agent-step latency and increase completed > steps per minute on a fixed cluster; concurrent-session validation is part > of the proposed qualification work. > > I’d particularly appreciate feedback on the initial scope, the > reader-reservation and producer-admission model, memory bounds and failure > handling, and the proposed qualification criteria. I’d also welcome a PMC > member interested in shepherding this SPIP. > > Please share feedback on the JIRA or in this thread. > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
