+1, the performance results look promising

Kent

________________________________
From: cheng xu via dev <[email protected]>
Sent: Thursday, November 12, 2026 9:34 AM
To: [email protected] <[email protected]>
Cc: cheng xu <[email protected]>
Subject: [DISCUSS] SPIP: Extend Real-Time Mode to Distributed Batch SQL and 
DataFrame Queries

Hi Spark community,

I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM) execution 
infrastructure to eligible distributed batch SQL and DataFrame queries.

Agentic applications increasingly use SQL for retrieval, feature generation, 
ranking, and analytical tool calls. For these interactive workloads, 
materialized shuffle boundaries can add latency by requiring downstream stages 
to wait for upstream shuffle output. The proposal allows downstream stages to 
consume intermediate data as it arrives, overlapping producer computation, 
cross-executor transfer, and consumer computation while avoiding unnecessary 
shuffle materialization.

SPIP document: 
https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le

JIRA: https://issues.apache.org/jira/browse/SPARK-61454

The proposed approach builds on RTM’s existing connected-stage execution and 
distributed transport, with three main additions:

  *   Planner selection of eligible batch shuffle boundaries, with explicit 
opt-in and materialized execution retained for unsupported plans and exchanges.
  *   Consumer-first admission with resident reader capacity, bounded producer 
concurrency, and aggregate buffer budgets to preserve progress under bounded 
resources.
  *   Batch-specific lifecycle handling to isolate execution attempts and 
release transport and scheduler state on completion, cancellation, or failure.

The initial scope targets shorter, read-only queries for which restarting the 
whole query after a transient failure is acceptable. Materialized shuffle 
remains the default. AQE integration and automatic group retry are follow-up 
work, not part of the initial milestone.

Early prototype results include a 1.64x speedup on distributed TPC-H 
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These 
are workload-specific results, not a general performance claim. The broader 
goal is to reduce p95 agent-step latency and increase completed steps per 
minute on a fixed cluster; concurrent-session validation is part of the 
proposed qualification work.

I’d particularly appreciate feedback on the initial scope, the 
reader-reservation and producer-admission model, memory bounds and failure 
handling, and the proposed qualification criteria. I’d also welcome a PMC 
member interested in shepherding this SPIP.

Please share feedback on the JIRA or in this thread.

Reply via email to