+1, I’m excited to see RTM extended to more use cases. I’d still like to
understand how you plan to handle blocking batch operators that consume all
their input before producing output, and whether making those operators
incremental is part of the proposal.

On Wed, Nov 11, 2026 at 5:35 PM cheng xu via dev <[email protected]>
wrote:

> Hi Spark community,
>
> I’d like to propose an SPIP to extend Spark’s Real-Time Mode (RTM)
> execution infrastructure to eligible distributed batch SQL and DataFrame
> queries.
>
> Agentic applications increasingly use SQL for retrieval, feature
> generation, ranking, and analytical tool calls. For these interactive
> workloads, materialized shuffle boundaries can add latency by requiring
> downstream stages to wait for upstream shuffle output. The proposal allows
> downstream stages to consume intermediate data as it arrives, overlapping
> producer computation, cross-executor transfer, and consumer computation
> while avoiding unnecessary shuffle materialization.
>
> SPIP document:
> https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le
>
>
> JIRA: https://issues.apache.org/jira/browse/SPARK-61454
>
> The proposed approach builds on RTM’s existing connected-stage execution
> and distributed transport, with three main additions:
>
>    - Planner selection of eligible batch shuffle boundaries, with
>    explicit opt-in and materialized execution retained for unsupported plans
>    and exchanges.
>    - Consumer-first admission with resident reader capacity, bounded
>    producer concurrency, and aggregate buffer budgets to preserve progress
>    under bounded resources.
>    - Batch-specific lifecycle handling to isolate execution attempts and
>    release transport and scheduler state on completion, cancellation, or
>    failure.
>
> The initial scope targets shorter, read-only queries for which restarting
> the whole query after a transient failure is acceptable. Materialized
> shuffle remains the default. AQE integration and automatic group retry are
> follow-up work, not part of the initial milestone.
>
> Early prototype results include a 1.64x speedup on distributed TPC-H
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%.
> These are workload-specific results, not a general performance claim. The
> broader goal is to reduce p95 agent-step latency and increase completed
> steps per minute on a fixed cluster; concurrent-session validation is part
> of the proposed qualification work.
>
> I’d particularly appreciate feedback on the initial scope, the
> reader-reservation and producer-admission model, memory bounds and failure
> handling, and the proposed qualification criteria. I’d also welcome a PMC
> member interested in shepherding this SPIP.
>
> Please share feedback on the JIRA or in this thread.
>

Reply via email to