[
https://issues.apache.org/jira/browse/SPARK-61454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143298#comment-18143298
]
Ferdinand Xu commented on SPARK-61454:
--------------------------------------
[~kabhwan] thanks for you suggestion. I just duplicated it in
[https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le]
via my personal account.
> SPIP: Extend Real-Time Mode to Distributed Batch SQL and DataFrame Queries
> --------------------------------------------------------------------------
>
> Key: SPARK-61454
> URL: https://issues.apache.org/jira/browse/SPARK-61454
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.0.5
> Reporter: Ferdinand Xu
> Priority: Major
>
> {\{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution
> infrastructure to eligible distributed batch SQL and DataFrame queries. }}
> {{The goal is to reduce query latency and improve completed-query throughput
> on a fixed cluster for interactive and agentic workloads, such as retrieval,
> feature generation, ranking, and analytical tool calls, while preserving
> exact Spark SQL results and bounded resource use.}}
> {{The proposed execution path allows downstream stages to consume
> intermediate data as it arrives across eligible shuffle boundaries,
> overlapping producer computation, cross-executor transfer, and consumer
> computation while avoiding unnecessary shuffle materialization.}}
> {{The proposal builds on RTM's existing connected-stage execution and
> distributed transport, adding batch-specific plan selection, consumer-first
> bounded admission, aggregate buffer accounting, and execution-attempt
> lifecycle management.}}
> {{This is an opt-in execution path, disabled by default. Materialized shuffle
> remains the default, and unsupported plans and exchanges retain ordinary
> batch execution. The initial scope targets shorter, read-only queries for
> which restarting the whole query after a transient failure is acceptable; AQE
> integration and automatic group retry are follow-up work.}}
> {{Early prototype results include a 1.64x speedup on distributed TPC-H
> scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These
> results are workload-specific; fixed-cluster concurrent-session latency and
> throughput validation remain part of the proposed qualification work.}}
> {{SPIP doc:
> [https://docs.google.com/document/d/1sRHXxDeeO-bLazZpIuyH60I_DDtMn7iP9TsaNEHlf5w/edit?tab=t.0#heading=h.bbeudbgc60le]
> }}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]