Ferdinand Xu created SPARK-61454:
------------------------------------
Summary: SPIP: Extend Real-Time Mode to Distributed Batch SQL and
DataFrame Queries
Key: SPARK-61454
URL: https://issues.apache.org/jira/browse/SPARK-61454
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.0.5
Reporter: Ferdinand Xu
{{This SPIP proposes extending Spark's Real-Time Mode (RTM) execution
infrastructure to eligible distributed batch SQL and DataFrame queries.
The goal is to reduce query latency and improve completed-query throughput on a
fixed cluster for interactive and agentic workloads, such as retrieval, feature
generation, ranking, and analytical tool calls, while preserving exact Spark
SQL results and bounded resource use.}}
{{The proposed execution path allows downstream stages to consume intermediate
data as it arrives across eligible shuffle boundaries, overlapping producer
computation, cross-executor transfer, and consumer computation while avoiding
unnecessary shuffle materialization.}}
{{The proposal builds on RTM's existing connected-stage execution and
distributed transport, adding batch-specific plan selection, consumer-first
bounded admission, aggregate buffer accounting, and execution-attempt lifecycle
management.}}
{{This is an opt-in execution path, disabled by default. Materialized shuffle
remains the default, and unsupported plans and exchanges retain ordinary batch
execution. The initial scope targets shorter, read-only queries for which
restarting the whole query after a transient failure is acceptable; AQE
integration and automatic group retry are follow-up work.}}
{{Early prototype results include a 1.64x speedup on distributed TPC-H
scale-factor 10,000 Q3, with physical shuffle writes reduced by 95.6%. These
results are workload-specific; fixed-cluster concurrent-session latency and
throughput validation remain part of the proposed qualification work. }}
{{SPIP doc:
https://docs.google.com/document/d/1nl56B2ZrSBZBnJCSpkVomnX6Tsrfb5ti7Fat7NlsCqA/edit}}
(Please request access since we can't make it public.)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]