[
https://issues.apache.org/jira/browse/SPARK-61583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-61583:
------------------------------
Description:
h3. Pull request
[apache/spark#60417|https://github.com/apache/spark/pull/60417]
h3. Motivation
Each setup insert incurs SQL planning and execution overhead. The affected
fixtures exercise
runtime partition filtering rather than repeated append behavior, so batching
reduces setup from
89 write jobs and append commits to 18.
h3. Coverage traceability
* All 20 tests remain.
* Every fixture retains the same rows and partition values.
* All DPP, scalar-subquery, predicate-pushdown, partition, result, and
negative-path assertions
remain unchanged.
* The intentional coverage difference is 89 append operations becoming 18.
h3. Benchmark
Two local runs of the full suite on each commit produced:
|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{b33faf5d24b}}) | 12.141 s | 12.416 s | 12.279 s |
| PR ({{de26c22fd5a}}) | 11.136 s | 11.062 s | 11.099 s |
This saves 1.180 seconds, or 9.6%, on average.
> Batch setup inserts in enhanced runtime partition filter tests
> --------------------------------------------------------------
>
> Key: SPARK-61583
> URL: https://issues.apache.org/jira/browse/SPARK-61583
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60417|https://github.com/apache/spark/pull/60417]
> h3. Motivation
> Each setup insert incurs SQL planning and execution overhead. The affected
> fixtures exercise
> runtime partition filtering rather than repeated append behavior, so batching
> reduces setup from
> 89 write jobs and append commits to 18.
> h3. Coverage traceability
> * All 20 tests remain.
> * Every fixture retains the same rows and partition values.
> * All DPP, scalar-subquery, predicate-pushdown, partition, result, and
> negative-path assertions
> remain unchanged.
> * The intentional coverage difference is 89 append operations becoming 18.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{b33faf5d24b}}) | 12.141 s | 12.416 s | 12.279 s |
> | PR ({{de26c22fd5a}}) | 11.136 s | 11.062 s | 11.099 s |
> This saves 1.180 seconds, or 9.6%, on average.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]