[ 
https://issues.apache.org/jira/browse/SPARK-61583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated SPARK-61583:
------------------------------
    Description: 
h3. Pull request

[apache/spark#60417|https://github.com/apache/spark/pull/60417]

h3. Motivation

Each setup insert incurs SQL planning and execution overhead. The affected 
fixtures exercise
runtime partition filtering rather than repeated append behavior, so batching 
reduces setup from
89 write jobs and append commits to 18.

h3. Coverage traceability

* All 20 tests remain.
* Every fixture retains the same rows and partition values.
* All DPP, scalar-subquery, predicate-pushdown, partition, result, and 
negative-path assertions
  remain unchanged.
* The intentional coverage difference is 89 append operations becoming 18.

h3. Benchmark

Two local runs of the full suite on each commit produced:

|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{b33faf5d24b}}) | 12.141 s | 12.416 s | 12.279 s |
| PR ({{de26c22fd5a}}) | 11.136 s | 11.062 s | 11.099 s |

This saves 1.180 seconds, or 9.6%, on average.


> Batch setup inserts in enhanced runtime partition filter tests
> --------------------------------------------------------------
>
>                 Key: SPARK-61583
>                 URL: https://issues.apache.org/jira/browse/SPARK-61583
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Szehon Ho
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60417|https://github.com/apache/spark/pull/60417]
> h3. Motivation
> Each setup insert incurs SQL planning and execution overhead. The affected 
> fixtures exercise
> runtime partition filtering rather than repeated append behavior, so batching 
> reduces setup from
> 89 write jobs and append commits to 18.
> h3. Coverage traceability
> * All 20 tests remain.
> * Every fixture retains the same rows and partition values.
> * All DPP, scalar-subquery, predicate-pushdown, partition, result, and 
> negative-path assertions
>   remain unchanged.
> * The intentional coverage difference is 89 append operations becoming 18.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{b33faf5d24b}}) | 12.141 s | 12.416 s | 12.279 s |
> | PR ({{de26c22fd5a}}) | 11.136 s | 11.062 s | 11.099 s |
> This saves 1.180 seconds, or 9.6%, on average.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to