[ 
https://issues.apache.org/jira/browse/SPARK-61581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated SPARK-61581:
------------------------------
    Description: 
h3. Pull request

[apache/spark#60408|https://github.com/apache/spark/pull/60408]

h3. Motivation

Three runtime-filter tests used ten separate single-row inserts only to prepare 
their fixtures.
Batching each fixture into one multi-row insert reduces setup from 30 insert 
jobs and append
commits to three.

h3. Coverage traceability

* All 116 suite tests remain.
* Each fixture retains the same rows and partition values. The in-memory table 
still groups them
  into the same ten single-row partitions in each affected test.
* All runtime-filter, result, and partition-count assertions remain unchanged.
* The intentional coverage difference is 30 append operations becoming three. 
Other tests
  continue to cover repeated appends to partitioned in-memory tables.

h3. Benchmark

Two local runs of the full suite on each commit produced:

|| Commit || Run 1 || Run 2 || Average ||
| Parent ({{0d52f01d106}}) | 17.740 s | 17.569 s | 17.655 s |
| PR ({{55000f019f3}}) | 17.547 s | 17.767 s | 17.657 s |

The 0.003-second (0.01%) difference is within run-to-run noise, so the 
reduction from 30 to three
insert jobs does not produce a measurable change in full-suite wall time.


> Batch setup inserts in DSv2 runtime filter tests
> ------------------------------------------------
>
>                 Key: SPARK-61581
>                 URL: https://issues.apache.org/jira/browse/SPARK-61581
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Szehon Ho
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Pull request
> [apache/spark#60408|https://github.com/apache/spark/pull/60408]
> h3. Motivation
> Three runtime-filter tests used ten separate single-row inserts only to 
> prepare their fixtures.
> Batching each fixture into one multi-row insert reduces setup from 30 insert 
> jobs and append
> commits to three.
> h3. Coverage traceability
> * All 116 suite tests remain.
> * Each fixture retains the same rows and partition values. The in-memory 
> table still groups them
>   into the same ten single-row partitions in each affected test.
> * All runtime-filter, result, and partition-count assertions remain unchanged.
> * The intentional coverage difference is 30 append operations becoming three. 
> Other tests
>   continue to cover repeated appends to partitioned in-memory tables.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{0d52f01d106}}) | 17.740 s | 17.569 s | 17.655 s |
> | PR ({{55000f019f3}}) | 17.547 s | 17.767 s | 17.657 s |
> The 0.003-second (0.01%) difference is within run-to-run noise, so the 
> reduction from 30 to three
> insert jobs does not produce a measurable change in full-suite wall time.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to