[
https://issues.apache.org/jira/browse/SPARK-61578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-61578:
----------------------------------
Parent: SPARK-58540
Issue Type: Sub-task (was: Improvement)
> Avoid collecting expected DataFrames in DSv2 write tests
> --------------------------------------------------------
>
> Key: SPARK-61578
> URL: https://issues.apache.org/jira/browse/SPARK-61578
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
> Affects Versions: 4.4.0
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h3. Pull request
> [apache/spark#60407|https://github.com/apache/spark/pull/60407]
> h3. Motivation
> {{checkAnswer(actual, expectedDataFrame)}} collects the expected DataFrame
> before delegating to
> the row-based result check. In this suite that DataFrame is also the input
> already executed by
> the write command, so collecting it again duplicates execution without adding
> an assertion.
> h3. Coverage traceability
> * All 99 test cases and configuration combinations remain.
> * The write-source execution remains.
> * The written-table result comparison remains, using expected rows derived
> from the same input
> tuples.
> * The actual table's {{materializedRdd}} deserialization check performed by
> {{checkAnswer}}
> remains.
> * The only removed execution is the redundant collection of the expected
> DataFrame.
> h3. Benchmark
> Two local runs of the full suite on each commit produced:
> || Commit || Run 1 || Run 2 || Average ||
> | Parent ({{0d52f01d106}}) | 47.601 s | 48.217 s | 47.909 s |
> | PR ({{19944049a7a}}) | 44.982 s | 44.947 s | 44.965 s |
> This saves 2.945 seconds, or 6.1%, on average.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]