[
https://issues.apache.org/jira/browse/SPARK-52362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129369#comment-18129369
]
Aaron Gubrud commented on SPARK-52362:
--------------------------------------
I was able to reproduce this. I generated the input with the
\{{databricks/tpch-dbgen}} generator (SF1 \{{lineitem}}, 6,001,215 rows) and
confirmed a substantial CSV read -> CSV write regression in 3.5.1 relative to
3.3.1.
One correction to the original report, though is that this is not an S3
problem. I reproduced it entirely on a local filesystem with \{{--master
local[8]}} with no object storage involved. The cost is in the write path
itself. Introducing S3 actually diminishes the effect and my measured data
using local storage was even larger than the 16% stated in the original ticket.
Measurements on an 8 vCPU host (JDK 17, \{{local[8]}}, local disk), write-phase
elapsed, mean of 3 rounds:
|| build || write phase ||
| 3.3.1 | ~6,000 ms |
| 3.4.4 | ~8,100 ms (+35%) |
| 3.5.1 | ~8,250 ms (+37%) |
3.4.4 shows it as well, which points at 3.4.0 rather than anything introduced
in 3.5. The change in behavior traces back to commit \{{6ed03dceafd}}
(SPARK-41572, "Assign name to _LEGACY_ERROR_TEMP_2149"), which shipped in
3.4.0. That commit changed \{{UnivocityParser.convert}} to build the
malformed-record exception through the error-class framework and to materialise
the bad record via \{{getCurrentInput}}/\{{toString}}. Both of those now happen
per row.
Error handling is implicated here only because the generated input has a
column-count mismatch. TPC-H \{{.tbl}} output terminates every line with
\{{|}}, so Univocity yields 17 tokens, while the attached
\{{SparkPerfBenchmark.scala}} declares a 16-column schema; Spark reports this
as \{{Header length: 17, schema size: 16}}. \{{tokens.length !=
parsedSchema.length}} is therefore true for all 6M rows, and each one
constructs an exception whose message is never consumed in \{{PERMISSIVE}}
mode. Re-running 3.4.4 against the same data with the trailing delimiter
stripped, so the token count matches the schema size and the malformed branch
is never taken, drops write time from ~8,100 ms to ~3,100 ms (-62%). With that
branch avoided, no version shows the regression.
The good news is that this is already fixed upstream. SPARK-48143
("UnivocityParser is slow when parsing partially-malformed CSV in PERMISSIVE
mode", commit \{{a6632ffa16f6}}) made the exception construction lazy and
removed stack-trace collection from the control-flow path, and it shipped in
4.0.0. It was filed as an Improvement with Affects Version 4.0.0, so it was
never backported, and 3.4.x plus every 3.5.x release through 3.5.9 still
carries the regression. I have a backport for \{{branch-3.5}} built and
measured. Write phase drops to ~3,300 ms, which is actually faster than 3.3.1,
since the fix also removes the per-row stack-trace capture that 3.3.1 paid. I
plan to open a PR for this shortly.
> S3 (CSV read) → S3 (CSV write) task shows 16% performance regression in Spark
> 3.5.1 over 3.3.1 with TPC-H lineitem.tbl dataset
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-52362
> URL: https://issues.apache.org/jira/browse/SPARK-52362
> Project: Spark
> Issue Type: Bug
> Components: Input/Output, Spark Core, SQL
> Affects Versions: 3.5.1
> Environment: * OS: RHEL8
> * Hardware: Intel Xeon Gold 6258R @ 2.70GHz, 8 cores (VMware VM)
> * Spark versions tested: 3.3.1 vs 3.5.1 (VanillaSpark)
> * Hadoop: 3.2.3
> * Executor configuration: Single executor
> Reporter: Piyush Chamria
> Priority: Major
> Attachments: SparkPerfBenchmark.scala
>
>
> *Issue Summary:*
> Performance regression of ~16% observed in Spark 3.5.1 compared to 3.3.1 for
> S3 CSV read → S3 CSV write operations.
> *Test Methodology:*
> Simple pass-through task: S3 (CSV read) → S3 (CSV write)
> *Benchmark Results:*
> Dataset : TPC-H lineitem.tbl
> - Performance degradation: ~16%
> Attached is the code used for the S3 CSV read → S3 CSV write operations:
> [^SparkPerfBenchmark.scala]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
