[
https://issues.apache.org/jira/browse/SPARK-29171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-29171:
-------------------------------
Description:
The updated benchmark results show performance regression *more than 4 times*
in Parquet MR on scanning 100 columns, see
[https://github.com/apache/spark/pull/25828/files#diff-e37c3433287c7371ff9c2404db821095R248]
. Need to find out the root cause of this and fix it.
Old results:
{code}
Single Column Scan from 100 columns: Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
SQL Parquet MR 275 / 286 3.8
262.3 49.0X
{code}
New results:
{code}
Single Column Scan from 100 columns: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
SQL Parquet MR 1209 1213
5 0.9 1153.1 12.2X
{code}
was:The updated benchmark results show performance regression *more than 4
times* in Parquet MR on scanning 100 columns, see
https://github.com/apache/spark/pull/25828/files#diff-e37c3433287c7371ff9c2404db821095R248
. Need to find out the root cause of this and fix it.
> Performance regression in Parquet MR on scans of 100 columns
> ------------------------------------------------------------
>
> Key: SPARK-29171
> URL: https://issues.apache.org/jira/browse/SPARK-29171
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Major
>
> The updated benchmark results show performance regression *more than 4 times*
> in Parquet MR on scanning 100 columns, see
> [https://github.com/apache/spark/pull/25828/files#diff-e37c3433287c7371ff9c2404db821095R248]
> . Need to find out the root cause of this and fix it.
> Old results:
> {code}
> Single Column Scan from 100 columns: Best/Avg Time(ms) Rate(M/s) Per
> Row(ns) Relative
> SQL Parquet MR 275 / 286 3.8
> 262.3 49.0X
> {code}
> New results:
> {code}
> Single Column Scan from 100 columns: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> SQL Parquet MR 1209 1213
> 5 0.9 1153.1 12.2X
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
