[
https://issues.apache.org/jira/browse/HIVE-23014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057945#comment-17057945
]
Peter Toth commented on HIVE-23014:
-----------------------------------
I've attached the output of the benchmark runs with the 2 Hive profiles. Based
on those only "Single Column Scans From Wide" benchmark is affected.
The 2 profiles are here:
[https://github.com/apache/spark/blob/master/pom.xml#L3029-L3048] vs
[https://github.com/apache/spark/blob/master/pom.xml#L3050-L3053]. They do some
other changes as well, but I measured this particular call on Hive's ORC
reader:
[https://github.com/apache/spark/blob/master/sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java#L85]
and the performance gap clearly comes from here.
> ORC reading performance
> -----------------------
>
> Key: HIVE-23014
> URL: https://issues.apache.org/jira/browse/HIVE-23014
> Project: Hive
> Issue Type: Bug
> Components: ORC
> Affects Versions: 2.3.6
> Reporter: Peter Toth
> Priority: Major
> Attachments: OrcReadBenchmark-results.txt.hive-1.2.1,
> OrcReadBenchmark-results.txt.hive-2.3.6
>
>
> Spark 3 adds support for using Hive 2.3.6 besides the old Hive 1.2.1 version.
> Some of the ORC reading benchmark shows that there is a huge performance
> difference in ORC reading between the 2 versions. I measured that
> {{org.apache.hadoop.hive.ql.io.orc.ReaderImpl}} in hive-exec-2.3.6-core.jar
> is ~3-5 times slower than in hive-exec-1.2.1.spark2.jar.
> I'm not sure if more recent Hive versions still suffer from this performance
> regression.
> Please see some details here: SPARK-30565
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
