[
https://issues.apache.org/jira/browse/SPARK-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or resolved SPARK-18507.
-------------------------------
Resolution: Fixed
Assignee: Wenchen Fan
Fix Version/s: 2.1.0
Target Version/s: 2.1.0
> Major performance regression in SHOW PARTITIONS on partitioned Hive tables
> --------------------------------------------------------------------------
>
> Key: SPARK-18507
> URL: https://issues.apache.org/jira/browse/SPARK-18507
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Michael Allman
> Assignee: Wenchen Fan
> Priority: Critical
> Fix For: 2.1.0
>
>
> Commit {{ccb11543048dccd4cc590a8db1df1d9d5847d112}}
> (https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112)
> appears to have introduced a major regression in the performance of the Hive
> {{SHOW PARTITIONS}} command. Running that command on a Hive table with 17,337
> partitions in the {{spark-sql}} shell with the parent commit of {{ccb1154}}
> takes approximately 7.3 seconds. Running the same command with commit
> {{ccb1154}} takes approximately 250 seconds.
> I have not had the opportunity to complete a thorough investigation, but I
> suspect the problem lies in the diff hunk beginning at
> https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112#diff-159191585e10542f013cb3a714f26075L675.
> If that's the case, this performance issue should manifest itself in other
> areas as this programming pattern was used elsewhere in this commit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
