[ https://issues.apache.org/jira/browse/SPARK-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678215#comment-15678215 ]
Michael Allman edited comment on SPARK-18507 at 11/19/16 12:30 AM: ------------------------------------------------------------------- CC [~ekhliang] [~yhuai] [~cloud_fan] was (Author: michael): CC [~ekhliang] > Major performance regression in SHOW PARTITIONS on partitioned Hive tables > -------------------------------------------------------------------------- > > Key: SPARK-18507 > URL: https://issues.apache.org/jira/browse/SPARK-18507 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.0 > Reporter: Michael Allman > Priority: Critical > > Commit {{ccb11543048dccd4cc590a8db1df1d9d5847d112}} > (https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112) > appears to have introduced a major regression in the performance of the Hive > {{SHOW PARTITIONS}} command. Running that command on a Hive table with 17,337 > partitions in the {{spark-sql}} shell with the parent commit of {{ccb1154}} > takes approximately 7.3 seconds. Running the same command with commit > {{ccb1154}} takes approximately 250 seconds. > I have not had the opportunity to complete a thorough investigation, but I > suspect the problem lies in the diff hunk beginning at > https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112#diff-159191585e10542f013cb3a714f26075L675. > If that's the case, this performance issue should manifest itself in other > areas as this programming pattern was used elsewhere in this commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org