[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349461#comment-16349461 ]
Xiao Li commented on SPARK-23304: --------------------------------- That is fine. Obviously, at least, we need to submit a PR to document the behavior changes introduced by the native ORC reader. This is missing. I am just trying to confirm no behavior change we made in this release. How did you get the number partitions? Is it through something like? {noformat} sql("xyz").rdd.partitions.length {noformat} > Spark SQL coalesce() against hive not working > --------------------------------------------- > > Key: SPARK-23304 > URL: https://issues.apache.org/jira/browse/SPARK-23304 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Thomas Graves > Assignee: Xiao Li > Priority: Major > Attachments: spark22_oldorc_explain.txt, spark23_oldorc_explain.txt > > > The query below seems to ignore the coalesce. This is running spark 2.2 or > spark 2.3 against hive, which is reading orc: > > Query: > spark.sql("SELECT COUNT(DISTINCT(something)) FROM sometable WHERE dt >= > '20170301' AND dt <= '20170331' AND something IS NOT > NULL").coalesce(160000).show() > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org