[jira] [Commented] (SPARK-23523) Incorrect result caused by the rule OptimizeMetadataOnlyQuery
[ https://issues.apache.org/jira/browse/SPARK-23523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390461#comment-16390461 ] Apache Spark commented on SPARK-23523: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/20763 > Incorrect result caused by the rule OptimizeMetadataOnlyQuery > - > > Key: SPARK-23523 > URL: https://issues.apache.org/jira/browse/SPARK-23523 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.2, 2.2.1, 2.3.0 >Reporter: Xiao Li >Assignee: Xiao Li >Priority: Major > Fix For: 2.4.0 > > > {code:scala} > val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e") > Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5") > .write.json(tablePath.getCanonicalPath) > val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", > "CoL3").distinct() > df.show() > {code} > This returns a wrong result > {{[c,e,a]}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23523) Incorrect result caused by the rule OptimizeMetadataOnlyQuery
[ https://issues.apache.org/jira/browse/SPARK-23523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385673#comment-16385673 ] guoxiaolongzte commented on SPARK-23523: What is the correct result? The description did not write the correct result. [~smilegator] > Incorrect result caused by the rule OptimizeMetadataOnlyQuery > - > > Key: SPARK-23523 > URL: https://issues.apache.org/jira/browse/SPARK-23523 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.2, 2.2.1, 2.3.0 >Reporter: Xiao Li >Assignee: Xiao Li >Priority: Major > Fix For: 2.4.0 > > > {code:scala} > val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e") > Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5") > .write.json(tablePath.getCanonicalPath) > val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", > "CoL3").distinct() > df.show() > {code} > This returns a wrong result > {{[c,e,a]}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23523) Incorrect result caused by the rule OptimizeMetadataOnlyQuery
[ https://issues.apache.org/jira/browse/SPARK-23523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380247#comment-16380247 ] Apache Spark commented on SPARK-23523: -- User 'jiangxb1987' has created a pull request for this issue: https://github.com/apache/spark/pull/20693 > Incorrect result caused by the rule OptimizeMetadataOnlyQuery > - > > Key: SPARK-23523 > URL: https://issues.apache.org/jira/browse/SPARK-23523 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.2, 2.2.1, 2.3.0 >Reporter: Xiao Li >Assignee: Xiao Li >Priority: Major > Fix For: 2.4.0 > > > {code:scala} > val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e") > Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5") > .write.json(tablePath.getCanonicalPath) > val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", > "CoL3").distinct() > df.show() > {code} > This returns a wrong result > {{[c,e,a]}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23523) Incorrect result caused by the rule OptimizeMetadataOnlyQuery
[ https://issues.apache.org/jira/browse/SPARK-23523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378060#comment-16378060 ] Apache Spark commented on SPARK-23523: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/20684 > Incorrect result caused by the rule OptimizeMetadataOnlyQuery > - > > Key: SPARK-23523 > URL: https://issues.apache.org/jira/browse/SPARK-23523 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.1, 2.3.0 >Reporter: Xiao Li >Assignee: Xiao Li >Priority: Major > > {code:scala} > val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e") > Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5") > .write.json(tablePath.getCanonicalPath) > val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", > "CoL3").distinct() > df.show() > {code} > This returns a wrong result > {{[c,e,a]}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org