[jira] [Commented] (SPARK-14993) Inconsistent behavior of partitioning discovery
[ https://issues.apache.org/jira/browse/SPARK-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266073#comment-15266073 ] Apache Spark commented on SPARK-14993: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/12828 > Inconsistent behavior of partitioning discovery > --- > > Key: SPARK-14993 > URL: https://issues.apache.org/jira/browse/SPARK-14993 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Yin Huai >Priority: Critical > > When we load a dataset, if we set the path to {{/path/a=1}}, we will not take > a as the partitioning column. However, if we set the path to > {{/path/a=1/file.parquet}}, we take a as the partitioning column and it shows > up in the schema. We should make the behaviors of these two cases consistent > by not putting a into the schema for the second case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14993) Inconsistent behavior of partitioning discovery
[ https://issues.apache.org/jira/browse/SPARK-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265794#comment-15265794 ] Xiao Li commented on SPARK-14993: - Doing it now. Thanks! > Inconsistent behavior of partitioning discovery > --- > > Key: SPARK-14993 > URL: https://issues.apache.org/jira/browse/SPARK-14993 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Yin Huai >Priority: Critical > > When we load a dataset, if we set the path to {{/path/a=1}}, we will not take > a as the partitioning column. However, if we set the path to > {{/path/a=1/file.parquet}}, we take a as the partitioning column and it shows > up in the schema. We should make the behaviors of these two cases consistent > by not putting a into the schema for the second case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14993) Inconsistent behavior of partitioning discovery
[ https://issues.apache.org/jira/browse/SPARK-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265397#comment-15265397 ] Xiao Li commented on SPARK-14993: - Ok, if nobody starts it, I will work on this. Thanks! > Inconsistent behavior of partitioning discovery > --- > > Key: SPARK-14993 > URL: https://issues.apache.org/jira/browse/SPARK-14993 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Yin Huai >Priority: Critical > > When we load a dataset, if we set the path to {{/path/a=1}}, we will not take > a as the partitioning column. However, if we set the path to > {{/path/a=1/file.parquet}}, we take a as the partitioning column and it shows > up in the schema. We should make the behaviors of these two cases consistent > by not putting a into the schema for the second case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org