[
https://issues.apache.org/jira/browse/SPARK-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595184#comment-15595184
]
Wenchen Fan edited comment on SPARK-17990 at 10/21/16 2:03 PM:
---------------------------------------------------------------
The root cause is that, hive table is not case-preserving. Although you are
writing `CREATE TABLE... PARTITIONED BY (partCol int)`, the partition column is
actually `partcol`. And then you use `DataFrameWriter` to write data files to
the table location directly, with the wrong partition column.
If we admit this defect that hive table is not case-preserving, I think it's
not a problem.
cc [~yhuai] [~rxin]
was (Author: cloud_fan):
The root cause is that, hive table is not case-preserving. Although you are
writing `CREATE TABLE... PARTITIONED BY (partCol int)`, the partition column is
actually `partcol`. And then you use `DataFrameWriter` to write data files to
the table location directly, with the wrong partition column.
If we admit this defect that hive table is not case-preserving, I think it's
not a problem.
cc @yhuai @rxin
> ALTER TABLE ... ADD PARTITION does not play nice with mixed-case partition
> column names
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-17990
> URL: https://issues.apache.org/jira/browse/SPARK-17990
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Environment: Linux
> Mac OS with a case-sensitive filesystem
> Reporter: Michael Allman
>
> Writing partition data to an external table's file location and then adding
> those as table partition metadata is a common use case. However, for tables
> with partition column names with upper case letters, the SQL command {{ALTER
> TABLE ... ADD PARTITION}} does not work, as illustrated in the following
> example:
> {code}
> scala> sql("create external table mixed_case_partitioning (a bigint)
> PARTITIONED BY (partCol bigint) STORED AS parquet LOCATION
> '/tmp/mixed_case_partitioning'")
> res0: org.apache.spark.sql.DataFrame = []
> scala> spark.sqlContext.range(10).selectExpr("id as a", "id as
> partCol").write.partitionBy("partCol").mode("overwrite").parquet("/tmp/mixed_case_partitioning")
> {code}
> At this point, doing a {{hadoop fs -ls /tmp/mixed_case_partitioning}}
> produces the following:
> {code}
> [msa@jupyter ~]$ hadoop fs -ls /tmp/mixed_case_partitioning
> Found 11 items
> -rw-r--r-- 3 msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/_SUCCESS
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=0
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=1
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=2
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=3
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=4
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=5
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=6
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=7
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=8
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=9
> {code}
> Returning to the Spark shell, we execute the following to add the partition
> metadata:
> {code}
> scala> (0 to 9).foreach { p => sql(s"alter table mixed_case_partitioning add
> partition(partCol=$p)") }
> {code}
> Examining the HDFS file listing again, we see:
> {code}
> [msa@jupyter ~]$ hadoop fs -ls /tmp/mixed_case_partitioning
> Found 21 items
> -rw-r--r-- 3 msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/_SUCCESS
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=0
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=1
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=2
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=3
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=4
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=5
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=6
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=7
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=8
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:52
> /tmp/mixed_case_partitioning/partCol=9
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=0
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=1
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=2
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=3
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=4
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=5
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=6
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=7
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=8
> drwxr-xr-x - msa supergroup 0 2016-10-18 17:53
> /tmp/mixed_case_partitioning/partcol=9
> {code}
> Note that {{msck repair table mixed_case_partitioning}} does not exhibit this
> behavior—it handles this use case correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
