GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/13486
[SPARK-15743][SQL] Prevent saving with all-column partitioning
## What changes were proposed in this pull request?
When saving datasets on storage, `partitionBy` provides an easy way to
construct the directory structure. However, if a user choose all columns as
partition columns, some exceptions occurs.
- **ORC with all column partitioning**: `AnalysisException` on **future
read** due to schema inference failure.
```
scala>
spark.range(10).write.format("orc").mode("overwrite").partitionBy("id").save("/tmp/data")
scala> spark.read.format("orc").load("/tmp/data").collect()
org.apache.spark.sql.AnalysisException: Unable to infer schema for ORC at
/tmp/data. It must be specified manually;
```
- **Parquet with all-column partitioning**: `InvalidSchemaException` on
**write execution** due to Parquet limitation.
```
scala>
spark.range(100).write.format("parquet").mode("overwrite").partitionBy("id").save("/tmp/data")
[Stage 0:> (0 + 8)
/ 8]16/06/02 16:51:17
ERROR Utils: Aborting task
org.apache.parquet.schema.InvalidSchemaException: A group type can not be
empty. Parquet does not support empty group without leaves. Empty group:
spark_schema
... (lots of error messages)
```
Although some formats like JSON support all-column partitioning without any
problem, it seems not a good idea to make lots of empty directories.
This PR prevents saving with all-column partitioning by consistently
raising `AnalysisException` before saving.
## How was this patch tested?
Newly added `PartitioningUtilsSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-15743
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13486.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13486
----
commit bb97467dba96604d26d45763f4115152640ff189
Author: Dongjoon Hyun <[email protected]>
Date: 2016-06-02T23:14:50Z
[SPARK-15743][SQL] Prevent saving with all-column partitioning
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
