Repository: spark Updated Branches: refs/heads/branch-2.0 a7f66ef62 -> ef0253ff6
[SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws non-intuitive exception ## What changes were proposed in this pull request? This PR allows `emptyDataFrame.write` since the user didn't specify any partition columns. **Before** ```scala scala> spark.emptyDataFrame.write.parquet("/tmp/t1") org.apache.spark.sql.AnalysisException: Cannot use all columns for partition columns; scala> spark.emptyDataFrame.write.csv("/tmp/t1") org.apache.spark.sql.AnalysisException: Cannot use all columns for partition columns; ``` After this PR, there occurs no exceptions and the created directory has only one file, `_SUCCESS`, as expected. ## How was this patch tested? Pass the Jenkins tests including updated test cases. Author: Dongjoon Hyun <dongj...@apache.org> Closes #13730 from dongjoon-hyun/SPARK-16006. (cherry picked from commit 9b1b3ae771babf127f64898d5dc110721597a760) Signed-off-by: Reynold Xin <r...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef0253ff Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ef0253ff Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ef0253ff Branch: refs/heads/branch-2.0 Commit: ef0253ff6d7fb9bf89ef023f2d5864c70d9d792d Parents: a7f66ef Author: Dongjoon Hyun <dongj...@apache.org> Authored: Wed Jun 29 15:00:41 2016 -0700 Committer: Reynold Xin <r...@databricks.com> Committed: Wed Jun 29 15:00:47 2016 -0700 ---------------------------------------------------------------------- .../spark/sql/execution/datasources/PartitioningUtils.scala | 2 +- .../org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/ef0253ff/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala index 388df70..c356109 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala @@ -351,7 +351,7 @@ private[sql] object PartitioningUtils { } } - if (partitionColumns.size == schema.fields.size) { + if (partitionColumns.nonEmpty && partitionColumns.size == schema.fields.length) { throw new AnalysisException(s"Cannot use all columns for partition columns") } } http://git-wip-us.apache.org/repos/asf/spark/blob/ef0253ff/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala index ebbcc1d..7308f85 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala @@ -246,8 +246,9 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSQLContext with Be spark.range(10).write.format("parquet").mode("overwrite").partitionBy("id").save(path) } intercept[AnalysisException] { - spark.range(10).write.format("orc").mode("overwrite").partitionBy("id").save(path) + spark.range(10).write.format("csv").mode("overwrite").partitionBy("id").save(path) } + spark.emptyDataFrame.write.format("parquet").mode("overwrite").save(path) } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org