spark git commit: [SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws non-intuitive exception
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a7f66ef62 -> ef0253ff6
[SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws
non-intuitive exception
## What changes were proposed in this pull request?
This PR allows `emptyDataFrame.write` since the user didn't specify any
partition columns.
**Before**
```scala
scala> spark.emptyDataFrame.write.parquet("/tmp/t1")
org.apache.spark.sql.AnalysisException: Cannot use all columns for partition
columns;
scala> spark.emptyDataFrame.write.csv("/tmp/t1")
org.apache.spark.sql.AnalysisException: Cannot use all columns for partition
columns;
```
After this PR, there occurs no exceptions and the created directory has only
one file, `_SUCCESS`, as expected.
## How was this patch tested?
Pass the Jenkins tests including updated test cases.
Author: Dongjoon Hyun
Closes #13730 from dongjoon-hyun/SPARK-16006.
(cherry picked from commit 9b1b3ae771babf127f64898d5dc110721597a760)
Signed-off-by: Reynold Xin
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef0253ff
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ef0253ff
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ef0253ff
Branch: refs/heads/branch-2.0
Commit: ef0253ff6d7fb9bf89ef023f2d5864c70d9d792d
Parents: a7f66ef
Author: Dongjoon Hyun
Authored: Wed Jun 29 15:00:41 2016 -0700
Committer: Reynold Xin
Committed: Wed Jun 29 15:00:47 2016 -0700
--
.../spark/sql/execution/datasources/PartitioningUtils.scala | 2 +-
.../org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala| 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ef0253ff/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 388df70..c356109 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -351,7 +351,7 @@ private[sql] object PartitioningUtils {
}
}
-if (partitionColumns.size == schema.fields.size) {
+if (partitionColumns.nonEmpty && partitionColumns.size ==
schema.fields.length) {
throw new AnalysisException(s"Cannot use all columns for partition
columns")
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/ef0253ff/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
index ebbcc1d..7308f85 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
@@ -246,8 +246,9 @@ class DataFrameReaderWriterSuite extends QueryTest with
SharedSQLContext with Be
spark.range(10).write.format("parquet").mode("overwrite").partitionBy("id").save(path)
}
intercept[AnalysisException] {
-
spark.range(10).write.format("orc").mode("overwrite").partitionBy("id").save(path)
+
spark.range(10).write.format("csv").mode("overwrite").partitionBy("id").save(path)
}
+ spark.emptyDataFrame.write.format("parquet").mode("overwrite").save(path)
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws non-intuitive exception
Repository: spark
Updated Branches:
refs/heads/master 8b5a8b25b -> 9b1b3ae77
[SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws
non-intuitive exception
## What changes were proposed in this pull request?
This PR allows `emptyDataFrame.write` since the user didn't specify any
partition columns.
**Before**
```scala
scala> spark.emptyDataFrame.write.parquet("/tmp/t1")
org.apache.spark.sql.AnalysisException: Cannot use all columns for partition
columns;
scala> spark.emptyDataFrame.write.csv("/tmp/t1")
org.apache.spark.sql.AnalysisException: Cannot use all columns for partition
columns;
```
After this PR, there occurs no exceptions and the created directory has only
one file, `_SUCCESS`, as expected.
## How was this patch tested?
Pass the Jenkins tests including updated test cases.
Author: Dongjoon Hyun
Closes #13730 from dongjoon-hyun/SPARK-16006.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9b1b3ae7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9b1b3ae7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9b1b3ae7
Branch: refs/heads/master
Commit: 9b1b3ae771babf127f64898d5dc110721597a760
Parents: 8b5a8b2
Author: Dongjoon Hyun
Authored: Wed Jun 29 15:00:41 2016 -0700
Committer: Reynold Xin
Committed: Wed Jun 29 15:00:41 2016 -0700
--
.../spark/sql/execution/datasources/PartitioningUtils.scala | 2 +-
.../org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala| 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9b1b3ae7/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 388df70..c356109 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -351,7 +351,7 @@ private[sql] object PartitioningUtils {
}
}
-if (partitionColumns.size == schema.fields.size) {
+if (partitionColumns.nonEmpty && partitionColumns.size ==
schema.fields.length) {
throw new AnalysisException(s"Cannot use all columns for partition
columns")
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/9b1b3ae7/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
index 58b1d56..d454100 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
@@ -246,8 +246,9 @@ class DataFrameReaderWriterSuite extends QueryTest with
SharedSQLContext with Be
spark.range(10).write.format("parquet").mode("overwrite").partitionBy("id").save(path)
}
intercept[AnalysisException] {
-
spark.range(10).write.format("orc").mode("overwrite").partitionBy("id").save(path)
+
spark.range(10).write.format("csv").mode("overwrite").partitionBy("id").save(path)
}
+ spark.emptyDataFrame.write.format("parquet").mode("overwrite").save(path)
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
