Github user pashazm commented on a diff in the pull request:
https://github.com/apache/spark/pull/20525#discussion_r166561430
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
---
@@ -190,9 +190,13 @@ object FileFormatWriter extends Logging {
global = false,
child = plan).execute()
}
- val ret = new Array[WriteTaskResult](rdd.partitions.length)
+
+ // SPARK-23271 If we are attempting to write a zero partition rdd,
change the number of
+ // partition to 1 to make sure we at least set up one write task to
write the metadata.
+ val finalRdd = if (rdd.partitions.length == 0) rdd.repartition(1)
else rdd
--- End diff --
Looks like shuffle will be here if partitions number is zero. If so, maybe,
other solution is possible?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
