HyukjinKwon commented on a change in pull request #26671: Revert
"[SPARK-26081][SPARK-29999]"
URL: https://github.com/apache/spark/pull/26671#discussion_r350709827
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala
##########
@@ -137,14 +139,7 @@ class ManifestFileCommitProtocol(jobId: String, path:
String)
if (addedFiles.nonEmpty) {
val fs = new
Path(addedFiles.head).getFileSystem(taskContext.getConfiguration)
val statuses: Seq[SinkFileStatus] =
- addedFiles.flatMap { f =>
Review comment:
So, this if-else looked an overhead and it was added to avoid files not
being written?
FWIW, there's still an old case when files are not written:
https://github.com/apache/spark/blob/73183b3c8c2022846587f08e8dea5c387ed3b8d5/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala#L285-L297
```scala
spark.conf.set("spark.sql.orc.impl", "hive")
spark.range(10).filter(_ => false).write.orc("test.orc")
```
But I suspect it's a-okay since this behaviour will be superseded by
"native" implementation completely in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
