viirya commented on code in PR #6940:
URL: https://github.com/apache/datafusion-comet/pull/6940#discussion_r4316710320
##########
spark/src/main/scala/org/apache/comet/serde/operator/NativeWriteUtils.scala:
##########
@@ -187,6 +189,117 @@ object NativeWriteUtils {
"spark.comet.write.parquet.enabled=false to write this table with
Spark.")
}
+ /**
+ * Whether the native writer creates the file at `filePath` itself, which it
does for local and
+ * HDFS files. Any other file is created through Hadoop's FileSystem API and
the native writer
+ * only writes its bytes, see [[org.apache.comet.parquet.HadoopFileOutput]].
That is always
+ * correct, whatever the file system and whatever the committer expects of
it, which is why this
+ * is decided per task on the path the commit protocol actually returned
rather than at
+ * planning.
+ */
+ def nativeWriterCreatesFile(filePath: String): Boolean =
+ filePath.startsWith("file:") || filePath.startsWith("hdfs:")
+
+ /**
+ * A fallback reason when the native writer cannot write to `outputPath`'s
file system, or
+ * `None` when it can. Local and HDFS files are created natively, and so are
the files of a
Review Comment:
Small wording slip from the reshuffle: this now says the magic committer's
files are "created natively ... through S3A". The point is the opposite,
they're created by S3A and only written natively. Maybe "Local and HDFS files
are created natively. The files of a write that S3A's magic committer commits
are created through S3A and written natively (see `HadoopFileOutput`)."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
