HyukjinKwon commented on a change in pull request #33012:
URL: https://github.com/apache/spark/pull/33012#discussion_r659441743
##########
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -92,6 +92,35 @@ abstract class FileCommitProtocol extends Logging {
*/
def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String],
ext: String): String
+ /**
+ * Notifies the commit protocol to add a new file, and gets back the full
path that should be
+ * used. Must be called on the executors when running tasks.
+ *
+ * Note that the returned temp file may have an arbitrary path. The commit
protocol only
+ * promises that the file will be at the location specified by the arguments
after job commit.
+ *
+ * The "dir" parameter specifies the sub-directory within the base path,
used to specify
+ * partitioning. The "spec" parameter specifies the file name. The rest are
left to the commit
+ * protocol implementation to decide.
+ *
+ * Important: it is the caller's responsibility to add uniquely identifying
content to "spec"
+ * if a task is going to write out multiple files to the same dir. The file
commit protocol only
+ * guarantees that files written by different tasks will not conflict.
+ *
+ * This API should be implemented and called, instead of
+ * [[newTaskTempFile(taskContest, dir, ext)]]. Provide a default
implementation here to be
+ * backward compatible with custom [[FileCommitProtocol]] implementations
before Spark 3.2.0.
Review comment:
It's a bit weird to say about "backward compatible" here. If this isn't
an API, we should explicitly mention that this isn't an API at least here to
avoid giving a false impression to dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
