c21 commented on a change in pull request #33012:
URL: https://github.com/apache/spark/pull/33012#discussion_r660083272
##########
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -92,6 +92,35 @@ abstract class FileCommitProtocol extends Logging {
*/
def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String],
ext: String): String
+ /**
+ * Notifies the commit protocol to add a new file, and gets back the full
path that should be
+ * used. Must be called on the executors when running tasks.
+ *
+ * Note that the returned temp file may have an arbitrary path. The commit
protocol only
+ * promises that the file will be at the location specified by the arguments
after job commit.
+ *
+ * The "dir" parameter specifies the sub-directory within the base path,
used to specify
+ * partitioning. The "spec" parameter specifies the file name. The rest are
left to the commit
+ * protocol implementation to decide.
+ *
+ * Important: it is the caller's responsibility to add uniquely identifying
content to "spec"
+ * if a task is going to write out multiple files to the same dir. The file
commit protocol only
+ * guarantees that files written by different tasks will not conflict.
+ *
+ * This API should be implemented and called, instead of
+ * [[newTaskTempFile(taskContest, dir, ext)]]. Provide a default
implementation here to be
+ * backward compatible with custom [[FileCommitProtocol]] implementations
before Spark 3.2.0.
Review comment:
@HyukjinKwon - will it look better as below?
```
This method should be implemented and called, instead of
[[newTaskTempFile(taskContest, dir, ext)]]. Provide a default implementation
here to be
compatible with custom [[FileCommitProtocol]] implementations before Spark
3.2.0.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
