cloud-fan commented on a change in pull request #33012:
URL: https://github.com/apache/spark/pull/33012#discussion_r656093708
##########
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -90,8 +90,38 @@ abstract class FileCommitProtocol extends Logging {
* if a task is going to write out multiple files to the same dir. The file
commit protocol only
* guarantees that files written by different tasks will not conflict.
*/
+ @deprecated("use newTaskFile(taskContext, dir, spec)", "3.2.0")
def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String],
ext: String): String
+ /**
+ * Notifies the commit protocol to add a new file, and gets back the full
path that should be
+ * used. Must be called on the executors when running tasks.
+ *
+ * Note that the returned temp file may have an arbitrary path. The commit
protocol only
+ * promises that the file will be at the location specified by the arguments
after job commit.
+ *
+ * The "dir" parameter specifies the sub-directory within the base path,
used to specify
+ * partitioning. The "spec" parameter specifies the file name. The rest are
left to the commit
+ * protocol implementation to decide.
+ *
+ * Important: it is the caller's responsibility to add uniquely identifying
content to "spec"
Review comment:
This is not true, commit protocol needs to guarantee name uniqueness, as
caller side only gives prefix and suffix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
