c21 commented on a change in pull request #33012:
URL: https://github.com/apache/spark/pull/33012#discussion_r656692620
##########
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -90,8 +90,38 @@ abstract class FileCommitProtocol extends Logging {
* if a task is going to write out multiple files to the same dir. The file
commit protocol only
* guarantees that files written by different tasks will not conflict.
*/
+ @deprecated("use newTaskFile(taskContext, dir, spec)", "3.2.0")
def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String],
ext: String): String
+ /**
+ * Notifies the commit protocol to add a new file, and gets back the full
path that should be
+ * used. Must be called on the executors when running tasks.
+ *
+ * Note that the returned temp file may have an arbitrary path. The commit
protocol only
+ * promises that the file will be at the location specified by the arguments
after job commit.
+ *
+ * The "dir" parameter specifies the sub-directory within the base path,
used to specify
+ * partitioning. The "spec" parameter specifies the file name. The rest are
left to the commit
+ * protocol implementation to decide.
+ *
+ * Important: it is the caller's responsibility to add uniquely identifying
content to "spec"
Review comment:
The whole sentence is:
```
it is the caller's responsibility to add uniquely identifying content to
"spec"
if a task is going to write out multiple files to the same dir.
```
I think this refers to the case when in one task, caller calls
newTaskTempFile() multiple times with same `spec`, the caller should not expect
commit protocol returning unique different file path every time. The current
implementation of `HadoopMapReduceCommitProtocol` would return same path if
`ext` being same. This is a copy-paste from original comment of
`newTaskTempFile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
