dongjoon-hyun commented on a change in pull request #35188:
URL: https://github.com/apache/spark/pull/35188#discussion_r811459934
##########
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##########
@@ -175,10 +175,19 @@ abstract class FileCommitProtocol extends Logging {
/**
* Specifies that a file should be deleted with the commit of this job. The
default
- * implementation deletes the file immediately.
+ * implementation deletes the file immediately or moves file to trash based
on whether
+ * the trash feature is enabled.
+ *
+ * See
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
+ * for the relevant trash configuration
*/
def deleteWithJob(fs: FileSystem, path: Path, recursive: Boolean): Boolean =
{
- fs.delete(path, recursive)
+ if (fs.getConf.getInt("fs.trash.interval", 0) > 0 &&
+ Trash.moveToAppropriateTrash(fs, path, fs.getConf)) {
Review comment:
Yes, the trash feature was merged and reverted historically from Apache
Spark repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
