[
https://issues.apache.org/jira/browse/HADOOP-18568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646271#comment-17646271
]
Steve Loughran commented on HADOOP-18568:
-----------------------------------------
another thought -what about if the {{loadAndCommit}} operation invoked on each
task attempt manifest to load that file and POST commit all its pending uploads
did the delete of its task attempt dir as it went along. It'd be adding 1 LIST
plus the (bulk) DELETE, so 2 write calls per file. But it would be incremental
and not that serialized/paged deep tree delete
interesting question as to what the threshold of switching to delete-in-job vs
delete-afterwards is reached. that single dir list is 1 LIST per 1000 objects
and one bulk DELETE per 250 files (configurable BTW... set it to 1000 and
there'd be less, but still 1000 write op capacity used up)j. the bulk delete is
serialized now (it can overload the store which is why we've never really tried
to go overboard there, especially as with s3guard we had to handle partial
failures too)
[[email protected]] try a job with fs.s3a.bulk.delete.page.size
set to 1000 and see how much faster it gets?
> Magic Committer optional clean up
> ----------------------------------
>
> Key: HADOOP-18568
> URL: https://issues.apache.org/jira/browse/HADOOP-18568
> Project: Hadoop Common
> Issue Type: Wish
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: André F.
> Priority: Minor
>
> It seems that deleting the `__magic` folder, depending on the number of
> tasks/partitions used on a given spark job, can take really long time. I'm
> having the following behavior on a given Spark job (processing ~30TB, with
> ~420k tasks) using the magic committer:
> {code:java}
> 2022-12-10T21:25:19.629Z pool-3-thread-32 INFO MagicS3GuardCommitter:
> Starting: Deleting magic directory s3a://my-bucket/random_hash/__magic
> 2022-12-10T21:52:03.250Z pool-3-thread-32 INFO MagicS3GuardCommitter:
> Deleting magic directory s3a://my-bucket/random_hash/__magic: duration
> 26:43.620s {code}
> I don't see a way out of it since the deletion of s3 objects needs to list
> all objects under a prefix and this is what may be taking too much time.
> Could we somehow make this cleanup optional? (the idea would be to delegate
> it through s3 lifecycle policies in order to not create this overhead on the
> commit phase).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
