[
https://issues.apache.org/jira/browse/HADOOP-18568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646839#comment-17646839
]
André F. commented on HADOOP-18568:
-----------------------------------
Hi Steve, sorry I took a bit of time to set up this experiment again. For some
reason, I cannot reach the clean-up stage (the job just started taking too long
on the commit phase). I don't think it is related to
`spark.hadoop.fs.s3a.bulk.delete.page.size=1000` and so I'll investigate a bit
more in the next few days. For the moment, I can point out I'm using the
following configuration for this job:
{code:java}
spark.hadoop.fs.s3a.bucket.all.committer.magic.enabled=true
spark.hadoop.fs.s3a.connection.maximum=1000
spark.hadoop.fs.s3a.threads.max=300
spark.hadoop.fs.s3a.block.size=64
spark.hadoop.fs.s3a.committer.threads=512
spark.hadoop.fs.s3a.max.total.tasks=128
spark.hadoop.fs.s3a.fast.upload.active.blocks=24
spark.hadoop.fs.s3a.directory.marker.retention=keep
spark.hadoop.fs.s3a.bulk.delete.page.size=1000 {code}
As I understood from your comment, if we could implement an `on-the-fly`
deletion after the commit, the clean-up wouldn't be necessary, correct? In this
case, I can try to follow this path or propose both options in different PRs if
you think it is a good idea.
> Magic Committer optional clean up
> ----------------------------------
>
> Key: HADOOP-18568
> URL: https://issues.apache.org/jira/browse/HADOOP-18568
> Project: Hadoop Common
> Issue Type: Wish
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: André F.
> Priority: Minor
>
> It seems that deleting the `__magic` folder, depending on the number of
> tasks/partitions used on a given spark job, can take really long time. I'm
> having the following behavior on a given Spark job (processing ~30TB, with
> ~420k tasks) using the magic committer:
> {code:java}
> 2022-12-10T21:25:19.629Z pool-3-thread-32 INFO MagicS3GuardCommitter:
> Starting: Deleting magic directory s3a://my-bucket/random_hash/__magic
> 2022-12-10T21:52:03.250Z pool-3-thread-32 INFO MagicS3GuardCommitter:
> Deleting magic directory s3a://my-bucket/random_hash/__magic: duration
> 26:43.620s {code}
> I don't see a way out of it since the deletion of s3 objects needs to list
> all objects under a prefix and this is what may be taking too much time.
> Could we somehow make this cleanup optional? (the idea would be to delegate
> it through s3 lifecycle policies in order to not create this overhead on the
> commit phase).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
