[
https://issues.apache.org/jira/browse/HADOOP-18679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759269#comment-17759269
]
Steve Loughran commented on HADOOP-18679:
-----------------------------------------
Done a first pass at an API in a PR: no actual time allocated to implement it
-others very welcome to!
Minimal API of (basepath, RemoteIterator<Path>) for enumerating files; caller
gets to implement the iterator of their choice.
progress report callbacks allow for the operation to be aborted.
final outcome report lists files not deleted (would that scale? I've left out
the list of deleted files for that reason), exception to raise, some numbers
and any IOStats to return.
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-18679-bulk-delete-api/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BulkDelete.java
> Add API for bulk/paged object deletion
> --------------------------------------
>
> Key: HADOOP-18679
> URL: https://issues.apache.org/jira/browse/HADOOP-18679
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.5
> Reporter: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> iceberg and hbase could benefit from being able to give a list of individual
> files to delete -files which may be scattered round the bucket for better
> read peformance.
> Add some new optional interface for an object store which allows a caller to
> submit a list of paths to files to delete, where
> the expectation is
> * if a path is a file: delete
> * if a path is a dir, outcome undefined
> For s3 that'd let us build these into DeleteRequest objects, and submit,
> without any probes first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
