[
https://issues.apache.org/jira/browse/HADOOP-11572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014765#comment-16014765
]
Aaron Fabbri commented on HADOOP-11572:
---------------------------------------
{code}
@@ -688,12 +693,32 @@ public boolean rename(Path src, Path dst) throws
IOException {
*/
private void removeKeys(List<DeleteObjectsRequest.KeyVersion> keysToDelete,
boolean clearKeys) {
+ List<DeleteObjectsRequest.KeyVersion> keysToDeleteSingly;
if (enableMultiObjectsDelete) {
DeleteObjectsRequest deleteRequest
= new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
s3.deleteObjects(deleteRequest);
statistics.incrementWriteOps(1);
+ try {
+ s3.deleteObjects(deleteRequest);
+ keysToDeleteSingly = null;
{code}
It looks like you need to remove the call to {{s3.deleteObjects(..)}} before
your try block, and maybe move the write ops increment down?
{code}
+ } catch (MultiObjectDeleteException e) {
+ // partial delete of files
+ List<MultiObjectDeleteException.DeleteError> errors = e.getErrors();
+ LOG.warn("Partial failure of delete, {} objects remaining",
+ errors.size());
+ keysToDeleteSingly = new ArrayList<>(errors.size());
+ for (MultiObjectDeleteException.DeleteError error : errors) {
+ keysToDeleteSingly.add(
+ new DeleteObjectsRequest.KeyVersion(error.getKey(),
+ error.getVersionId()));
+ }
+ }
} else {
+ keysToDeleteSingly = keysToDelete;
+ }
+
+ if (keysToDeleteSingly != null && !keysToDeleteSingly.isEmpty()) {
int writeops = 0;
for (DeleteObjectsRequest.KeyVersion keyVersion : keysToDelete) {
{code}
Should this loop be using {{keysToDeleteSingly}} instead of {{keysToDelete}}?
> s3a delete() operation fails during a concurrent delete of child entries
> ------------------------------------------------------------------------
>
> Key: HADOOP-11572
> URL: https://issues.apache.org/jira/browse/HADOOP-11572
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-11572-001.patch, HADOOP-11572-branch-2-002.patch,
> HADOOP-11572-branch-2-003.patch
>
>
> Reviewing the code, s3a has the problem raised in HADOOP-6688: deletion of a
> child entry during a recursive directory delete is propagated as an
> exception, rather than ignored as a detail which idempotent operations should
> just ignore.
> the exception should be caught and, if a file not found problem, logged
> rather than propagated
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
