steveloughran commented on a change in pull request #1359:
HADOOP-16430.S3AFilesystem.delete to incrementally update s3guard with deletions
URL: https://github.com/apache/hadoop/pull/1359#discussion_r321248863
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java
##########
@@ -207,7 +211,7 @@ public DeleteOperation(final StoreContext context,
"page size out of range: %d", pageSize);
this.pageSize = pageSize;
metadataStore = context.getMetadataStore();
- executor = context.createThrottledExecutor(2);
+ executor = context.createThrottledExecutor(1);
Review comment:
yeah, for now. It means that the delete and list can go in parallel, without
having to deal with the complexity of multiple parallel deletes and failures
within them. It's the error handling which scared me. And with the same pool of
connections to Dynamo, you wouldn't automatically get a speed up. As the
Javadoc says: do more experimentation here -but do it on EC2, so that the
answers are valid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
