mackrorysd commented on a change in pull request #843: HADOOP-15183 S3Guard
store becomes inconsistent after partial failure of rename
URL: https://github.com/apache/hadoop/pull/843#discussion_r293117096
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -232,8 +248,33 @@
/** Principal who created the FS; recorded during initialization. */
private UserGroupInformation owner;
- // The maximum number of entries that can be deleted in any call to s3
+ /**
+ * The maximum number of entries that can be deleted in any bulk delete
+ * call to S3 {@value}.
+ */
private static final int MAX_ENTRIES_TO_DELETE = 1000;
+
+ /**
+ * This is an arbitrary value: {@value}.
+ * It declares how many parallel copy operations
+ * in a single rename can be queued before the operation pauses
+ * and awaits completion.
+ * A very large value wouldn't just starve other threads from
+ * performing work, there's a risk that the S3 store itself would
+ * throttle operations (which all go to the same shard).
+ * It is not currently configurable just to avoid people choosing values
+ * which work on a microbenchmark (single rename, no other work, ...)
+ * but don't scale well to execution in a large process against a common
+ * store, all while separate processes are working with the same shard
+ * of storage.
+ *
+ * It should be a factor of {@link #MAX_ENTRIES_TO_DELETE} so that
+ * all copies will have finished before deletion is contemplated.
+ * (There's always a block for that, it just makes more sense to
+ * perform the bulk delete after another block of copies have completed).
+ */
+ public static final int RENAME_PARALLEL_LIMIT = 10;
Review comment:
There should be some coordination with the Hive community on this, where
there's another layer of parallelization. I think they'd currently have to be
doing individual file renames, and if so I think they can just keep going, this
change can go out, and after that they might want to remove their
parallelization layer. And they don't have to interact. But I hope I'm right
about that and this isn't going to cause some huge spike in number of
concurrent copies or something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
