steveloughran commented on code in PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#discussion_r1549880873
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##########
@@ -124,7 +132,32 @@ public Boolean execute() throws IOException {
return true;
}
- // Walk path to root, ensuring closest ancestor is a directory, not file
+ // if performance creation mode is set, no need to check
Review Comment:
i was wrong; now the patch is in I see where I was mistaken. Its the
versioned buckets where problems surface. sorry!
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MkdirOperation.java:
##########
@@ -98,14 +107,13 @@ public Boolean execute() throws IOException {
}
// get the file status of the path.
- // this is done even for a magic path, to avoid always issuing PUT
- // requests. Doing that without a check wouild seem to be an
- // optimization, but it is not because
- // 1. PUT is slower than HEAD
- // 2. Write capacity is less than read capacity on a shard
- // 3. It adds needless entries in versioned buckets, slowing
- // down subsequent operations.
- FileStatus fileStatus = getPathStatusExpectingDir(dir);
+ // this is not done for magic path i.e. performanceCreation mode.
+ // For performanceCreation mode, we would probe for HEAD only.
+ // For non-performance or regular mode, the probe for both HEAD and LIST
would
+ // be done.
+ S3AFileStatus fileStatus = performanceCreation
+ ? probePathStatusOrNull(dir, StatusProbeEnum.HEAD_ONLY)
Review Comment:
this will trigger a needless PUT if there isn't a marker, just children,
which hits all the problems in the comments of the existing code.
afrid we need to revert to the old code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
