ben-roling commented on issue #794: HADOOP-16085: use object version or etags to protect against inconsistent read after replace/overwrite URL: https://github.com/apache/hadoop/pull/794#issuecomment-491144178 Some weird stuff is going on in `testRenameEventuallyConsistentDirectory` contributing to the test failure (investigating etag-server mode). First, listFilesAndEmptyDirectories() returns S3ALocatedFileStatus that doesn't have the etag. When combined with the stubbing to return the old eTag, this causes ChangeTracker to get seeded with the old eTag and as such the first several retry passes are going past changeTracker.processMetadata() because the old eTag actually looks to ChangeTracker like it is correct. Since the code gets past that, it gets to transfers.copy(). That fails with NullPointerException. That seems like a bug in the AWS SDK. The NPE happens because com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk() calls s3.copyObject(copyObjectRequest), which returns null when the ETag constraint isn't met. The method goes on assuming that would have never returned null, resulting in the NPE. The NPE causes a trip back through the retry loop. The same thing happens again until the inconsistent metadata stub count is exceeded and then any retry after that causes the ChangeTracker to trigger RemoteFileChangedException when the correct eTag passes through since it thinks the old eTag is actually the right one. The test ultimately fails with that RemoteFileChangedException when the overall retry limit is met. I need to look closer at why listFilesAndEmptyDirectories() doesn't return the status containing the etag.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
