ben-roling commented on issue #794: HADOOP-16085: use object version or etags to protect against inconsistent read after replace/overwrite URL: https://github.com/apache/hadoop/pull/794#issuecomment-491856023 > The issue seems to suggest that a 412 response would never actually happen, which implies our simulation of this type of failure is invalid. I need to digest it a bit further. After discovering this I was wondering how I never realized it sooner. For example, why did ITestS3ARemoteFileChanged.testRenameChangedFile() not surface it? The answer lies in the fact that S3AFileSystem.copyFile() has ChangeTracker test the ObjectMetadata before it calls transfers.copy(). Because of this, the tests generate RemoteFileChangedException there instead of getting to the NPE induced AWS exception. As such, to see the NPE AWS exception, the caller would have to hit a scenario where getObjectMetadata() is showing the correct object version but then transfers.copy() does not. This seems like it should be a pretty rare scenario, but that's not to say it couldn't happen. I'm going to go ahead and change the stubbing in ITestS3ARemoteFileChanged where I had copyObject throwing the 412 exception to return null instead like the real impl would. I'll make sure there is a test that expects the behavior that comes along with that (which won't be RemoteFileChangedException since I don't think we want to write code that assumes an NPE means that), with comments linking back to the AWS SDK issue. Additionally, I will add something to the troubleshooting doc to show a trace for this kind of problem.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
