ben-roling commented on a change in pull request #666: HADOOP-16221 add option to fail operation on metadata write failure URL: https://github.com/apache/hadoop/pull/666#discussion_r279558177
########## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md ########## @@ -183,6 +186,46 @@ removed on `S3AFileSystem` level. </property> ``` +#### Fail on Error + +By default, S3AFileSystem write operations will fail when updates to +S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then +updates the metadata in S3Guard. If the metadata write fails, +`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled +back. + +If the write operation cannot be programmatically retried, the S3Guard metadata +for the given file can be corrected with a command like the following: + +```bash +hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata +``` + +Programmatic retries of the original operation would require overwrite=true. +Suppose the original operation was FileSystem.create(myFile, ovewrite=false). Review comment: fixed in https://github.com/apache/hadoop/pull/666/commits/056b52a292f3d57c7a6d7eb8ac0d5de6b0cca0d5 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
