steveloughran opened a new pull request #1407: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1407 This patch avoids issuing any HEAD path request when creating a file with overwrite=true, so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists in their own code Special S3Guard FNFE retry policy independently configurable from other retry policies, * and use a setting with exponential backoff * new config names * copy raises a RemoteFileChangedException which is *not* caught in rename() and downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated * tests for this * More logging @ debug in change policies as to policy type and when options are not set, as well as being set. Currently to work out the policy involves looking for the absence of messages, not the presence. It makes the file more verbose, but will aid with debugging these problems. Also: tests turning auth mode on/off have to handle the auth state being set through an authoritative path, rather than a single flag. Caught me out as of course the first test I saw with this was the ITestS3ARemoteFileChanged rename ones, and I assumed that it was my new code. It was actually due to me setting an auth path last week. Change-Id: I7bb468aca0f4019537d82bc083f0a9887eaa282b
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
