steveloughran edited a comment on issue #666: HADOOP-16221 add option to fail operation on metadata write failure URL: https://github.com/apache/hadoop/pull/666#issuecomment-487220946 I've just been retesting this...happy with the changes in the operation, just two things I want to make sure we are all good with 1. Do we need to wrap any existing IOExceptions raised in the `finishedWrite()` calls with their own exception. I'd going to say "yes" primarily because thats how we can guarantee that the failure won't trigger any of the retry logic used in existing operations which assume that an IOE only ever gets raised during the main operation against S3, rather than the subsequent metastore calls. 2. Do we make this a new switch or bond it to auth mode? (1) There's no way you'd ever want this to be disabled when in auth mode. (2) When not in auth mode, we are meant to be more tolerant of OOB changes in the store, and you could consider files which have a file changed but not metastore update as "just" an OOB update But in condition #2, even if we recover, there will be a period of inconsistency. Should we silently swallow this? Or raise an exception? I'm coming round to the "this will always be on unless you somehow want to disable it" viewpoint too. Because if you aren't updating the store for some reason (example: you don't have write perms to the table), well, that merits a failure -doesn't it? Accordingly, I'm going to propose * we do have the new config option * its _true_ by default. That is, unless you say otherwise, if you can't update the metastore, it's an error. saying "swallow metastore updates" is a special case people should be explicitly asking for. Returning to this patch then, I'm happy with it with some small changes: 1. we switch the default value of `FAIL_ON_METADATA_WRITE_ERROR_DEFAULT` to true 1. change the docs to match.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
