zsxwing commented on issue #25333: [SPARK-28597][SS] Add config to retry spark streaming's meta log when it met error URL: https://github.com/apache/spark/pull/25333#issuecomment-553553368 `HDFSMetaDataLog.writeBatchToFile` is not an idempotent operation. For example, the first write may be successful, but the client doesn't receive the successful message. Then the second one will fail because of `FileAlreadyExistsException`. The will lead very confusing error because our error message indicates there is a different query running somewhere else. IMO, since the issue has been fixed by HDFS. You can upgrade HDFS to fix it. We don't need to add a workaround for an already fixed bug in HDFS.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
