pvary commented on pull request #1559: URL: https://github.com/apache/iceberg/pull/1559#issuecomment-704798105
> That said, delete -> rename can be (should be?) replaced with rename with overwrite = true. If I'm not missing anything, this ensures last one wins and partial file is not exposed. I think we could not use rename with overwrite since it behaves differently on different [FileSystems](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html#boolean_rename.28Path_src.2C_Path_d.29): ``` Destination exists and is a file Renaming a file atop an existing file is specified as failing, raising an exception. - Local FileSystem : the rename succeeds; the destination file is replaced by the source file. - HDFS : The rename fails, no exception is raised. Instead the method call simply returns false. ``` That is why we decided to use delete + rename which works consistently for every FS ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
