beliefer edited a comment on issue #23841: [SPARK-26936][SQL] Fix bug of insert overwrite local dir and inconsistent behavior with Hive URL: https://github.com/apache/spark/pull/23841#issuecomment-468123350 I have check the source of Hadoop `LocalFileSystem `.`LocalFileSystem` don't implement the method `rename`. `LocalFileSystem` extends `ChecksumFileSystem` and the latter implement the method `rename`. The method `rename` of `ChecksumFileSystem` as follows: ``` public boolean rename(Path src, Path dst) throws IOException { if (fs.isDirectory(src)) { return fs.rename(src, dst); } else { if (fs.isDirectory(dst)) { dst = new Path(dst, src.getName()); } boolean value = fs.rename(src, dst); if (!value) return false; Path srcCheckFile = getChecksumFile(src); Path dstCheckFile = getChecksumFile(dst); if (fs.exists(srcCheckFile)) { //try to rename checksum value = fs.rename(srcCheckFile, dstCheckFile); } else if (fs.exists(dstCheckFile)) { // no src checksum, so remove dst checksum value = fs.delete(dstCheckFile, true); } return value; } } ``` If target path is a directory, `ChecksumFileSystem` will move source file into target path. If target path is not a directory, `ChecksumFileSystem` will rename source file to target file. There exists a variable named `fs` that is a `RawLocalFileSystem`. `RawLocalFileSystem` will call the method `rename` of `UNIXFileSystem` or `WinNTFileSystem`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
