dongjoon-hyun commented on issue #24018: [SPARK-23749][SQL] Replace built-in Hive API (isSub/toKryo) and remove OrcProto.Type usage URL: https://github.com/apache/spark/pull/24018#issuecomment-472602348 Ur, @srowen It's replaced with a `private` version as I mentioned before. > HIVE-14259 purposely replaced public isSubDir to private isSubDir For example, if you see HIVE-14259, it was like the following. ```java - isOldPathUnderDestf = FileUtils.isSubDir(oldPath, destf, fs2); + isOldPathUnderDestf = isSubDir(oldPath, destf, oldFs, destFs, false); ``` The new version on the lastest master branch looks like the following. ```java private static boolean isSubDir(Path srcf, Path destf, FileSystem srcFs, FileSystem destFs, boolean isSrcLocal) { if (srcf == null) { LOG.debug("The source path is null for isSubDir method."); return false; } String fullF1 = getQualifiedPathWithoutSchemeAndAuthority(srcf, srcFs).toString() + Path.SEPARATOR; String fullF2 = getQualifiedPathWithoutSchemeAndAuthority(destf, destFs).toString() + Path.SEPARATOR; boolean isInTest = HiveConf.getBoolVar(srcFs.getConf(), ConfVars.HIVE_IN_TEST); // In the automation, the data warehouse is the local file system based. LOG.debug("The source path is " + fullF1 + " and the destination path is " + fullF2); if (isInTest) { return fullF1.startsWith(fullF2); } // schema is diff, return false String schemaSrcf = srcf.toUri().getScheme(); String schemaDestf = destf.toUri().getScheme(); // if the schemaDestf is null, it means the destination is not in the local file system if (schemaDestf == null && isSrcLocal) { LOG.debug("The source file is in the local while the dest not."); return false; } // If both schema information are provided, they should be the same. if (schemaSrcf != null && schemaDestf != null && !schemaSrcf.equals(schemaDestf)) { LOG.debug("The source path's schema is " + schemaSrcf + " and the destination path's schema is " + schemaDestf + "."); return false; } LOG.debug("The source path is " + fullF1 + " and the destination path is " + fullF2); return fullF1.startsWith(fullF2); } ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
