BELUGA BEHR created HIVE-16806: ---------------------------------- Summary: Utilities isEmptyPath Loads All Files Key: HIVE-16806 URL: https://issues.apache.org/jira/browse/HIVE-16806 Project: Hive Issue Type: Improvement Affects Versions: 2.1.1, 3.0.0 Reporter: BELUGA BEHR Priority: Minor
{code:title=org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Configuration, Path)} public static boolean isEmptyPath(Configuration job, Path dirPath) throws IOException { FileSystem inpFs = dirPath.getFileSystem(job); try { FileStatus[] fStats = inpFs.listStatus(dirPath, FileUtils.HIDDEN_FILES_PATH_FILTER); if (fStats.length > 0) { return false; } } catch(FileNotFoundException fnf) { return true; } return true; } {code} You can see here that the code is loading every instance of {{FileStatus}} even though all we care about here is if there are any. I propose adding a new filter which stops collecting files into this array once it has found at least one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)