SourabhBadhya commented on code in PR #4520:
URL: https://github.com/apache/hive/pull/4520#discussion_r1304518717
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java:
##########
@@ -229,11 +249,20 @@ public List<Task<?>> getTasks(HiveConf conf, Object
objCtx) {
private void generateActualTasks(HiveConf conf, List<Task<?>> resTsks,
long trgtSize, long avgConditionSize, Task<?> mvTask,
Task<?> mrTask, Task<?> mrAndMvTask, Path dirPath,
- FileSystem inpFs, ConditionalResolverMergeFilesCtx ctx, MapWork work,
int dpLbLevel)
+ FileSystem inpFs, ConditionalResolverMergeFilesCtx ctx, MapWork work,
int dpLbLevel,
+ boolean manifestFilePresent)
throws IOException {
DynamicPartitionCtx dpCtx = ctx.getDPCtx();
- // get list of dynamic partitions
- List<FileStatus> statusList = HiveStatsUtils.getFileStatusRecurse(dirPath,
dpLbLevel, inpFs);
+ List<FileStatus> statusList;
+ if (manifestFilePresent) {
+ // Get the list of files from manifest file.
+ List<FileStatus> fileStatuses = getManifestFilePaths(conf, dirPath);
+ // Setup the work to include all the files present in the manifest.
+ setupWorkWhenUsingManifestFile(work, fileStatuses, dirPath);
+ statusList =
HiveStatsUtils.getFileStatusRecurse(Utilities.toTempPath(dirPath), dpLbLevel,
inpFs);
Review Comment:
Removed this as well, and identified directories using the manifest file
itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
