[
https://issues.apache.org/jira/browse/HIVE-24774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284958#comment-17284958
]
Rajesh Balamohan commented on HIVE-24774:
-----------------------------------------
Thanks [~pvargacl], I will go through both PRs you mentioned.
> Reduce FS listing during dynamic partition loading
> --------------------------------------------------
>
> Key: HIVE-24774
> URL: https://issues.apache.org/jira/browse/HIVE-24774
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Reporter: Rajesh Balamohan
> Priority: Major
>
> When loading large number of partitions in cloud storage, notification log
> takes lot longer time to list newly added files.
> It would be good to explore if FileStatus can be reused from
> Hive::listFilesCreatedByQuery or from copyFiles
> {noformat}
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4171)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3519)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3504)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:2984)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleDynParts(MoveTask.java:562)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:440)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:730)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:490)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:484)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
