[
https://issues.apache.org/jira/browse/IMPALA-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699431#comment-17699431
]
Quanlong Huang commented on IMPALA-9332:
----------------------------------------
The recursive listing in HdfsTable.getAllPartitionsNotInHms() might be a good
candidate as well.
https://github.com/apache/impala/blob/39fea06f2bca44cc2148e4fda3f72be2dfd6d280/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L2499-L2516
> Investigate and use the new batch listing API from HDFS-13616
> -------------------------------------------------------------
>
> Key: IMPALA-9332
> URL: https://issues.apache.org/jira/browse/IMPALA-9332
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Vihang Karajgaonkar
> Assignee: Quanlong Huang
> Priority: Critical
>
> HDFS-13616 provides a new batch listing API which can potentially speed up
> the file listing on HDFS tables when reloading the table file metadata. We
> should investigate if this API is helpful for Impala and use it if there are
> any performance benefits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
