[
https://issues.apache.org/jira/browse/IMPALA-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047976#comment-17047976
]
Fang-Yu Rao commented on IMPALA-9363:
-------------------------------------
We have benchmarked the performance of a patch that calls
{{getTableObjectsByName()}}
([https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L2111])
to load the table metadata. This patch incurs a prohibitively expensive cost.
Specifically, for a database consisting of 40,000 tables, it takes around 170
seconds to load the table metadata. The detailed results are provided in
[^Benchmark_related_to_IMPALA-9363.pdf].
> Add support to skip given table storage formats
> -----------------------------------------------
>
> Key: IMPALA-9363
> URL: https://issues.apache.org/jira/browse/IMPALA-9363
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Vihang Karajgaonkar
> Assignee: Fang-Yu Rao
> Priority: Critical
> Attachments: Benchmark_related_to_IMPALA-9363.pdf
>
>
> Catalog by default pulls in all the tables from HMS. Impala however, only can
> work with some table formats. Such tables are shown in the table listing and
> when they are queries, Impala also loads them. It is only during querying
> that we throw an error saying its an unsupported table format. It would be
> good to have a config which can take in list of storage handlers and skip
> loading such tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
