Hi Eugene, Hive has a feature called automatic query rewrite [1]. This feature needs up-to-date information about the materialized views available. [2] The feature can be disable by the setting: hive.materializedview.rewriting [3]
Hope this helps. regards, Krisztian [1] https://cwiki.apache.org/confluence/display/Hive/Materialized+views#Materializedviews-Materializedview-basedqueryrewriting [2] https://github.com/apache/hive/blob/af7059e2bdc8b18af42e0b7f7163b923a0bfd424/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L2091-L2094 [3] https://github.com/apache/hive/blob/af7059e2bdc8b18af42e0b7f7163b923a0bfd424/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1605C51-L1606 On Fri, Nov 17, 2023 at 2:06 AM Eugene Miretsky <eug...@badal.io> wrote: > Hey! > > Hive version is 3.1.3 > > On Wed, Nov 15, 2023 at 9:23 PM Butao Zhang <butaozha...@163.com> wrote: > >> Hi, which version of hms are you using now? I have checked the master >> branch and beta-1 branch source code, but I can't find the place where >> this method *get_materialized_views_for_rewriting* is called by mistake. >> >> Thanks, >> >> Butao Zhang >> ---- Replied Message ---- >> From Eugene Miretsky<eug...@badal.io> <eug...@badal.io> >> Date 11/16/2023 02:21 >> To <user@hive.apache.org> <user@hive.apache.org> >> Subject Slow Hive query with a lot of >> 'get_materialized_views_for_rewriting' >> Hey! >> >> We have a catalog with fairly a lot of databases and tables. >> >> Where we do a simple query (select * from table limit 5;) on an ideal >> cluster, it takes around 20seconds, sometimes longer (usually first run >> takes 40s+) >> >> Looking at the hive-metastore logs during most of the query time the logs >> show "metastore.Hivemetastore: 13: get_materialized_views_for_rewriting: >> db = <Database name>" for each database. When these calls finish, the query >> executes pretty quickly. >> >> My interpretation of this is that most of the time is spent on analyzing >> the metastore and building a query plan, perhaps some sort of Metastore >> in-memory cache is building built (but it happens on every call). But I am >> not really sure how to debug it further, nor could find in the code where >> this is happening. >> >> Any advice on what's causing it or how to troubleshoot? >> >> p.s >> The metadata (i.e original tables and databases) is actually coming from >> a very old Hive version (1.1), and we migrated it to the newest version of >> the Metastore using the upgrade tool. In the original Hive version >> materialized views were not even supported. >> >> Cheers, >> Eugene >> >> >> > > -- >
