[
https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149145#comment-16149145
]
kevin zou commented on DRILL-5738:
----------------------------------
on 1.10, the number of metadata query reduced from millions to thousands. But
overall time to spend for sql path generation is still very high (7 minutes)
> Drill query takes 10+ minutes before start executing, excessive Hive
> metastore queries
> --------------------------------------------------------------------------------------
>
> Key: DRILL-5738
> URL: https://issues.apache.org/jira/browse/DRILL-5738
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Affects Versions: 1.6.0, 1.10.0
> Environment: mapr 5.2
> Reporter: kevin zou
> Priority: Critical
> Attachments: log.tar.gz
>
>
> I ve a Drill query on 14 tables in Hive. The query took a few seconds to
> execute. However, the query would stay in "Starting" state for 10+ minutes
> before execution.
>
> I set up the log to "Debug" mode to figure out what Drill had been doing
> during the 10+ minutes, only to find out Drill generated excessive number of
> meta data queries to hive meta store.
>
> Although each query took a few micro seconds (meta data cached in memory),
> the number of queries was 3438793.
> drillbit.log:2017-06-05 18:50:57,201
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG
> o.a.d.e.s.hive.HiveMetadataProvider
> - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory
> drillbit.log:2017-06-05 18:50:57,201
> [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG
> o.a.drill.exec.store.hive.HiveScan
> - HiveStats: numRows: 15, sizeInBytes: 15
> ........
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
