[
https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15300:
------------------------------------
Attachment: HIVE-15300.1.patch
{noformat}
Without Patch:
==============
2016-11-29 02:51:44,145 INFO [HiveServer2-Handler-Pool: Thread-58]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480387883229 end=1480387904145 duration=20916
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 02:51:44,145 INFO [HiveServer2-Handler-Pool: Thread-58]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms :
getAggrColStatsFor_(String, String, List, List, )=1767
2016-11-29 02:51:44,145 INFO [HiveServer2-Handler-Pool: Thread-58]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=10659
..
..
2016-11-29 03:01:07,740 INFO [HiveServer2-Handler-Pool: Thread-142]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480388450477 end=1480388467740 duration=17263
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 03:01:07,740 INFO [HiveServer2-Handler-Pool: Thread-142]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms :
getAggrColStatsFor_(String, String, List, List, )=1310
2016-11-29 03:01:07,740 INFO [HiveServer2-Handler-Pool: Thread-142]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=11685
With Patch:
==============
2016-11-29 04:06:55,869 INFO [HiveServer2-Handler-Pool: Thread-56]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480392403722 end=1480392415869 duration=12147
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 04:06:55,870 INFO [HiveServer2-Handler-Pool: Thread-56]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms :
getAggrColStatsFor_(String, String, List, List, )=1771
2016-11-29 04:06:55,870 INFO [HiveServer2-Handler-Pool: Thread-56]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=1948
...
...
2016-11-29 04:07:53,766 INFO [HiveServer2-Handler-Pool: Thread-106]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480392466981 end=1480392473766 duration=6785
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 04:07:53,767 INFO [HiveServer2-Handler-Pool: Thread-106]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms :
getAggrColStatsFor_(String, String, List, List, )=1198
2016-11-29 04:07:53,767 INFO [HiveServer2-Handler-Pool: Thread-106]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=1384
With Patch + HIVE-15069 (which optimizes for cache):
====================================================
2016-11-29 03:47:08,735 INFO [HiveServer2-Handler-Pool: Thread-60]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480391218305 end=1480391228735 duration=10430
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 03:47:08,735 INFO [HiveServer2-Handler-Pool: Thread-60]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=2091
...
...
2016-11-29 03:54:17,798 INFO [HiveServer2-Handler-Pool: Thread-147]:
log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=compile
start=1480391653429 end=1480391657798 duration=4369
from=org.apache.hadoop.hive.ql.Driver>
2016-11-29 03:54:17,799 INFO [HiveServer2-Handler-Pool: Thread-147]:
metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(3518)) - Total time spent
in this metastore function was greater than 1000ms : getTable_(String, String,
)=1044
{noformat}
Patch was with checked with 1.2.x, but same issue is applicable in master as
well.
> Reuse table information in SemanticAnalyzer::getMetaData to reduce
> compilation time
> -----------------------------------------------------------------------------------
>
> Key: HIVE-15300
> URL: https://issues.apache.org/jira/browse/HIVE-15300
> Project: Hive
> Issue Type: Improvement
> Components: Parser
> Reporter: Rajesh Balamohan
> Priority: Minor
> Attachments: HIVE-15300.1.patch
>
>
> E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the
> table details for the same table repeatedly. It took 20+seconds to compile
> the query.
> It would be good to reuse the table information in
> SemanticAnalyzer::getMetadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
