-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50788/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-14423
https://issues.apache.org/jira/browse/HIVE-14423
Repository: hive-git
Description
-------
When partition stats are not available in metastore, it tries to get the file
sizes from FS.
e.g
at
org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
at
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
This can be quite expensive in some FS like S3. Especially when table is
partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can
spend 1000s of seconds just waiting for these information to be pulled in.
Also, it would be good to remove FS.getContentSummary usage to find out file
sizes.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f5f619
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 2a9dc11
Diff: https://reviews.apache.org/r/50788/diff/
Testing
-------
Thanks,
Rajesh Balamohan
