Rajesh Balamohan created HIVE-14423:
---------------------------------------
Summary: S3: Fetching partition sizes from FS can be expensive
when stats are not available in metastore
Key: HIVE-14423
URL: https://issues.apache.org/jira/browse/HIVE-14423
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Priority: Minor
When partition stats are not available in metastore, it tries to get the file
sizes from FS.
e.g
{noformat}
at
org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
at
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
{noformat}
This can be quite expensive in some FS like S3. Especially when table is
partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can
spend 1000s of seconds just waiting for these information to be pulled in.
Also, it would be good to remove FS.getContentSummary usage to find out file
sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
