[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410491#comment-15410491 ]
Lefty Leverenz commented on HIVE-14423: --------------------------------------- Doc note: This changes the default value of *hive.metastore.fshandler.threads*, which was introduced by HIVE-13901 (2.1.1 and 2.2.0) but is not documented in the wiki yet. Will this also be committed to branch-2.1 for release 2.1.1? > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > ------------------------------------------------------------------------------------------------ > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement > Reporter: Rajesh Balamohan > Assignee: Rajesh Balamohan > Priority: Minor > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14423.1.patch, HIVE-14423.2.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)