[
https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958148#comment-14958148
]
Dongwook Kwon commented on HIVE-3959:
-------------------------------------
This patch is causing performance issue for external table on S3.
I wonder since hive.stats.reliable = false by default, why this fast stat
gathering only relies on hive.stats.autogather(which default is on).
if (HiveConf.getBoolVar(hiveConf,
HiveConf.ConfVars.HIVESTATSAUTOGATHER) &&
!MetaStoreUtils.isView(tbl)) {
if (tbl.getPartitionKeysSize() == 0) { // Unpartitioned table
MetaStoreUtils.updateUnpartitionedTableStatsFast(db, tbl, wh,
madeDir);
} else { // Partitioned table with no partitions.
MetaStoreUtils.updateUnpartitionedTableStatsFast(db, tbl, wh, true);
}
}
Does this should be determined by (hive.stats.reliable = true &&
hive.stats.autogather = true && ! isView) ?
When hive.stats.reliable is false, are fast stats really required at the table
creation time?
On S3, even getting number of files and sizes is expensive for large table.
> Update Partition Statistics in Metastore Layer
> ----------------------------------------------
>
> Key: HIVE-3959
> URL: https://issues.apache.org/jira/browse/HIVE-3959
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Statistics
> Reporter: Bhushan Mandhani
> Assignee: Ashutosh Chauhan
> Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch,
> HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.5.patch,
> HIVE-3959.6.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt,
> HIVE-3959.patch.12.txt, HIVE-3959.patch.2
>
>
> When partitions are created using queries ("insert overwrite" and "insert
> into") then the StatsTask updates all stats. However, when partitions are
> added directly through metadata-only partitions (either CLI or direct calls
> to Thrift Metastore) no stats are populated even if hive.stats.reliable is
> set to true. This puts us in a situation where we can't decide if stats are
> truly reliable or not.
> We propose that the "fast stats" (numFiles and totalSize) which don't require
> a scan of the data should always be populated and be completely reliable. For
> now we are still excluding rowCount and rawDataSize because that will make
> these operations very expensive. Currently they are quick metadata-only ops.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
