[ https://issues.apache.org/jira/browse/HIVE-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368883#comment-16368883 ]
Zoltan Haindrich edited comment on HIVE-18743 at 2/19/18 9:00 AM: ------------------------------------------------------------------ I feel that we may probably want to consider to abandon the stats collection in the metastore entirely; it should be done from only the hive task - which already sets DO_NOT_UPDATE_STATS because it have already done it. linked HIVE-17478 was (Author: kgyrtkirk): I fell that we may probably consider to abandon the stats collection in the metastore entirely; it should be done from only the hive task - which already sets DO_NOT_UPDATE_STATS - I think there is a ticket for this somewhere... > CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround > is buggy. > --------------------------------------------------------------------------------------- > > Key: HIVE-18743 > URL: https://issues.apache.org/jira/browse/HIVE-18743 > Project: Hive > Issue Type: Improvement > Components: Metastore > Affects Versions: 1.2.0, 1.1.0 > Reporter: Alexander Behm > Assignee: Alexander Kolbasov > Priority: Major > > When hive.stats.autogather=true then the Metastore lists all files under the > table directory to populate basic stats like file counts and sizes. This file > listing operation can be very expensive particularly on filesystems like S3. > One way to address this issue is to reconfigure hive.stats.autogather=false. > *Here's the bug* > It is my understanding that the DO_NOT_UPDATE_STATS table property is > intended to selectively prevent this stats collection. Unfortunately, this > table property is checked *after* the expensive file listing operation, so > the DO_NOT_UPDATE_STATS does not seem to work as intended. See: > https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java#L633 > Relevant code snippet: > {code} > public static boolean updateTableStatsFast(Database db, Table tbl, > Warehouse wh, > boolean madeDir, boolean > forceRecompute, EnvironmentContext environmentContext) throws MetaException { > if (tbl.getPartitionKeysSize() == 0) { > // Update stats only when unpartitioned > FileStatus[] fileStatuses = wh.getFileStatusesForUnpartitionedTable(db, > tbl); > return updateTableStatsFast(tbl, fileStatuses, madeDir, forceRecompute, > environmentContext); <--- DO_NOT_UPDATE_STATS is checked in here after > wh.getFileStatusesForUnpartitionedTable() has already been called > } else { > return false; > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)