[ https://issues.apache.org/jira/browse/HIVE-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385296#comment-16385296 ]
Zoltan Haindrich commented on HIVE-18743: ----------------------------------------- I don't think so...you left out the other parts of that code... https://github.com/apache/hive/blob/05d4719eefc56676a3e0e8f706e1c5e5e1f6b345/standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java#L232 [~akolb] Could you please add a high level qtest ? the testcase from testmetastore will also be removed... > CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround > is buggy. > --------------------------------------------------------------------------------------- > > Key: HIVE-18743 > URL: https://issues.apache.org/jira/browse/HIVE-18743 > Project: Hive > Issue Type: Improvement > Components: Metastore > Affects Versions: 1.2.0, 1.1.0, 2.0.2, 3.0.0 > Reporter: Alexander Behm > Assignee: Alexander Kolbasov > Priority: Major > Attachments: HIVE-18743.06.patch, HIVE-18743.07.patch > > > When hive.stats.autogather=true then the Metastore lists all files under the > table directory to populate basic stats like file counts and sizes. This file > listing operation can be very expensive particularly on filesystems like S3. > One way to address this issue is to reconfigure hive.stats.autogather=false. > *Here's the bug* > It is my understanding that the DO_NOT_UPDATE_STATS table property is > intended to selectively prevent this stats collection. Unfortunately, this > table property is checked *after* the expensive file listing operation, so > the DO_NOT_UPDATE_STATS does not seem to work as intended. See: > https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java#L633 > Relevant code snippet: > {code} > public static boolean updateTableStatsFast(Database db, Table tbl, > Warehouse wh, > boolean madeDir, boolean > forceRecompute, EnvironmentContext environmentContext) throws MetaException { > if (tbl.getPartitionKeysSize() == 0) { > // Update stats only when unpartitioned > FileStatus[] fileStatuses = wh.getFileStatusesForUnpartitionedTable(db, > tbl); > return updateTableStatsFast(tbl, fileStatuses, madeDir, forceRecompute, > environmentContext); <--- DO_NOT_UPDATE_STATS is checked in here after > wh.getFileStatusesForUnpartitionedTable() has already been called > } else { > return false; > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)