----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66663/#review201594 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java Line 126 (original), 122 (patched) <https://reviews.apache.org/r/66663/#comment282919> this should pass in default values from conf. ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java Line 96 (original), 97 (patched) <https://reviews.apache.org/r/66663/#comment282920> Cache size is used as number of entries. I see your intention but I am not sure it will work as intended. Also, its easier to reason about cache size as number of entries. So, lets use that. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java Line 11643 (original), 11627 (patched) <https://reviews.apache.org/r/66663/#comment282923> Deletion shouldn't be based on total weight. It may skew entries from one large entry. It could be based on min weight, that is delete all entries whose weight < minWeight. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java Line 1636 (original), 1636 (patched) <https://reviews.apache.org/r/66663/#comment282911> This method can be renamed : deleteRuntimeStats() ? standalone-metastore/src/main/thrift/hive_metastore.thrift Line 2180 (original), 2180 (patched) <https://reviews.apache.org/r/66663/#comment282917> should throw MetaException. standalone-metastore/src/main/thrift/hive_metastore.thrift Line 2181 (original), 2181 (patched) <https://reviews.apache.org/r/66663/#comment282918> create a param request object for minCreateTime, i32 maxCount to follow req/resp object convention. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java Lines 59 (patched) <https://reviews.apache.org/r/66663/#comment282922> delete call can return how many rows it deleted. Useful for logging. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java Lines 63 (patched) <https://reviews.apache.org/r/66663/#comment282912> update message. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java Lines 66 (patched) <https://reviews.apache.org/r/66663/#comment282913> update message. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java Lines 584 (patched) <https://reviews.apache.org/r/66663/#comment282914> this wil turn off cleaner thread. Lets use default value of 86400 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java Lines 586 (patched) <https://reviews.apache.org/r/66663/#comment282915> sum of weights is little hard to reason about. Lets just have max weight as config. - Ashutosh Chauhan On April 19, 2018, 3:54 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66663/ > ----------------------------------------------------------- > > (Updated April 19, 2018, 3:54 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-19171 > https://issues.apache.org/jira/browse/HIVE-19171 > > > Repository: hive-git > > > Description > ------- > > * wip patch > * also contains HIVE-19009 > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff99c > > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java > 801de7aca2 > itests/src/test/resources/testconfiguration.properties d26f0ccb17 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > fad8c0ffe9 > metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/Context.java 9ca8b0007e > ql/src/java/org/apache/hadoop/hive/ql/Driver.java f761fffbf1 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > e15c5b7b66 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > e87bbceb7a > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > c3dc848a32 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java e15a49f838 > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > a61a47e390 > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 5b7f4c3560 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 54b705db6e > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > 72092ce4ae > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > a37280407d > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > b5a3c2459f > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > df5aa0c679 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > 2b0d23c6f2 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java 8a5595d210 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > 4ee3c14b39 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > f731315956 > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java 52e18a8030 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > 0afc533da8 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestRuntimeStatsPersistence.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > 18aeb3338b > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > b7263005ed > ql/src/test/queries/clientpositive/runtime_stats_hs2.q PRE-CREATION > ql/src/test/results/clientpositive/llap/runtime_stats_hs2.q.out > PRE-CREATION > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h > 802d8e3fb2 > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp > dfa13a0614 > > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp > c0a39f80e0 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h > 2c95007daa > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp > 99024279c5 > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RuntimeStat.java > PRE-CREATION > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java > a354f27cad > > standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php > 9c949429c5 > standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php c4969d567f > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote > 079c7fc322 > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py > d241414bc3 > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py > 9bf9843314 > standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb > 3dbe4d8068 > standalone-metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb > 58ebd29523 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > ae9ec5cad8 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 7ba053d3e8 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 27f8775a10 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > a7acdcbc23 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java > f6c46ee7bd > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java > PRE-CREATION > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java > ebdcbc237e > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java > 88b07cbdbf > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MRuntimeStat.java > PRE-CREATION > standalone-metastore/src/main/resources/package.jdo 9ddf598d36 > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > 240a282f17 > standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql > 334ddfadd9 > standalone-metastore/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql > e15fc2e31c > standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql > 53bdcc4f59 > standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql > f9efd56833 > standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql > 1c540379d7 > standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql > a87e4464d5 > standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql > 13b0d96bb4 > standalone-metastore/src/main/sql/postgres/hive-schema-3.0.0.postgres.sql > 07dd83d747 > > standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql > ddb5e5009b > standalone-metastore/src/main/thrift/hive_metastore.thrift 5bba3294fa > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > 304f567533 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 85c67270d1 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java > 74c057bc26 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestRuntimeStats.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/66663/diff/2/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
