----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66663/#review201359 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java Lines 126 (patched) <https://reviews.apache.org/r/66663/#comment282555> This may become an expensive call. Currently, first query which loads this cache pays for it which is unfair for that query. This cache should be loaded async when HS2 starts. See : HiveMaterializedViewsRegistry.java This cache should behave exactly same as that in terms of how and when its initialized and loads from Metastore. ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java Lines 154 (patched) <https://reviews.apache.org/r/66663/#comment282551> This should be an async fire and forget call. This thread shouldn't wait for this to complete. If that async call fails, that should be just logged and ignored (which is what current Catch is doing). ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java Lines 97 (patched) <https://reviews.apache.org/r/66663/#comment282557> This shouldn't be part of cache. Cache is size limited which expires oldest entry to keep constant size. Guava cache has those features. standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java Lines 11630 (patched) <https://reviews.apache.org/r/66663/#comment282558> This shouldn't be an api which MSC invokes. This needs to be a thread which runs in Metastore. We already have bunch of these cleaner tasks in Metastore. Config metastore.task.threads.always has list of such cleaner threads where this should be added. MaterializationsCacheCleanerTask is an example of one such cleaner thread. standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql Lines 687 (patched) <https://reviews.apache.org/r/66663/#comment282561> Is it an unused field for future? How do you envision it being used? standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql Lines 688 (patched) <https://reviews.apache.org/r/66663/#comment282560> Each entry should store its create time which can then be used to purge it later. See e.g., Notification_log tables which stores creation_time and uses it later to purge rows. standalone-metastore/src/main/thrift/hive_metastore.thrift Lines 2180 (patched) <https://reviews.apache.org/r/66663/#comment282559> There shouldn't be any need to pass maxRetained and maxRetainSecs params. This call shouldn't do any cleanup. Thats the job of cleaner thread which should read this params from config. - Ashutosh Chauhan On April 17, 2018, 12:49 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66663/ > ----------------------------------------------------------- > > (Updated April 17, 2018, 12:49 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-19171 > https://issues.apache.org/jira/browse/HIVE-19171 > > > Repository: hive-git > > > Description > ------- > > * wip patch > * also contains HIVE-19009 > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > e540d023bdda5fbb6152d7ba93c134c0542bf9dd > > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java > 801de7aca288fddcb974603190f342ae9d1b3a71 > itests/src/test/resources/testconfiguration.properties > 48d62a8bf9bcffc2723acb864973a9a53172dceb > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 3cdad284efe92b38f84809ec1a4034aee408ecd8 > metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/Context.java > 9ca8b0007e4cbd0c030a37aa35155526d96bb616 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > a88453c97835db847d74b4b4c3ef318d4d6c0ce5 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > d3fbf07de4613a35f2d172d40053bb26de01407d > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > e87bbceb7aa2a1d0a1f1cc1f00ece2990b9bfd34 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > c3dc848a32a39c30981a0a95678eed01ba547856 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java > e15a49f8380a551f69e11d6b0c560116ffc4a83a > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > a61a47e39054d7f4cca2d8178be904e8073ec0ea > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java > 5b7f4c356000fd745f58addbf879ca9e67d25248 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java > 54b705db6e56bdc82d624518fb764b46f7b7eb72 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > 72092ce4aecec2d0137e106e00eadc6946f28eb8 > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > b5a3c2459f3510f6b1eeea9ff1b13caff4a8fc93 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > df5aa0c679b160bc7ad02fe9a36d6faefd191e3b > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > 2b0d23c6f237915684d59856cb62001c01c8209c > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java > 8a5595d210c0721224fac51be6b3d31c6436a78f > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > 4ee3c14b3988521b3f44d02f032163734ed9e4d9 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > f7313159561e775a91de8c1f65cd5f2b23586ce5 > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java > 52e18a803075e7a47fb2c7175248a97403682199 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > 0afc533da8ddf2bb6c669e122d6919bacab52b21 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestRuntimeStatsPersistence.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > 18aeb3338b1623c125d30b8e069c64d39801db9a > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > b7263005ed122d257f74995e7a8203f9c0c93b03 > ql/src/test/queries/clientpositive/runtime_stats_hs2.q PRE-CREATION > ql/src/test/results/clientpositive/llap/runtime_stats_hs2.q.out > PRE-CREATION > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h > 802d8e3fb24e36f2fa59a29921f00b8c050c3772 > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp > fd52f09fd6c048065c1bbfd373e89700aaa0de02 > > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp > c0a39f80e02916d1b2175c4941ffece4857acf98 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h > 55a97de28fde5b45b240cddd0a4f69bef4d58b17 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp > 1904047f44a84ecadf299eb7af35628cca2a3233 > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RuntimeStat.java > PRE-CREATION > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java > afe82e3a004267bc75fba7b2d939e72db06ac39a > > standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php > 7a8a42a800436454aa418d8d1bbdda02d8d3fc65 > standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php > 14416b4875ff8219ac4ddbc10b13bf84628af5ab > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote > 079c7fc3222a8c09a818f75559d149bfcaba8ba3 > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py > b0e64d8cef7512c7991b13b1b57e000a0b48080e > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py > f2f61e04994ddc16b0a97b10988d408e5b2cc933 > standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb > 0e70e8900d03fedfbf9314649340da969738c9ab > standalone-metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb > 58ebd2952392cba2739fab3a8e177066c9cd57fe > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 565549a2c4bad5f9c7b1f8d5d20f78ef2b947758 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 9a43b2c79847e11ef6819b16b56d75cca6294120 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 98674cf7def25b8994ab75ca0f7f9be0232dcbae > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > c5da7b5d34578d0119cb96ed80fce82f6e6412df > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java > f6c46ee7bdc9c21959eb39b6c40b784a488302ed > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java > c47856de8789a9b2022e8921fff056176006e1a5 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MRuntimeStat.java > PRE-CREATION > standalone-metastore/src/main/resources/package.jdo > 9ddf598d3658c23cd47ffbdc0dc77b2115769291 > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > 240a282f179627b37facb7fba1b98091c6c7e3fc > standalone-metastore/src/main/thrift/hive_metastore.thrift > 612afe1cc07a37fb8255e445208e5d69b3e7591b > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > 304f567533a2e8cfa3a6643462e1e2a430fe6155 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 85c67270d141efc83a3bbf910396ddc708501da5 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java > ecddc7bf28cba599002f543abe932482784b9a70 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestRuntimeStats.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/66663/diff/1/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
