> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > > Lines 97 (patched) > > <https://reviews.apache.org/r/66663/diff/1/?file=2004568#file2004568line98> > > > > This shouldn't be part of cache. Cache is size limited which expires > > oldest entry to keep constant size. Guava cache has those features.
this was here also because all eviction was controlled from hs2 side ; now there is a cleaner there - so this is not needed > On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > > Lines 11630 (patched) > > <https://reviews.apache.org/r/66663/diff/1/?file=2004593#file2004593line11632> > > > > This shouldn't be an api which MSC invokes. This needs to be a thread > > which runs in Metastore. We already have bunch of these cleaner tasks in > > Metastore. Config metastore.task.threads.always has list of such cleaner > > threads where this should be added. MaterializationsCacheCleanerTask is an > > example of one such cleaner thread. added cleaner thread > On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > > Lines 687 (patched) > > <https://reviews.apache.org/r/66663/diff/1/?file=2004598#file2004598line687> > > > > Is it an unused field for future? How do you envision it being used? this is being used; because an entry has the stats for a whole plan; the weight is the number of operators it contains information for; the limit at both hs2/metastore side is for per operator stat entries; and not "query" count > On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > > Lines 688 (patched) > > <https://reviews.apache.org/r/66663/diff/1/?file=2004598#file2004598line688> > > > > Each entry should store its create time which can then be used to purge > > it later. See e.g., Notification_log tables which stores creation_time and > > uses it later to purge rows. there is a create time :) however it's missing from this sql file... > On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote: > > standalone-metastore/src/main/thrift/hive_metastore.thrift > > Lines 2180 (patched) > > <https://reviews.apache.org/r/66663/diff/1/?file=2004599#file2004599line2180> > > > > There shouldn't be any need to pass maxRetained and maxRetainSecs > > params. This call shouldn't do any cleanup. Thats the job of cleaner thread > > which should read this params from config. ok - Zoltan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66663/#review201359 ----------------------------------------------------------- On April 17, 2018, 12:49 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66663/ > ----------------------------------------------------------- > > (Updated April 17, 2018, 12:49 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-19171 > https://issues.apache.org/jira/browse/HIVE-19171 > > > Repository: hive-git > > > Description > ------- > > * wip patch > * also contains HIVE-19009 > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > e540d023bdda5fbb6152d7ba93c134c0542bf9dd > > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java > 801de7aca288fddcb974603190f342ae9d1b3a71 > itests/src/test/resources/testconfiguration.properties > 48d62a8bf9bcffc2723acb864973a9a53172dceb > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 3cdad284efe92b38f84809ec1a4034aee408ecd8 > metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/Context.java > 9ca8b0007e4cbd0c030a37aa35155526d96bb616 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > a88453c97835db847d74b4b4c3ef318d4d6c0ce5 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > d3fbf07de4613a35f2d172d40053bb26de01407d > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > e87bbceb7aa2a1d0a1f1cc1f00ece2990b9bfd34 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > c3dc848a32a39c30981a0a95678eed01ba547856 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java > e15a49f8380a551f69e11d6b0c560116ffc4a83a > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > a61a47e39054d7f4cca2d8178be904e8073ec0ea > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java > 5b7f4c356000fd745f58addbf879ca9e67d25248 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java > 54b705db6e56bdc82d624518fb764b46f7b7eb72 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > 72092ce4aecec2d0137e106e00eadc6946f28eb8 > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > b5a3c2459f3510f6b1eeea9ff1b13caff4a8fc93 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > df5aa0c679b160bc7ad02fe9a36d6faefd191e3b > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > 2b0d23c6f237915684d59856cb62001c01c8209c > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java > 8a5595d210c0721224fac51be6b3d31c6436a78f > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > 4ee3c14b3988521b3f44d02f032163734ed9e4d9 > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > f7313159561e775a91de8c1f65cd5f2b23586ce5 > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java > 52e18a803075e7a47fb2c7175248a97403682199 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > 0afc533da8ddf2bb6c669e122d6919bacab52b21 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestRuntimeStatsPersistence.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > 18aeb3338b1623c125d30b8e069c64d39801db9a > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > b7263005ed122d257f74995e7a8203f9c0c93b03 > ql/src/test/queries/clientpositive/runtime_stats_hs2.q PRE-CREATION > ql/src/test/results/clientpositive/llap/runtime_stats_hs2.q.out > PRE-CREATION > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h > 802d8e3fb24e36f2fa59a29921f00b8c050c3772 > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp > fd52f09fd6c048065c1bbfd373e89700aaa0de02 > > standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp > c0a39f80e02916d1b2175c4941ffece4857acf98 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h > 55a97de28fde5b45b240cddd0a4f69bef4d58b17 > standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp > 1904047f44a84ecadf299eb7af35628cca2a3233 > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RuntimeStat.java > PRE-CREATION > > standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java > afe82e3a004267bc75fba7b2d939e72db06ac39a > > standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php > 7a8a42a800436454aa418d8d1bbdda02d8d3fc65 > standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php > 14416b4875ff8219ac4ddbc10b13bf84628af5ab > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote > 079c7fc3222a8c09a818f75559d149bfcaba8ba3 > > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py > b0e64d8cef7512c7991b13b1b57e000a0b48080e > standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py > f2f61e04994ddc16b0a97b10988d408e5b2cc933 > standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb > 0e70e8900d03fedfbf9314649340da969738c9ab > standalone-metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb > 58ebd2952392cba2739fab3a8e177066c9cd57fe > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 565549a2c4bad5f9c7b1f8d5d20f78ef2b947758 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 9a43b2c79847e11ef6819b16b56d75cca6294120 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 98674cf7def25b8994ab75ca0f7f9be0232dcbae > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > c5da7b5d34578d0119cb96ed80fce82f6e6412df > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java > f6c46ee7bdc9c21959eb39b6c40b784a488302ed > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java > c47856de8789a9b2022e8921fff056176006e1a5 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MRuntimeStat.java > PRE-CREATION > standalone-metastore/src/main/resources/package.jdo > 9ddf598d3658c23cd47ffbdc0dc77b2115769291 > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql > 240a282f179627b37facb7fba1b98091c6c7e3fc > standalone-metastore/src/main/thrift/hive_metastore.thrift > 612afe1cc07a37fb8255e445208e5d69b3e7591b > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > 304f567533a2e8cfa3a6643462e1e2a430fe6155 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 85c67270d141efc83a3bbf910396ddc708501da5 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java > ecddc7bf28cba599002f543abe932482784b9a70 > > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestRuntimeStats.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/66663/diff/1/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
