> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote: > > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java > > Lines 414 (patched) > > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line429> > > > > I think more realistic scenario is having up to 10 tables with 10-1000 > > (configurable) partitions and locking not only on table level but partition. > > Zoltan Chovan wrote: > I agree with that, however from a strictly api-testing viewpoint I'm not > sure if there's any practical difference between creating locks for 10-1000 > tables or 10-1000 partitions. The lock objects would just point to a > different object, the number of HMS calls should be the same.
what I ment, we should have a combination of table level and partition level locks (i.e. table level lock would join with blocking partition level locks and other way around) > On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote: > > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java > > Lines 422 (patched) > > <https://reviews.apache.org/r/72112/diff/1/?file=2210962#file2210962line437> > > > > I think, for more realistic scenario we should move commit inside of > > bench + add some sleep to simulate query execution. Currently locks number > > is just keep increasing and could go very high that shouldn't happen in > > real system. > > Zoltan Chovan wrote: > If we start going down the query simulation road then this will be an e2e > test, not an api benchmark :) > Let's say we add sleep to simulate queries. If every query takes the same > time (which is again not realistic) then we just add a wait and every client > would try the calls at the same time again. If we add variable lenghts to > quereis then this will not be a benchmark that is reproducable at will. Could you please elaborate on your test scenarios? If I am reading this right, currently we are only interested in a read-locks. Is it correct? Also we are releasing locks almost instantaneously via commit, so we won't have big number of rows in HIVE_LOCKS table. Should we consider doing iniatial load of locks into this table? - Denys ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72112/#review220213 ----------------------------------------------------------- On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72112/ > ----------------------------------------------------------- > > (Updated April 9, 2020, 2:58 p.m.) > > > Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter > Vary. > > > Repository: hive-git > > > Description > ------- > > Add the possibility to run benchmarks on opening lock in the HMS. Currently > this change only introduces single-threaded/single client testing. I'm > planning to add multi-client support in a separate change. > > Example parametrisation is as follows: > hbench -M "lock" -N 10 -d hive_test -W 0 -L 100 > hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100 > > This will create N number (10) of tables to lock and it'll execute the lock() > for L number (100) of times on T (8) threads where each thread will strart an > HMS client > > > Diffs > ----- > > > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java > 041cd76234 > > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java > f53f2ef43b > > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java > 7cc1e42a8b > > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java > 101d6759c5 > > > Diff: https://reviews.apache.org/r/72112/diff/1/ > > > Testing > ------- > > > File Attachments > ---------------- > > HIVE-22869.2.patch > > https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch > HIVE-22869.3.patch > > https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch > > > Thanks, > > Zoltan Chovan > >
