subject:"\[jira\] \[Work logged\] \(HIVE\-22869\) Add locking benchmark to metastore\-tools\/metastore\-benchmarks"

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460952
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 20/Jul/20 08:04
Start Date: 20/Jul/20 08:04
Worklog Time Spent: 10m 
  Work Description: deniskuzZ merged pull request #1073:
URL: https://github.com/apache/hive/pull/1073


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460952)
Time Spent: 2h 10m  (was: 2h)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460300
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 13:16
Start Date: 17/Jul/20 13:16
Worklog Time Spent: 10m 
  Work Description: zchovan commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456434704



##
File path: 
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
##
@@ -345,21 +348,44 @@ boolean openTxn(int numTxns) throws TException {
 return openTxns;
   }
 
+  List getOpenTxnsInfo() throws TException {
+return client.get_open_txns_info().getOpen_txns();
+  }
+
   boolean commitTxn(long txnId) throws TException {
 client.commit_txn(new CommitTxnRequest(txnId));
 return true;
   }
 
-  boolean abortTxn(long txnId) throws TException {
-client.abort_txn(new AbortTxnRequest(txnId));
+  boolean abortTxns(List txnIds) throws TException {
+client.abort_txns(new AbortTxnsRequest(txnIds));
 return true;
   }
 
-  boolean abortTxns(List txnIds) throws TException {
-client.abort_txns(new AbortTxnsRequest(txnIds));
+  boolean allocateTableWriteIds(String dbName, String tableName, List 
openTxns) throws TException {
+AllocateTableWriteIdsRequest awiRqst = new 
AllocateTableWriteIdsRequest(dbName, tableName);
+openTxns.forEach(t -> {
+  awiRqst.addToTxnIds(t);
+});
+
+client.allocate_table_write_ids(awiRqst);
 return true;
   }
 
+  boolean getValidWriteIds(List fullTableNames) throws TException {

Review comment:
   ah sorry, I was mistaken, the reason why it never returned the writeIds 
is because they are never used, the benchmark is just executing the api call. 
The return value from the hms is actually a GetValidWriteIdsResponse object, 
not a list. As it is never used I'm not sure if we need to change this.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460300)
Time Spent: 2h  (was: 1h 50m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460294
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 13:05
Start Date: 17/Jul/20 13:05
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456421989



##
File path: 
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
##
@@ -345,21 +348,44 @@ boolean openTxn(int numTxns) throws TException {
 return openTxns;
   }
 
+  List getOpenTxnsInfo() throws TException {
+return client.get_open_txns_info().getOpen_txns();
+  }
+
   boolean commitTxn(long txnId) throws TException {
 client.commit_txn(new CommitTxnRequest(txnId));
 return true;
   }
 
-  boolean abortTxn(long txnId) throws TException {
-client.abort_txn(new AbortTxnRequest(txnId));
+  boolean abortTxns(List txnIds) throws TException {
+client.abort_txns(new AbortTxnsRequest(txnIds));
 return true;
   }
 
-  boolean abortTxns(List txnIds) throws TException {
-client.abort_txns(new AbortTxnsRequest(txnIds));
+  boolean allocateTableWriteIds(String dbName, String tableName, List 
openTxns) throws TException {
+AllocateTableWriteIdsRequest awiRqst = new 
AllocateTableWriteIdsRequest(dbName, tableName);
+openTxns.forEach(t -> {
+  awiRqst.addToTxnIds(t);
+});
+
+client.allocate_table_write_ids(awiRqst);
 return true;
   }
 
+  boolean getValidWriteIds(List fullTableNames) throws TException {

Review comment:
   I don't get what does it have to do with throwingSupplierWrapper. 
throwingSupplierWrapper just handles checked exceptions. Could you please 
elaborate here?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460294)
Time Spent: 1h 50m  (was: 1h 40m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460291
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 13:02
Start Date: 17/Jul/20 13:02
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456427550



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/ACIDBenchmarks.java
##
@@ -0,0 +1,247 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.api.DataOperationType;
+import org.apache.hadoop.hive.metastore.api.LockComponent;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.thrift.TException;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.createManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.dropManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class ACIDBenchmarks {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CoreContext.class);
+
+  @State(Scope.Benchmark)
+  public static class CoreContext {
+@Param("1")
+protected int howMany;
+
+@State(Scope.Thread)
+public static class ThreadState {
+  HMSClient client;
+
+  @Setup
+  public void doSetup() throws Exception {
+LOG.debug("Creating client");
+client = HMSConfig.getInstance().newClient();
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.close();
+LOG.debug("Closed a connection to metastore.");
+  }
+}
+
+@Setup
+public void setup() {
+  LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
+  Configuration ctxConfig = ctx.getConfiguration();
+  
ctxConfig.getLoggerConfig(CoreContext.class.getName()).setLevel(Level.INFO);
+  ctx.updateLoggers(ctxConfig);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestOpenTxn extends CoreContext {
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+LOG.debug("aborted all opened txns");
+  }
+
+  void addTxn(List openTxn) {
+openTxns.addAll(openTxn);
+  }
+}
+
+@Benchmark
+public void openTxn(TestOpenTxn.ThreadState state) throws TException {
+  state.addTxn(state.client.openTxn(howMany));
+  LOG.debug("opened txns, count=", howMany);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestLocking extends CoreContext {
+private int nTables;
+
+@Param("0")
+private int nPartitions;
+
+private List lockComponents;
+
+@Setup
+public void setup() {
+  this.nTables = (nPartitions != 0) ? howMany / nPartitions : howMany;
+  createLockComponents();
+}
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+  long txnId;
+
+  @Setup(org.openjdk.jmh.annotations.Level.Invocation)
+  public void iterSetup() {
+txnId = executeOpenTxnAndGetTxnId(client);
+LOG.debug("opened txn, id={}", txnId);
+openTxns.add(txnId);
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+if (BenchmarkUtils.checkTxnsCleaned(client, openTxns) == false) {
+  LOG.error("Something went wrong with the cleanup of txns");
+}
+LOG.debug("aborted all opened txns");
+  }
+}
+
+@Benchmark
+public void lock(TestLocking.ThreadState state) {
+  LOG.debug("sending lock request");
+  executeLock(state.client, state.txnId, lockComponents);
+}
+
+private void createLockComponents() {
+  lockComponents = new ArrayList<>();
+
+  for (int i = 0; i < nTables; i++) {
+for (int j = 0; j < nPartitions - (nPartitions > 1 ? 1 : 0); j++) {
+  lockComponents.add(
+

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460285
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 12:55
Start Date: 17/Jul/20 12:55
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456423568



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkUtils.java
##
@@ -0,0 +1,72 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.TxnInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.IntStream;
+
+import static org.apache.hadoop.hive.metastore.tools.Util.createSchema;
+import static 
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class BenchmarkUtils {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BenchmarkUtils.class);
+
+
+  static void createManyTables(HMSClient client, int howMany, String dbName, 
String format) {
+List columns = createSchema(new 
ArrayList<>(Arrays.asList("name", "string")));
+List partitions = createSchema(new 
ArrayList<>(Arrays.asList("date", "string")));
+IntStream.range(0, howMany)
+.forEach(i ->
+throwingSupplierWrapper(() -> client.createTable(
+new Util.TableBuilder(dbName, String.format(format, i))
+.withType(TableType.MANAGED_TABLE)
+.withColumns(columns)
+.withPartitionKeys(partitions)
+.build(;
+  }
+
+  static void dropManyTables(HMSClient client, int howMany, String dbName, 
String format) {
+IntStream.range(0, howMany)
+.forEach(i ->
+throwingSupplierWrapper(() -> client.dropTable(dbName, 
String.format(format, i;
+  }
+
+  // Create a simple table with a single column and single partition
+  static void createPartitionedTable(HMSClient client, String dbName, String 
tableName) {
+throwingSupplierWrapper(() -> client.createTable(
+new Util.TableBuilder(dbName, tableName)
+.withType(TableType.MANAGED_TABLE)
+
.withColumns(createSchema(Collections.singletonList("name:string")))
+.withPartitionKeys(createSchema(Collections.singletonList("date")))
+.build()));
+  }
+
+  static boolean checkTxnsCleaned(HMSClient client, List 
txnsOpenedByBenchmark) throws InterruptedException {
+// let's wait the default cleaner run period
+Thread.sleep(10);
+List notCleanedTxns = new ArrayList<>();
+throwingSupplierWrapper(() -> {
+  List txnInfos = client.getOpenTxnsInfo();

Review comment:
   can't see any change here





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460285)
Time Spent: 1.5h  (was: 1h 20m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460284
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 12:54
Start Date: 17/Jul/20 12:54
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456423093



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
##
@@ -141,12 +175,62 @@ private static void saveDataFile(String location, String 
name,
 }
   }
 
-
   @Override
   public void run() {
-LOG.info("Using warmup " + warmup +
-" spin " + spinCount + " nparams " + nParameters + " threads " + 
nThreads);
+LOG.info("Using warmup " + warmup + " spin " + spinCount + " nparams " + 
Arrays.toString(nParameters) + " threads "
++ nThreads);
+HMSConfig.getInstance().init(host, port, confDir);
+
+if (runMode == RunModes.ALL) {

Review comment:
   can't see change here





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460284)
Time Spent: 1h 20m  (was: 1h 10m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=460283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-460283
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 17/Jul/20 12:52
Start Date: 17/Jul/20 12:52
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r456421989



##
File path: 
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
##
@@ -345,21 +348,44 @@ boolean openTxn(int numTxns) throws TException {
 return openTxns;
   }
 
+  List getOpenTxnsInfo() throws TException {
+return client.get_open_txns_info().getOpen_txns();
+  }
+
   boolean commitTxn(long txnId) throws TException {
 client.commit_txn(new CommitTxnRequest(txnId));
 return true;
   }
 
-  boolean abortTxn(long txnId) throws TException {
-client.abort_txn(new AbortTxnRequest(txnId));
+  boolean abortTxns(List txnIds) throws TException {
+client.abort_txns(new AbortTxnsRequest(txnIds));
 return true;
   }
 
-  boolean abortTxns(List txnIds) throws TException {
-client.abort_txns(new AbortTxnsRequest(txnIds));
+  boolean allocateTableWriteIds(String dbName, String tableName, List 
openTxns) throws TException {
+AllocateTableWriteIdsRequest awiRqst = new 
AllocateTableWriteIdsRequest(dbName, tableName);
+openTxns.forEach(t -> {
+  awiRqst.addToTxnIds(t);
+});
+
+client.allocate_table_write_ids(awiRqst);
 return true;
   }
 
+  boolean getValidWriteIds(List fullTableNames) throws TException {

Review comment:
   I don't get what does it have to do with throwingSupplierWrapper. 
throwingSupplierWrapper just handles checked exceptions.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 460283)
Time Spent: 1h 10m  (was: 1h)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=459831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-459831
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 16/Jul/20 14:03
Start Date: 16/Jul/20 14:03
Worklog Time Spent: 10m 
  Work Description: zchovan commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r455810520



##
File path: 
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
##
@@ -345,21 +348,44 @@ boolean openTxn(int numTxns) throws TException {
 return openTxns;
   }
 
+  List getOpenTxnsInfo() throws TException {
+return client.get_open_txns_info().getOpen_txns();
+  }
+
   boolean commitTxn(long txnId) throws TException {
 client.commit_txn(new CommitTxnRequest(txnId));
 return true;
   }
 
-  boolean abortTxn(long txnId) throws TException {
-client.abort_txn(new AbortTxnRequest(txnId));
+  boolean abortTxns(List txnIds) throws TException {
+client.abort_txns(new AbortTxnsRequest(txnIds));
 return true;
   }
 
-  boolean abortTxns(List txnIds) throws TException {
-client.abort_txns(new AbortTxnsRequest(txnIds));
+  boolean allocateTableWriteIds(String dbName, String tableName, List 
openTxns) throws TException {
+AllocateTableWriteIdsRequest awiRqst = new 
AllocateTableWriteIdsRequest(dbName, tableName);
+openTxns.forEach(t -> {
+  awiRqst.addToTxnIds(t);
+});
+
+client.allocate_table_write_ids(awiRqst);
 return true;
   }
 
+  boolean getValidWriteIds(List fullTableNames) throws TException {

Review comment:
   HMClient methods return boolean, so they can be invoked with 
throwingSupplierWrapper()





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 459831)
Time Spent: 1h  (was: 50m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-07-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=459830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-459830
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 16/Jul/20 14:02
Start Date: 16/Jul/20 14:02
Worklog Time Spent: 10m 
  Work Description: zchovan commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r455810030



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkUtils.java
##
@@ -0,0 +1,72 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.TxnInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.IntStream;
+
+import static org.apache.hadoop.hive.metastore.tools.Util.createSchema;
+import static 
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class BenchmarkUtils {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BenchmarkUtils.class);
+
+
+  static void createManyTables(HMSClient client, int howMany, String dbName, 
String format) {
+List columns = createSchema(new 
ArrayList<>(Arrays.asList("name", "string")));

Review comment:
   fixed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 459830)
Time Spent: 50m  (was: 40m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-06-15 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=445872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445872
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 15/Jun/20 14:36
Start Date: 15/Jun/20 14:36
Worklog Time Spent: 10m 
  Work Description: zchovan commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r440221959



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/ACIDBenchmarks.java
##
@@ -0,0 +1,247 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.api.DataOperationType;
+import org.apache.hadoop.hive.metastore.api.LockComponent;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.thrift.TException;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.createManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.dropManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class ACIDBenchmarks {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CoreContext.class);
+
+  @State(Scope.Benchmark)
+  public static class CoreContext {
+@Param("1")
+protected int howMany;
+
+@State(Scope.Thread)
+public static class ThreadState {
+  HMSClient client;
+
+  @Setup
+  public void doSetup() throws Exception {
+LOG.debug("Creating client");
+client = HMSConfig.getInstance().newClient();
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.close();
+LOG.debug("Closed a connection to metastore.");
+  }
+}
+
+@Setup
+public void setup() {
+  LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
+  Configuration ctxConfig = ctx.getConfiguration();
+  
ctxConfig.getLoggerConfig(CoreContext.class.getName()).setLevel(Level.INFO);
+  ctx.updateLoggers(ctxConfig);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestOpenTxn extends CoreContext {
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+LOG.debug("aborted all opened txns");
+  }
+
+  void addTxn(List openTxn) {
+openTxns.addAll(openTxn);
+  }
+}
+
+@Benchmark
+public void openTxn(TestOpenTxn.ThreadState state) throws TException {
+  state.addTxn(state.client.openTxn(howMany));
+  LOG.debug("opened txns, count=", howMany);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestLocking extends CoreContext {
+private int nTables;
+
+@Param("0")
+private int nPartitions;
+
+private List lockComponents;
+
+@Setup
+public void setup() {
+  this.nTables = (nPartitions != 0) ? howMany / nPartitions : howMany;
+  createLockComponents();
+}
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+  long txnId;
+
+  @Setup(org.openjdk.jmh.annotations.Level.Invocation)
+  public void iterSetup() {
+txnId = executeOpenTxnAndGetTxnId(client);
+LOG.debug("opened txn, id={}", txnId);
+openTxns.add(txnId);
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+if (BenchmarkUtils.checkTxnsCleaned(client, openTxns) == false) {
+  LOG.error("Something went wrong with the cleanup of txns");
+}
+LOG.debug("aborted all opened txns");
+  }
+}
+
+@Benchmark
+public void lock(TestLocking.ThreadState state) {
+  LOG.debug("sending lock request");
+  executeLock(state.client, state.txnId, lockComponents);
+}
+
+private void createLockComponents() {
+  lockComponents = new ArrayList<>();
+
+  for (int i = 0; i < nTables; i++) {
+for (int j = 0; j < nPartitions - (nPartitions > 1 ? 1 : 0); j++) {
+  lockComponents.add(
+

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-06-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=444355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444355
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 11/Jun/20 15:44
Start Date: 11/Jun/20 15:44
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #1073:
URL: https://github.com/apache/hive/pull/1073#discussion_r438868981



##
File path: 
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/ACIDBenchmarks.java
##
@@ -0,0 +1,247 @@
+package org.apache.hadoop.hive.metastore.tools;
+
+import org.apache.hadoop.hive.metastore.api.DataOperationType;
+import org.apache.hadoop.hive.metastore.api.LockComponent;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.thrift.TException;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.createManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.BenchmarkUtils.dropManyTables;
+import static 
org.apache.hadoop.hive.metastore.tools.Util.throwingSupplierWrapper;
+
+public class ACIDBenchmarks {
+
+  private static final Logger LOG = LoggerFactory.getLogger(CoreContext.class);
+
+  @State(Scope.Benchmark)
+  public static class CoreContext {
+@Param("1")
+protected int howMany;
+
+@State(Scope.Thread)
+public static class ThreadState {
+  HMSClient client;
+
+  @Setup
+  public void doSetup() throws Exception {
+LOG.debug("Creating client");
+client = HMSConfig.getInstance().newClient();
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.close();
+LOG.debug("Closed a connection to metastore.");
+  }
+}
+
+@Setup
+public void setup() {
+  LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
+  Configuration ctxConfig = ctx.getConfiguration();
+  
ctxConfig.getLoggerConfig(CoreContext.class.getName()).setLevel(Level.INFO);
+  ctx.updateLoggers(ctxConfig);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestOpenTxn extends CoreContext {
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+LOG.debug("aborted all opened txns");
+  }
+
+  void addTxn(List openTxn) {
+openTxns.addAll(openTxn);
+  }
+}
+
+@Benchmark
+public void openTxn(TestOpenTxn.ThreadState state) throws TException {
+  state.addTxn(state.client.openTxn(howMany));
+  LOG.debug("opened txns, count=", howMany);
+}
+  }
+
+  @State(Scope.Benchmark)
+  public static class TestLocking extends CoreContext {
+private int nTables;
+
+@Param("0")
+private int nPartitions;
+
+private List lockComponents;
+
+@Setup
+public void setup() {
+  this.nTables = (nPartitions != 0) ? howMany / nPartitions : howMany;
+  createLockComponents();
+}
+
+@State(Scope.Thread)
+public static class ThreadState extends CoreContext.ThreadState {
+  List openTxns = new ArrayList<>();
+  long txnId;
+
+  @Setup(org.openjdk.jmh.annotations.Level.Invocation)
+  public void iterSetup() {
+txnId = executeOpenTxnAndGetTxnId(client);
+LOG.debug("opened txn, id={}", txnId);
+openTxns.add(txnId);
+  }
+
+  @TearDown
+  public void doTearDown() throws Exception {
+client.abortTxns(openTxns);
+if (BenchmarkUtils.checkTxnsCleaned(client, openTxns) == false) {
+  LOG.error("Something went wrong with the cleanup of txns");
+}
+LOG.debug("aborted all opened txns");
+  }
+}
+
+@Benchmark
+public void lock(TestLocking.ThreadState state) {
+  LOG.debug("sending lock request");
+  executeLock(state.client, state.txnId, lockComponents);
+}
+
+private void createLockComponents() {
+  lockComponents = new ArrayList<>();
+
+  for (int i = 0; i < nTables; i++) {
+for (int j = 0; j < nPartitions - (nPartitions > 1 ? 1 : 0); j++) {
+  lockComponents.add(
+

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-06-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=444227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444227
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 11/Jun/20 11:20
Start Date: 11/Jun/20 11:20
Worklog Time Spent: 10m 
  Work Description: zchovan commented on pull request #1073:
URL: https://github.com/apache/hive/pull/1073#issuecomment-642579576


   @ deniskuzz could you review?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 444227)
Remaining Estimate: 0h
Time Spent: 10m

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

2020-06-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22869?focusedWorklogId=444228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-444228
 ]

ASF GitHub Bot logged work on HIVE-22869:
-

Author: ASF GitHub Bot
Created on: 11/Jun/20 11:20
Start Date: 11/Jun/20 11:20
Worklog Time Spent: 10m 
  Work Description: zchovan edited a comment on pull request #1073:
URL: https://github.com/apache/hive/pull/1073#issuecomment-642579576


   @deniskuzz could you review?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 444228)
Time Spent: 20m  (was: 10m)

> Add locking benchmark to metastore-tools/metastore-benchmarks
> -
>
> Key: HIVE-22869
> URL: https://issues.apache.org/jira/browse/HIVE-22869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, 
> HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, 
> HIVE-22869.7.patch, HIVE-22869.8.patch, HIVE-22869.9.patch, HIVE-22869.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add the possibility to run benchmarks on opening lock in the HMS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

[jira] [Work logged] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks

13 matches

Site Navigation

Mail list logo

Footer information