This is an automated email from the ASF dual-hosted git repository.
kuczoram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 8eee4aa9d1b HIVE-28099: Fix logging in HMS benchmarks (#5112) (Zsolt
Miskolczi, reviewed by Laszlo Bodor)
8eee4aa9d1b is described below
commit 8eee4aa9d1bd6f4193471f5d014324bbaf552041
Author: InvisibleProgrammer <[email protected]>
AuthorDate: Fri Mar 22 09:35:34 2024 +0100
HIVE-28099: Fix logging in HMS benchmarks (#5112) (Zsolt Miskolczi,
reviewed by Laszlo Bodor)
* HIVE-28099: Fix logging in HMS benchmarks
* Update ReadMe
* Fix formatting of pom.xml
* Fix indentation
* Address review comments
---
.../metastore-tools/metastore-benchmarks/README.md | 33 ++++++++-----
.../metastore-tools/metastore-benchmarks/pom.xml | 54 +++++++++++++++-------
2 files changed, 58 insertions(+), 29 deletions(-)
diff --git
a/standalone-metastore/metastore-tools/metastore-benchmarks/README.md
b/standalone-metastore/metastore-tools/metastore-benchmarks/README.md
index ab6683e1392..2510b1e2442 100644
--- a/standalone-metastore/metastore-tools/metastore-benchmarks/README.md
+++ b/standalone-metastore/metastore-tools/metastore-benchmarks/README.md
@@ -2,16 +2,20 @@
## Installation
- mvn clean install
+ mvn clean install -Pperf
You can run tests as well. Just set `HMS_HOST` environment variable to some
HMS instance which is
capable of running your requests (non-kerberised one) and run
- mvn install
+ mvn install -Pperf
target directory has two mega-jars which have all the dependencies.
-Alternatively you can use [bin/hbench](../bin/hbench) script which use Maven
to run the code.
+## Use local build
+
+ mvn clean package -Pperf
+
+**Note:** It does benchmarks on HMS. So do not forget to start a standalone
metastore service to be able to run.
## HmsBench usage
@@ -53,25 +57,30 @@ Alternatively you can use [bin/hbench](../bin/hbench)
script which use Maven to
### Using single jar
- java -jar hbench-jar-with-dependencies.jar <optins> [test]...
+ java -jar hmsbench.jar <optins> [test]...
### Using hbench on kerberized cluster
- java -jar hbench-jar-with-dependencies.jar -H `hostname` <optins> [test]...
+ java -jar hmsbench.jar -H `hostname` <optins> [test]...
### Examples
1. Run all tests with default settings
- java -jar hmsbench-jar-with-dependencies.jar -d `metastore_db_name` -H
`hostname`
-2. Run tests with 500 objects created, 10 times warm-up and exclude concurrent
operations and drop operations
+ ```java -jar hmsbench.jar -d `metastore_db_name` -H `hostname` ```
+
+2. Run a single test
+
+ ```java -jar hmsbench.jar -d `metastore_db_name` -H `hostname` -M
"TestGetValidWriteIds"```
+
+3. Run tests with 500 objects created, 10 times warm-up and exclude concurrent
operations and drop operations
- java -jar hmsbench-jar-with-dependencies.jar -d `metastore_db_name` -H
`hostname` -N 500 -W 10 -E 'drop.*' -E 'concurrent.*'
+ ```java -jar hmsbench.jar -d `metastore_db_name` -H `hostname` -N 500 -W
10 -E 'drop.*' -E 'concurrent.*'```
-3. Run tests, produce output in tab-separated format and write individual data
points in 'data' directory
+4. Run tests, produce output in tab-separated format and write individual data
points in 'data' directory
- java -jar hmsbench-jar-with-dependencies.jar -d `metastore_db_name` -H
`hostname` -o result.csv --csv --savedata data
+ ```java -jar hmsbench.jar -d `metastore_db_name` -H `hostname` -o
result.csv --csv --savedata data```
-4. Run tests on localhost
+5. Run tests on localhost
* save raw data in directory /tmp/benchdata
* sanitize results (remove outliers)
* produce tab-separated file
@@ -80,7 +89,7 @@ Alternatively you can use [bin/hbench](../bin/hbench) script
which use Maven to
* run with 100 and thousand partitions
- java -jar hmsbench-jar-with-dependencies.jar -H `hostname` \
+ java -jar hmsbench.jar -H `hostname` \
--savedata /tmp/benchdata \
--sanitize \
-N 100 -N 1000 \
diff --git a/standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml
b/standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml
index e997d31afe0..7d62642f925 100644
--- a/standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml
+++ b/standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml
@@ -68,6 +68,10 @@
<groupId>io.netty</groupId>
<artifactId>*</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.springframework</groupId>
+ <artifactId>spring-jcl</artifactId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
@@ -139,30 +143,46 @@
<build>
<plugins>
<plugin>
- <artifactId>maven-assembly-plugin</artifactId>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.4.0</version>
<executions>
<execution>
- <configuration>
- <archive>
- <manifest>
-
<mainClass>org.apache.hadoop.hive.metastore.tools.BenchmarkTool</mainClass>
- <addClasspath>true</addClasspath>
- </manifest>
- </archive>
- <descriptorRefs>
- <descriptorRef>jar-with-dependencies</descriptorRef>
- </descriptorRefs>
- <finalName>hmsbench</finalName>
- </configuration>
- <id>make-assembly-hclient</id>
- <!-- this is used for inheritance merges -->
<phase>package</phase>
- <!-- bind to the packaging phase -->
<goals>
- <goal>single</goal>
+ <goal>shade</goal>
</goals>
+ <configuration>
+ <finalName>hmsbench</finalName>
+ <transformers>
+ <transformer
implementation="org.apache.logging.log4j.maven.plugins.shade.transformer.Log4j2PluginCacheFileTransformer"/>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+
<mainClass>org.apache.hadoop.hive.metastore.tools.BenchmarkTool</mainClass>
+ </transformer>
+ </transformers>
+ <filters>
+ <filter>
+ <!--
+
https://logging.apache.org/log4j/transform/latest/#log4j-plugin-cache-transformer
+ -->
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
</execution>
</executions>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+
<artifactId>log4j-transform-maven-shade-plugin-extensions</artifactId>
+ <version>0.1.0</version>
+ </dependency>
+ </dependencies>
</plugin>
</plugins>
</build>
