[
https://issues.apache.org/jira/browse/SOLR-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350128#comment-17350128
]
Mark Robert Miller commented on SOLR-15428:
-------------------------------------------
Implementation is not currently based on the JMH Gradle Plugin with fat jars,
shading, it's own config setup, etc, but a more direct implementation into raw
jmh via a JavaExec task.
Simple example usage and output (via latest local ref branch, my 'apache/main'
work on this is not fully synced up yet)
The following would kick off a full integration type test called
PointsVsTrieIndex, utilizing MiniSolrCloudCluster and multiple shards +
replicas. Ideally, much like good Unit Testing, it would be augmented with
additional microbenchmarks at much lower levels that do not spin up clusters
and Jetty instances, etc. The implementation (not currently included) is
standard jmh annotated benchmark class Java code.
{noformat}
/data2/lucene-solr-3/solr/test-framework$ gw --no-daemon jmh
-Pinclude=org.apache.solr.bench.schema.PointsVsTrieIndex.*
...
# JMH version: 1.29
# VM version: JDK 11.0.11, OpenJDK 64-Bit Server VM, 11.0.11+9-Ubuntu-0ubuntu2
# VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
# VM options: -Xms4g -Djmh.separateClassLoader=true -Dlog4j2.is.webapp=false
-Dlog4j2.garbagefreeThreadContextMap=true -Dlog4j2.enableDirectEncoders=true
-Dlog4j2.enable.threadlocals=true -XX:ConcGCThreads=2 -XX:ParallelGCThreads=3
-XX:+UseG1GC -Djetty.insecurerandom=1 -Djava.security.egd=file:/dev/./urandom
-XX:-UseBiasedLocking -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=250 -Dsolr.log.dir= -Djava.class.path=...
-Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant
# Blackhole mode: full + dont-inline hint
# Warmup: 4 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 300 s per iteration
# Threads: 6 threads, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc
# Parameters: (docValues = useAsStored, fieldType = trie)# Run progress: 83.33%
complete, ETA 00:01:45
# Fork: 1 of 1
# Warmup Iteration 1: cluster base
path=/tmp/test-1_001-948879960720677/tempDir-001
4.062 ops/ms
# Warmup Iteration 2: cluster base
path=/tmp/test-1_001-948879960720677/tempDir-002
4.890 ops/ms
# Warmup Iteration 3: cluster base
path=/tmp/test-1_001-948879960720677/tempDir-003
5.503 ops/ms
# Warmup Iteration 4: cluster base
path=/tmp/test-1_001-948879960720677/tempDir-004
5.438 ops/ms
Iteration 1: cluster base path=/tmp/test-1_001-948879960720677/tempDir-005
<5.378 ops/ms
Iteration 2: cluster base path=/tmp/test-1_001-948879960720677/tempDir-006
5.541 ops/ms
Iteration 3: cluster base path=/tmp/test-1_001-948879960720677/tempDir-007
5.586 ops/ms
Iteration 4: cluster base path=/tmp/test-1_001-948879960720677/tempDir-008
5.523 ops/ms
Iteration 5: cluster base path=/tmp/test-1_001-948879960720677/tempDir-009
5.517 ops/ms
Result "org.apache.solr.bench.schema.PointsVsTrieIndex.indexSmallDoc":
5.509 ±(99.9%) 0.301 ops/ms [Average]
(min, avg, max) = (5.378, 5.509, 5.586), stdev = 0.078
CI (99.9%): [5.208, 5.810] (assumes normal distribution)
# Run complete. Total time: 00:10:35
REMEMBER: The numbers below are just data. To gain reusable insights, you need
to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design
factorial
experiments, perform baseline and negative tests that provide experimental
control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from
the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark (docValues) (fieldType) Mode Cnt Score
Error Units
PointsVsTrieIndex.indexSmallDoc disabled point thrpt 5 5.927 ±
0.192 ops/ms
PointsVsTrieIndex.indexSmallDoc disabled trie thrpt 5 5.559 ±
0.180 ops/ms
PointsVsTrieIndex.indexSmallDoc enabled point thrpt 5 5.776 ±
1.263 ops/ms
PointsVsTrieIndex.indexSmallDoc enabled trie thrpt 5 5.681 ±
0.264 ops/ms
PointsVsTrieIndex.indexSmallDoc useAsStored point thrpt 5 5.884 ±
0.211 ops/ms
PointsVsTrieIndex.indexSmallDoc useAsStored trie thrpt 5 5.509 ±
0.301 ops/ms
Benchmark result is saved to
/data2/lucene-solr-3/solr/test-framework/build/reports/jmh/result.json
{noformat}
Result format can be adjusted as desired as pretty much all options provided by
jmh ([https://openjdk.java.net/projects/code-tools/jmh/)]
For graphing, or comparing a run before and after your changes, I would
recommend the excellent JMH Visualizer,
[https://jmh.morethan.io/.|https://jmh.morethan.io/] Easy to use in a variety
of ways via that site and easy to run locally via GitHub source.
> Integrate the OpenJDK JMH micro benchmark framework for micro benchmarks and
> performance comparisons and investigation.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-15428
> URL: https://issues.apache.org/jira/browse/SOLR-15428
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Mark Robert Miller
> Priority: Major
>
> I’ve spent a fair amount of time over the years on work around integrating
> Lucene’s benchmark framework into Solr and while I’ve used this with
> additional local work off and on, JMH has become somewhat of a standard for
> micro benchmarks on the JVM. I have some work that provides an initial
> integration, allowing for more targeted micro benchmarks as well as more
> integration type benchmarking using JettySolrRunner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
