SamBarker commented on PR #90:
URL: https://github.com/apache/flink-benchmarks/pull/90#issuecomment-2138478974
> It compiles & install flink before run benchmarks via
/mnt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/M3/bin/mvn
-Dflink.version=1.20-SNAPSHOT clean install exec:exec -Penable-async-profiler
-DskipTests -Drat.skip=true '-Dbenchmarks=org.apache.flink.state.benchmark.*'
'-DbenchmarkExcludes=org.apache.flink.benchmark.full.*'
-DexecutableJava=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.372.b07-4.0.2.al8.x86_64/bin/java
-DasyncProfilerLib=/opt/async-profiler-3.0-linux-x64/lib/libasyncProfiler.so
-DprofResultPath=./profile-result
>It would be great if you could provide some options to speed this up.
It looks to me like it was handling the excludes and includes correctly
which is what I was concerned about. I can't reproduce a slower runtime with
the profiler on
With profiler
```
mvn -Dflink.version=1.20-SNAPSHOT -DexecutableJava=${JAVA_HOME}/bin/java
-DasyncProfilerLib=/home/sam/src/async-profiler/async-profiler-3.0-linux-x64/lib/libasyncProfiler.so
-Dbenchmarks="SerializationFrameworkMiniBenchmarks.serializerHeavyString"
exec:exec --offline
...
snip
...
# JMH version: 1.37
# VM version: JDK 11.0.23, OpenJDK 64-Bit Server VM, 11.0.23+9
# VM invoker: /home/sam/.sdkman/candidates/java/11.0.23-tem/bin/java
# VM options: -Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.ssl
# Blackhole mode: full + dont-inline hint (auto-detected, use
-Djmh.blackhole.autoDetect=false to disable)
# Warmup: 10 iterations, 10 s each
# Measurement: 10 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark:
org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString
# Run progress: 0.00% complete, ETA 00:10:00
...
snip
...
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 11:17 min
[INFO] Finished at: 2024-05-30T12:10:42+12:00
[INFO]
------------------------------------------------------------------------
```
without profiler
```
mvn -Dflink.version=1.20-SNAPSHOT -DexecutableJava=${JAVA_HOME}/bin/java
-Dbenchmarks="SerializationFrameworkMiniBenchmarks.serializerHeavyString"
exec:exec --offline
...
snip
...
# JMH version: 1.37
# VM version: JDK 11.0.23, OpenJDK 64-Bit Server VM, 11.0.23+9
# VM invoker: /home/sam/.sdkman/candidates/java/11.0.23-tem/bin/java
# VM options: -Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.ssl
# Blackhole mode: full + dont-inline hint (auto-detected, use
-Djmh.blackhole.autoDetect=false to disable)
# Warmup: 10 iterations, 10 s each
# Measurement: 10 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark:
org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString
# Run progress: 0.00% complete, ETA 00:10:00
...
snip
...
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 11:18 min
[INFO] Finished at: 2024-05-30T12:27:06+12:00
[INFO]
------------------------------------------------------------------------
```
~So I'm not really sure what to suggest.~
Wait a minute. This PR changes the version of JMH. Its not the profiler
being ON or OFF that affects the runtime its the JMH version.
going back to main I get
```
# JMH version: 1.19
# VM version: JDK 11.0.23, VM 11.0.23+9
# VM invoker: /home/sam/.sdkman/candidates/java/11.0.23-tem/bin/java
# VM options: -Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.ssl
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark:
org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString
# Run progress: 0.00% complete, ETA 00:01:00
```
This may be related to changes in warmups in [JMH
1.21](https://mail.openjdk.org/pipermail/jmh-dev/2018-May/002753.html)
> *) Defaults for warmup/measurement/forks were reconsidered to allow longer
in-JVM runs. This is
important for the VMs/benchmarks where time-to-performance is substantial.
Instead of 20x1s warmup,
20x1s measurement, 10 forks, we are now doing 5x10s warmup, 5x10s
measurement, 5 forks. It increases
the default time for around 1.25x, and single fork has 2.5x more time to
warmup and complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
