Hi Malith,
I'll analyze results with different number of threads.
The "Error" is related to "Score". What you have thought is correct. It's
shows the variability of the throughput. I don't know how the error is
calculated, but I'll check how that is calculated.
In JMH, we can have warm-up iterations and measurement iterations. We can
also specify how many forks we need. In the benchmark results I have
specified, 3 forks, 5 warm-up iterations and 10 measurement iterations. The
results for warm-up iterations are ignored. That's why you see 30 as the
count ("Cnt"). i.e. 3 forks x 10 measurement iterations.
We can run this benchmark in "SampleTime/sample" mode and then we can get
latency results.
I'm working on getting more results now. I'll share when it's ready.
Thanks!
Best Regards,
On Fri, Nov 11, 2016 at 9:16 PM, Malith Jayasinghe <[email protected]> wrote:
> Hi Isuru,
>
> Interesting results and findings and it would be interesting to see how
> these values vary under different conditions (e.g. number of threads). I
> notice that the error is quite high for Stormpot Blaze Pool (which has
> the highest throughput) . Any thoughts on how to interpret the "error"
> here ? (for example, does it reflect the variability of the throughput).
> When we bench mark the latency we want to see the min, max and latency
> percentiles etc. Is there a way to get this using JMH?
>
> On Fri, Nov 11, 2016 at 7:43 PM, Sanjeewa Malalgoda <[email protected]>
> wrote:
>
>> Thanks for bring this up isuru. Stack object pool is critical component
>> for API Manager.
>> We are using it to do two most critical operations (key validation and
>> throttle data process and publish) in our gateway. Also we are using it for
>> both blocking and non blocking operations. So improving this component will
>> surely help us. Please keep us update with your findings. When we implement
>> key validation for next version of API Manager we can add this improvement.
>>
>> Thanks,
>> sanjeewa.
>>
>> On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[email protected]> wrote:
>>
>>> Hi
>>>
>>> I wrote $subject using JMH and I committed the code to my GitHub repo
>>> [1].
>>>
>>> Currently API Manager uses a Stack Object Pool for key validation
>>> clients and we noticed that there are lot contentions when trying to use
>>> the pool. Therefore we thought of writing a benchmark to compare the
>>> performance with other Object Pool implementations.
>>>
>>> Following are the results after running the benchmark with 100 threads.
>>> The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar
>>> -t 100 -f 3 -wi 5 -i 10"
>>>
>>> # Run complete. Total time: 00:12:28
>>>
>>> Benchmark (poolSize)
>>> Mode Cnt Score Error Units
>>> CommonsPool2GenericObjectPoolBenchmark.useObject 100
>>> thrpt 30 2470759.779 ± 204309.885 ops/s
>>> CommonsPool2SoftReferenceObjectPoolBenchmark.useObject 100
>>> thrpt 30 1305244.745 ± 57823.313 ops/s
>>> CommonsPoolGenericObjectPoolBenchmark.useObject 100
>>> thrpt 30 2277105.922 ± 40957.282 ops/s
>>> CommonsPoolSoftReferenceObjectPoolBenchmark.useObject 100
>>> thrpt 30 11163239.940 ± 449147.426 ops/s
>>> FastObjectPoolBenchmark.useObject 100
>>> thrpt 30 30164548.113 ± 984247.958 ops/s
>>> FuriousObjectPoolBenchmark.useObject 100
>>> thrpt 30 14279182.642 ± 115644.976 ops/s
>>> StackObjectPoolBenchmark.useObject 100
>>> thrpt 30 5464597.797 ± 135726.389 ops/s
>>> StormpotBlazePoolBenchmark.useObject 100
>>> thrpt 30 77752385.827 ± 3015457.000 ops/s
>>> StormpotQueuePoolBenchmark.useObject 100
>>> thrpt 30 7670529.272 ± 283981.943 ops/s
>>> TestObjectBenchmark.expensiveObjectCreate N/A
>>> thrpt 30 101870.001 ± 2872.885 ops/s
>>> TestObjectBenchmark.simpleObjectCreate N/A
>>> thrpt 30 917400188.941 ± 20593977.714 ops/s
>>> ViburObjectPoolBenchmark.useObject 100
>>> thrpt 30 5924114.964 ± 281625.445 ops/s
>>>
>>>
>>> For this benchmark, Stormpot Blaze Pool [2] has the highest throughput
>>> and it performs 14 times better than the Stack Object Pool, which is used
>>> in API Manager.
>>>
>>> Stormpot is also Apache licensed.
>>>
>>> When I was writing the benchmarks, I found out that Stormpot author also
>>> has written similar benchmarks [3]. But I continued with my project and
>>> benchmark code has more implementations. I also use latest versions of all
>>> object pool implementations.
>>>
>>> I'm writing a blog about this and planning to get results for different
>>> scenarios. I need to analyze latency results as well. I'll share those
>>> results in this mail thread.
>>>
>>> Thanks!
>>>
>>> Best Regards,
>>>
>>> [1] https://github.com/chrishantha/microbenchmarks/tree/master/o
>>> bjectpools
>>> [2] https://github.com/chrisvest/stormpot
>>> [3] https://github.com/chrisvest/object-pool-benchmarks
>>>
>>> --
>>> Isuru Perera
>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
>>> Lean . Enterprise . Middleware
>>>
>>> about.me/chrishantha
>>> Contact: +IsuruPereraWSO2
>>> <https://www.google.com/+IsuruPereraWSO2/about>
>>>
>>
>>
>>
>> --
>>
>> *Sanjeewa Malalgoda*
>> WSO2 Inc.
>> Mobile : +94713068779
>>
>> <http://sanjeewamalalgoda.blogspot.com/>blog
>> :http://sanjeewamalalgoda.blogspot.com/
>> <http://sanjeewamalalgoda.blogspot.com/>
>>
>>
>>
>> _______________________________________________
>> Architecture mailing list
>> [email protected]
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Malith Jayasinghe
>
> WSO2, Inc. (http://wso2.com)
> Email : [email protected]
> Mobile : 0770704040
> Lean . Enterprise . Middleware
>
--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware
about.me/chrishantha
Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about>
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
