Hi all, I published results in a medium story: https://medium.com/@chrishantha/benchmarking-object-pools-6df007a31ada
Even though I started writing the benchmarks in last year, I couldn't visualize the results and demonstrate the exact problem we faced with WSO2 API Manager. I started learning Python few months back and I could finally write the Python scripts to visualize results in CSV files. I could prove the issue with Stack Object Pool by using the GC profiler in JMH and visualizing the GC allocation rates. Please let me know if you have any feedback on this. Thank you. On Tue, Nov 15, 2016 at 9:57 AM, Isuru Perera <[email protected]> wrote: > Hi Malith, > > I'll analyze results with different number of threads. > > The "Error" is related to "Score". What you have thought is correct. It's > shows the variability of the throughput. I don't know how the error is > calculated, but I'll check how that is calculated. > > In JMH, we can have warm-up iterations and measurement iterations. We can > also specify how many forks we need. In the benchmark results I have > specified, 3 forks, 5 warm-up iterations and 10 measurement iterations. The > results for warm-up iterations are ignored. That's why you see 30 as the > count ("Cnt"). i.e. 3 forks x 10 measurement iterations. > > We can run this benchmark in "SampleTime/sample" mode and then we can get > latency results. > > I'm working on getting more results now. I'll share when it's ready. > > Thanks! > > Best Regards, > > On Fri, Nov 11, 2016 at 9:16 PM, Malith Jayasinghe <[email protected]> > wrote: > >> Hi Isuru, >> >> Interesting results and findings and it would be interesting to see how >> these values vary under different conditions (e.g. number of threads). I >> notice that the error is quite high for Stormpot Blaze Pool (which has >> the highest throughput) . Any thoughts on how to interpret the "error" >> here ? (for example, does it reflect the variability of the throughput). >> When we bench mark the latency we want to see the min, max and latency >> percentiles etc. Is there a way to get this using JMH? >> >> On Fri, Nov 11, 2016 at 7:43 PM, Sanjeewa Malalgoda <[email protected]> >> wrote: >> >>> Thanks for bring this up isuru. Stack object pool is critical component >>> for API Manager. >>> We are using it to do two most critical operations (key validation and >>> throttle data process and publish) in our gateway. Also we are using it for >>> both blocking and non blocking operations. So improving this component will >>> surely help us. Please keep us update with your findings. When we implement >>> key validation for next version of API Manager we can add this improvement. >>> >>> Thanks, >>> sanjeewa. >>> >>> On Fri, Nov 11, 2016 at 3:55 PM, Isuru Perera <[email protected]> wrote: >>> >>>> Hi >>>> >>>> I wrote $subject using JMH and I committed the code to my GitHub repo >>>> [1]. >>>> >>>> Currently API Manager uses a Stack Object Pool for key validation >>>> clients and we noticed that there are lot contentions when trying to use >>>> the pool. Therefore we thought of writing a benchmark to compare the >>>> performance with other Object Pool implementations. >>>> >>>> Following are the results after running the benchmark with 100 threads. >>>> The command I used is "java -jar target/objectpools-0.0.1-SNAPSHOT.jar >>>> -t 100 -f 3 -wi 5 -i 10" >>>> >>>> # Run complete. Total time: 00:12:28 >>>> >>>> Benchmark (poolSize) >>>> Mode Cnt Score Error Units >>>> CommonsPool2GenericObjectPoolBenchmark.useObject 100 >>>> thrpt 30 2470759.779 ± 204309.885 ops/s >>>> CommonsPool2SoftReferenceObjectPoolBenchmark.useObject 100 >>>> thrpt 30 1305244.745 ± 57823.313 ops/s >>>> CommonsPoolGenericObjectPoolBenchmark.useObject 100 >>>> thrpt 30 2277105.922 ± 40957.282 ops/s >>>> CommonsPoolSoftReferenceObjectPoolBenchmark.useObject 100 >>>> thrpt 30 11163239.940 ± 449147.426 ops/s >>>> FastObjectPoolBenchmark.useObject 100 >>>> thrpt 30 30164548.113 ± 984247.958 ops/s >>>> FuriousObjectPoolBenchmark.useObject 100 >>>> thrpt 30 14279182.642 ± 115644.976 ops/s >>>> StackObjectPoolBenchmark.useObject 100 >>>> thrpt 30 5464597.797 ± 135726.389 ops/s >>>> StormpotBlazePoolBenchmark.useObject 100 >>>> thrpt 30 77752385.827 ± 3015457.000 ops/s >>>> StormpotQueuePoolBenchmark.useObject 100 >>>> thrpt 30 7670529.272 ± 283981.943 ops/s >>>> TestObjectBenchmark.expensiveObjectCreate N/A >>>> thrpt 30 101870.001 ± 2872.885 ops/s >>>> TestObjectBenchmark.simpleObjectCreate N/A >>>> thrpt 30 917400188.941 ± 20593977.714 ops/s >>>> ViburObjectPoolBenchmark.useObject 100 >>>> thrpt 30 5924114.964 ± 281625.445 ops/s >>>> >>>> >>>> For this benchmark, Stormpot Blaze Pool [2] has the highest throughput >>>> and it performs 14 times better than the Stack Object Pool, which is used >>>> in API Manager. >>>> >>>> Stormpot is also Apache licensed. >>>> >>>> When I was writing the benchmarks, I found out that Stormpot author >>>> also has written similar benchmarks [3]. But I continued with my project >>>> and benchmark code has more implementations. I also use latest versions of >>>> all object pool implementations. >>>> >>>> I'm writing a blog about this and planning to get results for different >>>> scenarios. I need to analyze latency results as well. I'll share those >>>> results in this mail thread. >>>> >>>> Thanks! >>>> >>>> Best Regards, >>>> >>>> [1] https://github.com/chrishantha/microbenchmarks/tree/master/o >>>> bjectpools >>>> [2] https://github.com/chrisvest/stormpot >>>> [3] https://github.com/chrisvest/object-pool-benchmarks >>>> >>>> -- >>>> Isuru Perera >>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >>>> Lean . Enterprise . Middleware >>>> >>>> about.me/chrishantha >>>> Contact: +IsuruPereraWSO2 >>>> <https://www.google.com/+IsuruPereraWSO2/about> >>>> >>> >>> >>> >>> -- >>> >>> *Sanjeewa Malalgoda* >>> WSO2 Inc. >>> Mobile : +94713068779 >>> >>> <http://sanjeewamalalgoda.blogspot.com/>blog >>> :http://sanjeewamalalgoda.blogspot.com/ >>> <http://sanjeewamalalgoda.blogspot.com/> >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Malith Jayasinghe >> >> WSO2, Inc. (http://wso2.com) >> Email : [email protected] >> Mobile : 0770704040 >> Lean . Enterprise . Middleware >> > > > > -- > Isuru Perera > Associate Technical Lead | WSO2, Inc. | http://wso2.com/ > Lean . Enterprise . Middleware > > about.me/chrishantha > Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about> > -- Isuru Perera Technical Lead | WSO2, Inc. | http://wso2.com/ Lean . Enterprise . Middleware about.me/chrishantha Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
