Thanks for the correctness, Stephan. For the micro benchmark, I think it should be better to make a new maven project inside Flink.
Thanks Chengxiang -----Original Message----- From: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] On Behalf Of Stephan Ewen Sent: Tuesday, September 22, 2015 8:14 PM To: dev@flink.apache.org Subject: Re: [Proposal] Create a separate sub module for benchmark test Sounds like a nice idea! Do you want to make this a new maven project as part of the Flink repository, or create a dedicated repository for that? BTW: We are currently not mixing microbenchmarks with test execution. The code for these benchmarks resides in the test scope of the projects (so it is not packaged), but it is not executed as part of the UnitTests or IntegrationTests. Greetings, Stephan On Tue, Sep 22, 2015 at 12:22 PM, Li, Chengxiang <chengxiang...@intel.com> wrote: > Hi, folks > During work on Flink, I found several micro benchmarks which come from > different modules, these benchmarks measure on manual, annotated with > Junit annotations, so they got executed during unit test as well. > There are some shortage on current implementation: > > 1. Benchmark test performance instead of feature, and normally, it > takes much more time than unit test. Mixed benchmark with unit test > would expand the CI check time. > > 2. With mixed with other tests, no warm up, no standalone process... > these benchmarks result may not very accurate. > Although looks easy, there are actually many pitfalls about benchmark, > so I suggest we create a new sub module for all benchmark test, and > import JMH( > http://openjdk.java.net/projects/code-tools/jmh/) as the benchmark > framework. With the help of JMH, we should get: > > 1. More available metrics. > > 2. More accurate result. > > 3. Focus on benchmark logic only, no need to worry about measure > logic. > > Thanks > Chengxiang >
