liyafan82 edited a comment on issue #8397: [FLINK-11421][Table SQL/Runtime]Add compilation options to allow comp… URL: https://github.com/apache/flink/pull/8397#issuecomment-493441719 > > I think JMH benchmark component is a further thing. It's under discussion. > > For this PR, I think you can do a simpler benchmark rather than a complicate component. > > You can dependent https://github.com/dataArtisans/flink-benchmarks or other things, just let us reproduce your benchmark easier. > > (you can also include testing for vector computation in this local benchmarks) > > Hi @JingsongLi and @KurtYoung , > > I have tried to provide a local benchmark, as you suggested. Unfortunately, I could not reproduce the results in the cluster environment when the operators were run in jobs. > > So let me show you what I did, and what results I had. > > I setup a JMH project (I believe this is a reliable way to conduct performance evaluations), and use randomly generated data to test the Calc operator method above. > > The results are as follows: > > 1. Compilation time: > > Compilation time for JCA: 713ms > Compilation time for JANINO: 267ms > > The compilation time for JCA is OK, but the time for Janino is not right. In the job environment, compilation by Janino never exceeds 100ms. This result can be reproduced consistently. > > 2. Processing time: > > The general performance improvements of JCA is about 2% to 7%, which is smaller than the 10% observed in the job environment. A typical report is given below: > > Benchmark Mode Cnt Score Error Units > CompilationBenchmark.testJCA avgt 5 94611.913 ± 868.944 us/op > CompilationBenchmark.testJanino avgt 5 100778.892 ± 1367.429 us/op > > If you are interested, please use this [branch](https://github.com/liyafan82/flink/tree/fly_11421_test) to reproduce the results. > > Anyway, I do not think it is a good idea to try to reproduce the results in a local machine. This is because, it is extremely difficult to simulate the environment in the cluster when a job is running. The following factors can be different from the local environment: > > 1. In the job environment, the CPU cache may be polluted by other processes (like HDFS, Yarn), or threads from the same process (like Task Manager, Job Manager, source threads, etc.) > 2. In the job environment, there are many concurrent worker threads processing the same operator and accessing the same resources. > 3. The JVM has been running for a while in the job environment (it has been running since the session is started, or even the cluster is started), so the JIT can be in a different state from the local JIT. > > In my opinion, the performance in the job environment matters the most, because that is what we want ultimately. I guess you must have such experiences: a improvement well-tested locally does not perform as expected in the production environment, and vice versa. > > So I think what we have is sufficient to prove the merit of this issue. If you are still in doubt, I sincerely invite you to take a look in our environment.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
