wsry commented on issue #11088: [FLINK-16012][runtime] Reduce the default number of buffers per channel from 2 to 1 URL: https://github.com/apache/flink/pull/11088#issuecomment-591752289 I did some further performance tests using the micro benchmark with changes in https://github.com/apache/flink/pull/11155 and the results are as follows. From the results we can see that if we reduce the buffer per channel of both upstream and downstream, then we need almost equivalent number of floating buffers to compensate the reduced ones to mitigate the performance regression. If we reduce the exclusive buffer of downstream only and keeps upstream unchanged, there is no visible regression and the performance of the fast flush cases (1ms flushTimeout) becomes even better, IMO, the reason is that less credit prevent the upstream from sending too many small buffers to downstream which can increase the TPS. It seems that the performance regression mainly comes from upstream. Why upstream is that sensitive to the reduction of buffers? IMO, it is because that our micro benchmark case is a little special, more specifically, it can request a large amount of buffers in a really short time when the first round of buffers is full. Because we use a round robin partitioner and send long record, so all the subpartitions will be filled up almost at the same time and will request next buffer at the same time, that is, requesting a large number of buffers in a really short time which in result leads to the sensibility. What do you think? Should we reduce the number of buffer statically? @pnowojski @zhijiangW **settings: 2 buffers per outgoing channel/2 buffer per incoming buffer/8 floating buffer** _round 1#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 18206.419 ± 533.180 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 25378.527 ± 747.900 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27937.504 ± 553.818 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7600.337 ± 1264.666 ops/ms ``` _round 2#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17277.088 ± 925.827 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24673.112 ± 3202.755 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27124.349 ± 1001.959 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6991.067 ± 497.079 ops/ms ``` _round 3#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 16575.334 ± 856.037 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 25445.706 ± 751.636 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27552.088 ± 761.005 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7328.127 ± 336.441 ops/ms ``` **settings: 1 buffers per outgoing channel/1 buffer per incoming buffer/8 floating buffer** _round 1#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 18317.961 ± 937.388 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 19269.037 ± 2387.532 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 22981.454 ± 2860.808 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7236.713 ± 302.389 ops/ms ``` _round 2#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 19105.135 ± 575.752 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 19845.468 ± 2697.928 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 22150.188 ± 953.592 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6960.408 ± 901.595 ops/ms ``` _round 3#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17473.902 ± 1233.335 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 19616.336 ± 743.647 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 21883.304 ± 2610.933 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6560.487 ± 1349.803 ops/ms ``` **settings: 1 buffers per outgoing channel/1 buffer per incoming buffer/508 floating buffer** _round 1#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17930.067 ± 990.492 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24189.003 ± 532.873 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 26015.059 ± 827.189 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6614.732 ± 1127.513 ops/ms ``` _round 2#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 16874.222 ± 1501.294 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 23835.932 ± 676.615 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 25974.407 ± 739.122 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6942.174 ± 860.100 ops/ms ``` _round 3#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17459.868 ± 2264.240 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 22951.319 ± 430.990 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 25847.325 ± 1076.960 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7060.140 ± 770.352 ops/ms ``` **1 buffers per outgoing channel/1 buffer per incoming buffer/1008 floating buffer** _round 1#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17528.332 ± 2490.253 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24555.332 ± 696.186 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27949.803 ± 523.245 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7055.183 ± 833.954 ops/ms ``` _round 2#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17191.842 ± 2327.408 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 25676.208 ± 2993.290 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27522.535 ± 695.564 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7096.016 ± 434.702 ops/ms ``` _round 3#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 15859.857 ± 2824.002 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 25809.716 ± 3054.423 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 28080.639 ± 653.612 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6841.500 ± 1163.974 ops/ms ``` **If we only reduce the exclusive buffer of downstream and keep the upstream unchanged:** **2 buffers per outgoing channel/1 buffer per incoming buffer/8 floating buffer** _round 1#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 18823.216 ± 1014.368 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24954.098 ± 924.354 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 26578.547 ± 570.707 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6903.210 ± 300.143 ops/ms ``` _round 2#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 16434.840 ± 1156.454 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 26084.775 ± 938.799 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27116.358 ± 983.822 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6627.874 ± 1159.660 ops/ms ``` _round 3#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 18914.084 ± 1073.938 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24259.843 ± 2971.212 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27398.183 ± 826.757 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6510.502 ± 1115.598 ops/ms ``` _round 4#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17077.960 ± 1069.516 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 24807.979 ± 3180.461 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 27735.840 ± 717.570 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 7248.710 ± 334.596 ops/ms ``` _round 5#_ ``` Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units DataSkewStreamNetworkThroughputBenchmarkExecutor.networkSkewedThroughput N/A thrpt 30 17437.883 ± 706.215 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,1ms thrpt 30 26347.947 ± 866.912 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms thrpt 30 28161.395 ± 819.070 ops/ms StreamNetworkThroughputBenchmarkExecutor.networkThroughput 1000,100ms,SSL thrpt 30 6785.905 ± 1137.939 ops/ms ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
