[ 
https://issues.apache.org/jira/browse/KAFKA-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434325#comment-16434325
 ] 

ASF GitHub Bot commented on KAFKA-6611:
---------------------------------------

guozhangwang opened a new pull request #4854: KAFKA-6611, PART II: Improve 
Streams SimpleBenchmark
URL: https://github.com/apache/kafka/pull/4854
 
 
   1. SimpleBenchmark:
   
       1.a Do not rely on manual num.records / bytes collection on atomic 
integers.
       1.b Rely on config files for num.threads, bootstrap.servers, etc.
       1.c Add parameters for key skewness and value size.
       1.d Refactor the tests for loading phase, adding tumbling-windowed count.
       1.e For consumer / consumeproduce, collect metrics on consumer instead.
       1.f Force stop the test after 3 minutes, this is based on empirical 
numbers of 10M records.
   
   2. Other tests: use config for kafka bootstrap servers.
   
   3. streams_simple_benchmark.py: only use scale 1 for system test, remove 
yahoo from benchmark tests.
   
   Note that the JMX based metrics is more accurate than the manually collected 
metrics. For example:
   
   ```
   Module: kafkatest.benchmarks.streams.streams_simple_benchmark_test
   Class:  StreamsSimpleBenchmarkTest
   Method: test_simple_benchmark
   Arguments:
   {
     "scale": 3,
     "test": "streamcount"
   }
   ----
   
   {
     "Streams Count Performance [records/latency/rec-sec/MB-sec counted]0": " 
3691724/180042/20504.79332600171/20.64996886468678\n",
     "Streams Count Performance [records/latency/rec-sec/MB-sec counted]1": " 
3337273/180037/18536.595255419717/18.667835797999594\n",
     "Streams Count Performance [records/latency/rec-sec/MB-sec counted]2": " 
2971003/180029/16502.913419504635/16.61975533580484\n",
     "jmx-avg0": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 473.9851851851854,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.03802973633453574,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 1.6337178935425791,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 41.49027662118575,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.0021335594038200878,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 21042.80038005481
     },
     "jmx-avg1": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 354.0805555555556,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.030462783608341253,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 2.2233941132541286,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 35.82136735561879,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.002402570337120884,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 19100.53692641491
     },
     "jmx-avg2": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 365.85555555555555,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.03052173041795845,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 1.1700956516025365,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 28.06739395694291,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.0016847557478484937,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 15594.917035789254
     },
     "jmx-max0": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 547.0,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.06657567990413102,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 13.04,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 196.4955857339561,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.04672288495817908,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 99520.53241178217
     },
     "jmx-max1": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 653.0,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.065242211710977,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 13.909090909090908,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 156.47243937161267,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.08381330685203575,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 83263.21836209696
     },
     "jmx-max2": {
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-latency-avg":
 632.0,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:commit-rate":
 0.06459531038046638,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-latency-avg":
 8.25,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:poll-rate":
 173.80159786950733,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-latency-avg":
 0.07433900760593988,
       
"kafka.streams:type=stream-metrics,client-id=simple-benchmark-StreamThread-1:process-rate":
 97828.82822902796
     }
   }
   
   ```
   
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Re-write simple benchmark in system tests with JMXTool
> ------------------------------------------------------
>
>                 Key: KAFKA-6611
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6611
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>            Priority: Major
>             Fix For: 1.2.0
>
>
> The current SimpleBenchmark is recording the num.records actively in order to 
> compute throughput and latency, which introduces extra cost plus is less 
> accurate for benchmarking purposes; instead, it's better to use JmxMixin with 
> SimpleBenchmark to record metrics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to