sijie commented on issue #5036: [pulsar-perf] Introduce number of threads in perf producer program URL: https://github.com/apache/pulsar/pull/5036#issuecomment-524747094 I am able to push the throughput to the limit when 2~3 threads on my laptop. 1 thread: ``` bin/pulsar-perf produce -r 1000000 -s 1024 -o 10000 -threads 1 test-topic ``` result: ``` 00:12:00.464 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 194654.9 msg/s --- 1520.7 Mbit/s --- Latency: mean: 11.303 ms - med: 6.485 - 95pct: 45.137 - 99pct: 68.027 - 99.9pct: 92.922 - 99.99pct: 96.613 - Max: 97.172 00:12:10.532 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 236724.4 msg/s --- 1849.4 Mbit/s --- Latency: mean: 10.805 ms - med: 5.296 - 95pct: 27.571 - 99pct: 99.669 - 99.9pct: 350.171 - 99.99pct: 351.947 - Max: 352.309 00:12:20.653 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 254950.9 msg/s --- 1991.8 Mbit/s --- Latency: mean: 8.138 ms - med: 4.476 - 95pct: 20.667 - 99pct: 38.882 - 99.9pct: 297.731 - 99.99pct: 301.597 - Max: 302.335 00:12:30.678 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 240353.1 msg/s --- 1877.8 Mbit/s --- Latency: mean: 10.532 ms - med: 5.317 - 95pct: 28.446 - 99pct: 157.266 - 99.9pct: 326.325 - 99.99pct: 329.209 - Max: 329.797 00:12:40.705 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 253417.0 msg/s --- 1979.8 Mbit/s --- Latency: mean: 9.085 ms - med: 4.355 - 95pct: 22.979 - 99pct: 57.399 - 99.9pct: 331.955 - 99.99pct: 334.425 - Max: 334.823 00:12:50.730 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 219773.5 msg/s --- 1717.0 Mbit/s --- Latency: mean: 14.533 ms - med: 4.822 - 95pct: 44.289 - 99pct: 223.977 - 99.9pct: 279.707 - 99.99pct: 371.679 - Max: 372.513 00:13:00.768 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 258459.7 msg/s --- 2019.2 Mbit/s --- Latency: mean: 8.126 ms - med: 4.684 - 95pct: 22.018 - 99pct: 38.195 - 99.9pct: 314.801 - 99.99pct: 317.155 - Max: 317.725 00:13:10.793 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 239840.0 msg/s --- 1873.7 Mbit/s --- Latency: mean: 10.509 ms - med: 4.793 - 95pct: 26.751 - 99pct: 154.428 - 99.9pct: 355.123 - 99.99pct: 358.319 - Max: 358.541 00:13:20.817 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 229792.9 msg/s --- 1795.3 Mbit/s --- Latency: mean: 10.516 ms - med: 4.379 - 95pct: 22.080 - 99pct: 255.290 - 99.9pct: 361.119 - 99.99pct: 362.403 - Max: 362.819 00:13:30.837 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 205172.4 msg/s --- 1602.9 Mbit/s --- Latency: mean: 18.843 ms - med: 5.355 - 95pct: 137.392 - 99pct: 308.997 - 99.9pct: 340.997 - 99.99pct: 342.481 - Max: 342.645 00:13:40.880 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 239861.8 msg/s --- 1873.9 Mbit/s --- Latency: mean: 9.853 ms - med: 4.465 - 95pct: 23.468 - 99pct: 115.637 - 99.9pct: 381.907 - 99.99pct: 383.829 - Max: 384.193 00:13:50.901 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 255023.1 msg/s --- 1992.4 Mbit/s --- Latency: mean: 8.785 ms - med: 4.554 - 95pct: 24.120 - 99pct: 37.702 - 99.9pct: 373.729 - 99.99pct: 375.851 - Max: 376.201 00:14:00.923 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 226279.1 msg/s --- 1767.8 Mbit/s --- Latency: mean: 14.535 ms - med: 5.061 - 95pct: 111.297 - 99pct: 134.601 - 99.9pct: 204.177 - 99.99pct: 207.197 - Max: 207.292 00:14:10.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 210576.2 msg/s --- 1645.1 Mbit/s --- Latency: mean: 18.282 ms - med: 5.168 - 95pct: 131.245 - 99pct: 242.050 - 99.9pct: 513.205 - 99.99pct: 515.811 - Max: 516.071 00:14:21.022 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 210019.9 msg/s --- 1640.8 Mbit/s --- Latency: mean: 17.970 ms - med: 4.797 - 95pct: 134.668 - 99pct: 238.287 - 99.9pct: 369.191 - 99.99pct: 370.045 - Max: 370.143 00:14:31.074 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 212835.3 msg/s --- 1662.8 Mbit/s --- Latency: mean: 15.993 ms - med: 5.071 - 95pct: 113.417 - 99pct: 146.539 - 99.9pct: 175.021 - 99.99pct: 211.234 - Max: 211.405 00:14:41.122 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 236308.1 msg/s --- 1846.2 Mbit/s --- Latency: mean: 10.882 ms - med: 5.138 - 95pct: 27.675 - 99pct: 147.606 - 99.9pct: 376.239 - 99.99pct: 377.557 - Max: 377.921 ^C00:14:47.849 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 41109621 records sent --- 231697.861 msg/s --- 1810.140 Mbit/s ``` --- 2 threads ``` bin/pulsar-perf produce -r 1000000 -s 1024 -o 10000 -threads 2 test-topic ``` results: ``` 00:15:23.119 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 316726.0 msg/s --- 2474.4 Mbit/s --- Latency: mean: 19.858 ms - med: 11.179 - 95pct: 49.611 - 99pct: 353.107 - 99.9pct: 392.813 - 99.99pct: 397.361 - Max: 398.169 00:15:33.217 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 333110.4 msg/s --- 2602.4 Mbit/s --- Latency: mean: 30.618 ms - med: 11.730 - 95pct: 151.895 - 99pct: 288.625 - 99.9pct: 342.355 - 99.99pct: 343.199 - Max: 343.487 00:15:43.351 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 284353.9 msg/s --- 2221.5 Mbit/s --- Latency: mean: 43.095 ms - med: 13.030 - 95pct: 231.816 - 99pct: 351.051 - 99.9pct: 418.729 - 99.99pct: 421.891 - Max: 422.947 00:15:53.444 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 345963.8 msg/s --- 2702.8 Mbit/s --- Latency: mean: 29.335 ms - med: 11.466 - 95pct: 157.086 - 99pct: 346.247 - 99.9pct: 455.489 - 99.99pct: 460.563 - Max: 461.015 ^C00:16:02.459 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 15951187 records sent --- 322503.161 msg/s --- 2519.556 Mbit/s ``` --- 3 threads: ``` 00:16:37.867 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 407660.3 msg/s --- 3184.8 Mbit/s --- Latency: mean: 53.397 ms - med: 51.446 - 95pct: 90.427 - 99pct: 207.051 - 99.9pct: 456.429 - 99.99pct: 457.397 - Max: 457.847 00:16:47.981 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 327646.5 msg/s --- 2559.7 Mbit/s --- Latency: mean: 85.390 ms - med: 58.209 - 95pct: 347.853 - 99pct: 424.205 - 99.9pct: 516.749 - 99.99pct: 518.719 - Max: 519.165 00:16:58.068 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 356509.2 msg/s --- 2785.2 Mbit/s --- Latency: mean: 73.149 ms - med: 55.273 - 95pct: 259.298 - 99pct: 381.965 - 99.9pct: 439.005 - 99.99pct: 442.055 - Max: 442.421 00:17:08.206 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 323249.3 msg/s --- 2525.4 Mbit/s --- Latency: mean: 86.977 ms - med: 56.079 - 95pct: 268.905 - 99pct: 420.713 - 99.9pct: 487.285 - 99.99pct: 490.479 - Max: 491.387 00:17:18.317 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 417818.4 msg/s --- 3264.2 Mbit/s --- Latency: mean: 67.648 ms - med: 58.555 - 95pct: 96.054 - 99pct: 414.709 - 99.9pct: 441.251 - 99.99pct: 446.535 - Max: 447.139 00:17:28.391 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 382491.1 msg/s --- 2988.2 Mbit/s --- Latency: mean: 68.429 ms - med: 53.680 - 95pct: 161.993 - 99pct: 445.171 - 99.9pct: 532.123 - 99.99pct: 533.823 - Max: 535.819 ^C00:17:30.293 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 23220081 records sent --- 371416.556 msg/s --- 2901.692 Mbit/s ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
