lhotari commented on issue #24695: URL: https://github.com/apache/pulsar/issues/24695#issuecomment-3249249055
The problem persists even with `maxPendingPublishRequestsPerConnection=1000`, after the consumption rate drops, it's never able to catch up. ``` ==> consume.1756905609.txt <== 2025-09-03T13:20:21,423+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 2480116 msg --- 247985.562 msg/s --- 242.174 Mbit/s --- Latency: mean: 475.180 ms - med: 364 - 95pct: 929 - 99pct: 1015 - 99.9pct: 1033 - 99.99pct: 1034 - Max: 1035 2025-09-03T13:20:31,616+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 5344177 msg --- 281067.409 msg/s --- 274.479 Mbit/s --- Latency: mean: 1477.922 ms - med: 1232 - 95pct: 2123 - 99pct: 2165 - 99.9pct: 2181 - 99.99pct: 2183 - Max: 2184 2025-09-03T13:20:41,837+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 8538541 msg --- 311929.958 msg/s --- 304.619 Mbit/s --- Latency: mean: 2242.693 ms - med: 2167 - 95pct: 2552 - 99pct: 2599 - 99.9pct: 2628 - 99.99pct: 2631 - Max: 2632 2025-09-03T13:20:52,033+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 11428523 msg --- 283930.574 msg/s --- 277.276 Mbit/s --- Latency: mean: 3200.968 ms - med: 3154 - 95pct: 3902 - 99pct: 3953 - 99.9pct: 3965 - 99.99pct: 3967 - Max: 3967 2025-09-03T13:21:02,220+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 13963339 msg --- 248431.573 msg/s --- 242.609 Mbit/s --- Latency: mean: 4840.201 ms - med: 4631 - 95pct: 5953 - 99pct: 6088 - 99.9pct: 6504 - 99.99pct: 6606 - Max: 6614 2025-09-03T13:21:12,389+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 14082039 msg --- 11675.276 msg/s --- 11.402 Mbit/s --- Latency: mean: 10595.487 ms - med: 9834 - 95pct: 15946 - 99pct: 16330 - 99.9pct: 16397 - 99.99pct: 16405 - Max: 16405 2025-09-03T13:21:22,574+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 14207139 msg --- 12281.006 msg/s --- 11.993 Mbit/s --- Latency: mean: 21423.869 ms - med: 21433 - 95pct: 25576 - 99pct: 25915 - 99.9pct: 26079 - 99.99pct: 26202 - Max: 26202 2025-09-03T13:21:32,766+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 14333239 msg --- 12396.715 msg/s --- 12.106 Mbit/s --- Latency: mean: 31015.429 ms - med: 31071 - 95pct: 35475 - 99pct: 35822 - 99.9pct: 35940 - 99.99pct: 35957 - Max: 35957 2025-09-03T13:21:42,948+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 14453639 msg --- 11787.760 msg/s --- 11.511 Mbit/s --- Latency: mean: 40793.841 ms - med: 40847 - 95pct: 45110 - 99pct: 45620 - 99.9pct: 45741 - 99.99pct: 45753 - Max: 45753 2025-09-03T13:21:53,113+0000 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 14574539 msg --- 11913.761 msg/s --- 11.635 Mbit/s --- Latency: mean: 50616.966 ms - med: 50630 - 95pct: 54988 - 99pct: 55331 - 99.9pct: 55410 - 99.99pct: 55419 - Max: 55419 ==> produce.1756905610.txt <== 2025-09-03T13:20:21,707+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 2909305 msg --- 290893.0 msg/s --- 284.1 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 61.583 ms - med: 59.997 - 95pct: 79.221 - 99pct: 94.191 - 99.9pct: 123.506 - 99.99pct: 126.534 - Max: 126.836 2025-09-03T13:20:31,973+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 6040341 msg --- 305160.6 msg/s --- 298.0 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 65.233 ms - med: 60.888 - 95pct: 96.654 - 99pct: 157.045 - 99.9pct: 197.298 - 99.99pct: 201.863 - Max: 205.311 2025-09-03T13:20:42,300+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 9542250 msg --- 339691.4 msg/s --- 331.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 58.848 ms - med: 57.397 - 95pct: 72.225 - 99pct: 82.660 - 99.9pct: 136.446 - 99.99pct: 138.024 - Max: 138.419 2025-09-03T13:20:52,537+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 12864236 msg --- 324643.1 msg/s --- 317.0 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 61.325 ms - med: 60.040 - 95pct: 73.703 - 99pct: 94.623 - 99.9pct: 137.981 - 99.99pct: 141.839 - Max: 142.164 2025-09-03T13:21:02,851+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 15862437 msg --- 290223.8 msg/s --- 283.4 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 68.900 ms - med: 63.146 - 95pct: 99.333 - 99pct: 229.158 - 99.9pct: 314.035 - 99.99pct: 316.513 - Max: 317.015 2025-09-03T13:21:13,092+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 19015246 msg --- 306440.0 msg/s --- 299.3 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 65.178 ms - med: 62.359 - 95pct: 95.343 - 99pct: 142.081 - 99.9pct: 170.086 - 99.99pct: 174.757 - Max: 176.125 2025-09-03T13:21:23,417+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 22157356 msg --- 305512.9 msg/s --- 298.4 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 65.361 ms - med: 62.668 - 95pct: 78.804 - 99pct: 120.910 - 99.9pct: 263.391 - 99.99pct: 266.913 - Max: 268.609 2025-09-03T13:21:33,712+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 25430649 msg --- 318750.0 msg/s --- 311.3 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 62.615 ms - med: 61.110 - 95pct: 78.980 - 99pct: 102.190 - 99.9pct: 109.717 - 99.99pct: 110.994 - Max: 112.107 2025-09-03T13:21:44,034+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 28480015 msg --- 295567.7 msg/s --- 288.6 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 67.564 ms - med: 64.679 - 95pct: 91.043 - 99pct: 151.943 - 99.9pct: 200.440 - 99.99pct: 201.588 - Max: 201.954 2025-09-03T13:21:54,333+0000 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 31505193 msg --- 293281.1 msg/s --- 286.4 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 67.970 ms - med: 65.604 - 95pct: 82.670 - 99pct: 126.416 - 99.9pct: 248.036 - 99.99pct: 248.534 - Max: 248.685 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
