jiazhai opened a new issue #195: perf tools performance about 1/10 of java client URL: https://github.com/apache/pulsar-client-go/issues/195 >Here are the details with the different perf tools. I have seen that the go perf tool that is provided along with the go sdk client is only able to push 6-7k msg/sec while the pulsar-perf tool provided and written in java seems to get at least 10x the performance. Even stranger is if I launch 2 or 3 go clients producing on the same topics it cuts the through put in half on the existing client running. That would make sense if there was a bottleneck in pulsar recieving the message but it appears the java client can push much more through the pulsar pipe. Is this an issue where the do SDK is some how more a more limited citizen when sending messages to pulsar? Here is an example of both clients running inside the bastion pod alonside pulsar. ``` pulsar@itomdipulsar-bastion-856887989b-7k4mq:/pulsar> ./bin/pulsar-perf produce -r 100000 -s 10 testtopic 20:01:43.750 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Starting Pulsar perf producer with config: { "confFile" : "/pulsar/conf/client.conf", "topics" : [ "testtopic" ], "numTestThreads" : 1, "msgRate" : 100000, "msgSize" : 10, "numTopics" : 1, "numProducers" : 1, "serviceURL" : "pulsar+ssl://itomdipulsar-proxy:6651/", "authPluginClassName" : "org.apache.pulsar.client.impl.auth.AuthenticationTls", "authParams" : "tlsCertFile:/var/run/secrets/boostport.com/server.crt,tlsKeyFile:/var/run/secrets/boostport.com/server.key", "maxOutstanding" : 1000, "maxPendingMessagesAcrossPartitions" : 50000, "maxConnections" : 100, "numMessages" : 0, "statsIntervalSeconds" : 0, "compression" : "NONE", "payloadFilename" : null, "payloadDelimiter" : "\\n", "batchTimeMillis" : 1.0, "batchMaxMessages" : 1000, "batchMaxBytes" : 4194304, "testTime" : 0, "warmupTimeSeconds" : 1.0, "tlsTrustCertsFilePath" : "/var/run/secrets/boostport.com/trustedCAs/RID_ca.crt", "encKeyName" : null, "encKeyFile" : null, "delay" : 0, "exitOnFailure" : false } 20:01:43.793 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Dumping latency stats to perf-producer-1584043303793.hgrm 20:01:43.794 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Started performance test thread 0 20:01:46.069 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Adding 1 publishers on topic testtopic 20:01:47.481 [pulsar-client-io-2-2] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x84022a1f, L:/172.16.56.93:38146 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651]] Connected to server 20:01:47.876 [pulsar-client-io-2-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x5ae5d2a9, L:/172.16.56.93:38148 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651]] Connected to server 20:01:49.209 [pulsar-client-io-2-4] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xfdc4651d, L:/172.16.56.93:38150 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651]] Connected to server 20:01:49.213 [pulsar-client-io-2-4] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xfdc4651d, L:/172.16.56.93:38150 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651] Connected through proxy to target broker at opsBsac0680Mast-02.swinfra.net:31412 20:01:49.305 [pulsar-client-io-2-4] INFO org.apache.pulsar.client.impl.ProducerImpl - [testtopic] [null] Creating producer on cnx [id: 0xfdc4651d, L:/172.16.56.93:38150 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651] 20:01:49.365 [pulsar-client-io-2-4] INFO org.apache.pulsar.client.impl.ProducerImpl - [testtopic] [itomdipulsar-2-4] Created producer on cnx [id: 0xfdc4651d, L:/172.16.56.93:38150 - R:itomdipulsar-proxy.opsbridge-mfarb.svc.cluster.local./172.17.17.229:6651] 20:01:49.410 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers 20:01:49.514 [pulsar-timer-5-1] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized 20:01:54.031 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 32471.4 msg/s --- 2.5 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 11.182 ms - med: 9.999 - 95pct: 15.537 - 99pct: 22.719 - 99.9pct: 187.701 - 99.99pct: 194.075 - Max: 196.387 20:02:04.228 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 48718.9 msg/s --- 3.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 20.190 ms - med: 14.525 - 95pct: 43.281 - 99pct: 71.753 - 99.9pct: 723.523 - 99.99pct: 747.219 - Max: 747.995 20:02:14.330 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 57713.4 msg/s --- 4.4 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 16.856 ms - med: 12.245 - 95pct: 28.740 - 99pct: 44.836 - 99.9pct: 758.999 - 99.99pct: 759.291 - Max: 774.999 20:02:24.427 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 49198.4 msg/s --- 3.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 19.845 ms - med: 12.062 - 95pct: 35.037 - 99pct: 67.693 - 99.9pct: 1755.903 - 99.99pct: 1757.647 - Max: 1760.007 20:02:34.499 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 58616.9 msg/s --- 4.5 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 16.577 ms - med: 12.046 - 95pct: 27.967 - 99pct: 52.169 - 99.9pct: 466.015 - 99.99pct: 468.053 - Max: 483.407 ^C20:02:37.691 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 2657056 records sent --- 49276.666 msg/s --- 3.760 Mbit/s 20:02:37.760 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 17.548 ms - med: 12.284 - 95pct: 32.654 - 99pct: 55.543 - 99.9pct: 723.351 - 99.99pct: 1757.503 - 99.999pct: 1757.679 - Max: 1760.007 ``` ``` pulsar@itomdipulsar-bastion-856887989b-7k4mq:/pulsar> ./pulsar-client-go produce -u pulsar+ssl://itomdipulsar-proxy:6651 -r 100000 -s 10 testtopic INFO[20:02:57.859] Client config: { "ServiceURL": "pulsar+ssl://itomdipulsar-proxy:6651" } INFO[20:02:57.859] Producer config: { "Topic": "testtopic", "Rate": 100000, "BatchingTimeMillis": 1, "MessageSize": 10, "ProducerQueueSize": 1000 } INFO[20:02:57.860] Connecting to broker remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:57.908] TCP connection established local_addr="172.16.56.93:38270" remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:57.920] Connection is ready local_addr="172.16.56.93:38270" remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:59.159] Connecting to broker remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:59.207] TCP connection established local_addr="172.16.56.93:38272" remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:59.259] Connection is ready local_addr="172.16.56.93:38272" remote_addr="pulsar+ssl://itomdipulsar-proxy:6651" INFO[20:02:59.272] Created producer cnx="172.16.56.93:38272 -> 172.17.17.229:6651" producer_name=itomdipulsar-2-5 topic="persistent://public/default/testtopic" INFO[20:03:09.273] Stats - Publish rate: 5875.4 msg/s - 0.4 Mbps - Latency ms: 50% 156.0 -95% 351.4 - 99% 728.9 - 99.9% 728.9 - max 728.9 INFO[20:03:19.274] Stats - Publish rate: 7483.8 msg/s - 0.6 Mbps - Latency ms: 50% 108.9 -95% 253.4 - 99% 896.8 - 99.9% 896.8 - max 896.8 INFO[20:03:29.273] Stats - Publish rate: 7067.7 msg/s - 0.5 Mbps - Latency ms: 50% 146.5 -95% 239.1 - 99% 313.4 - 99.9% 313.4 - max 313.4 INFO[20:03:39.279] Stats - Publish rate: 6512.7 msg/s - 0.5 Mbps - Latency ms: 50% 121.4 -95% 360.9 - 99% 952.7 - 99.9% 952.7 - max 952.7 ``` >Wanted to point out the perf tool I wrote using the sync method I was able to produce something close to 40K msg/sec on a small 100 byte message but with async method and using the code provided in the new go perf tool even after modifying to use 20 topics in parallel I have only been able to achieve 25K msg/sec with a 100 byte message. I will be out of the office tomorrow but would be happy to walk through more of these detail live with someone.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
