Hi Tomasz, There are some configurations needed to be tuned in producer side. Try searching kafka producer tuning, you should find many good articles describing it.
Good luck. Luke On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński < tomasz.sacin...@linuxpolska.pl> wrote: > Hello users. > > I test max throughput to a single kafka broker but results do not match > speed limits disk or network. > > /kafka-producer-perf-test.sh --topic perf-test --num-records 3000 > --record-size 1000000 --throughput -1 --producer-props > bootstrap.servers=localhost:9092 > > 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg > latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms > 99.9th. > > Smaller record size gets lower throughput. > > Is upper limit throughput for broker or something another limit throughput? > > What is bottleneck for this? > > There are detailed information: > > hardware: > CPU: 6230R 2.1GHz 2x26core, > RAM: 192 GB, > disks SATA SSD – 20 x1,92T (test run on single disk) > > > Java version: > java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64 > > > Kafka version: ./kafka-topics.sh --version > > 3.3.1 (Commit:e23c59d00e687ff5) > > kafka_2.13-3.3.1.jar > > Network speed tested by iperf3: 5,46 GB/s (testing kafka both localhost and > second node) > Disk speed: > > hdparm -tT /dev/mapper/vg1-kafka > > /dev/mapper/vg1-kafka: > > Timing cached reads: 19176 MB in 2.00 seconds = 9600.40 MB/sec > > Timing buffered disk reads: 10338 MB in 3.00 seconds = 3445.37 MB/sec > > > > writes: > > dd if=/dev/zero of=/kafka/test1.img bs=1G count=1 oflag=dsync > > 1+0 records in > > 1+0 records out > > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.13398 s, 947 MB/s > > Testing both on default jvm settings and with 6GB RAM: > > -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M > -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 > -XX:+ExplicitGCInvokesConcurrent > > Kafka installed and ran from instructions > https://kafka.apache.org/quickstart on a single node. > > > Pozdrawiam/Regards > > > > Tomasz Sąciński > > -- > <https://linuxpolska.pl/> > > > Linux Polska Sp. z o.o. > > Al. Jerozolimskie > 100, 00-807 Warszawa > < > https://www.google.com/maps/place/Linux+Polska+Sp.+z+o.+o./@52.221986,20.9744718,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x5756147639e870d6!8m2!3d52.221986!4d20.9766605?authuser=2 > > > > tel. +48 22 213 95 71, fax +48 22 213 96 71 > > KRS 00000326158, Sąd Rejonowy > dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS > > Kapitał > zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601 > > > > www.linuxpolska.pl <https://linuxpolska.pl/> | > <https://www.linkedin.com/company/linux-polska/> > <https://www.facebook.com/linuxpolskapl> > <https://twitter.com/linuxpolska> > > _________________________________________________________________________ > > > > > This message may contain confidential information that is covered by > legal privilege. If you are not the intended recipient or if you have > received this message by mistake, please notify the sender immediately and > delete this e-mail and its attachments from your system. Any unauthorized > copying, disclosure or distribution of the material in this e-mail and its > attachments is strictly forbidden. > > > -- > <https://linuxpolska.pl/baner-lp/> >
