Has the behavior changed after an upgrade or has it been consistent since the start?
Ismael On Thu, Jan 2, 2020 at 4:18 PM Navneeth Krishnan <reachnavnee...@gmail.com> wrote: > Hi All, > > We have a kafka cluster with 12 nodes and we are pretty much seeing 90% > cpu usage on all the nodes. Here is all the information. Need some help on > figuring out what the problem is and how to overcome this issue. > > *Cluster:* > Kafka version: 2.3.0 > Number of brokers in cluster: 12 > Node type: 4 vCores 32GB mem > Network In: 10Mbps per broker > Network Out: 16Mbps per broker > Topics: 10 (approximately) > Partitions: 20 (Max), some has only partitions > Replication Factor: 3 > > *CPU Usage:* > [image: image.png] > > *VMStat* > > [root]# vmstat 1 10 > > procs -----------memory---------- ---swap-- -----io---- -system-- > ------cpu----- > > r b swpd free buff cache si so bi bo in cs us sy id > wa st > > 8 0 0 234444 19064 24046980 0 0 17 2026 1 3 38 33 > 28 0 1 > > 7 0 0 256444 19036 24023880 0 0 768 0 64027 22708 44 > 40 16 0 1 > > 7 0 0 245356 19052 24034560 0 0 256 472 63509 23276 44 > 39 17 0 1 > > 7 0 0 235096 19052 24046616 0 0 0 0 62277 22516 46 > 38 15 0 1 > > 8 0 0 260548 19036 24020084 0 0 516 49888 62364 22894 43 > 38 18 0 1 > > 5 0 0 249232 19036 24030924 0 0 512 0 61022 24589 41 > 39 20 0 1 > > 6 0 0 238072 19036 24042512 0 0 1024 0 63358 23063 44 > 38 17 0 0 > > 5 0 0 262904 19052 24017972 0 0 0 440 63078 23499 46 > 37 17 0 1 > > 7 0 0 250324 19052 24030008 0 0 0 0 64615 22617 48 > 38 14 0 1 > > 6 0 0 237920 19052 24042372 0 0 1024 48900 63223 23029 42 > 40 18 0 1 > > > *IO Stat:* > > [root]# iostat -m > > Linux 4.14.72-73.55.amzn2.x86_64 (loc-kafka11.internal.dnaspaces.io) > 01/02/2020 _x86_64_ (4 CPU) > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 38.11 0.00 33.09 0.11 0.61 28.08 > > > > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn > > xvda 2.36 0.01 0.01 26760 43360 > > nvme0n1 0.00 0.00 0.00 2 0 > > xvdf 70.95 0.06 7.67 185908 25205338 > > *Top Kafka broker threads:* > [image: image.png] > > *Top 3:* > > "data-plane-kafka-network-thread-10-ListenerName(PLAINTEXT)-PLAINTEXT-0" > #60 prio=5 os_prio=0 tid=0x00007f8b1ab56000 nid=0x581f runnable > [0x00007f8a886ce000] > > "data-plane-kafka-network-thread-10-ListenerName(PLAINTEXT)-PLAINTEXT-2" > #62 prio=5 os_prio=0 tid=0x00007f8b1ab59000 nid=0x5821 runnable > [0x00007f8a6aefd000] > > "data-plane-kafka-network-thread-10-ListenerName(PLAINTEXT)-PLAINTEXT-1" > #61 prio=5 os_prio=0 tid=0x00007f8b1ab57800 nid=0x5820 runnable > [0x00007f8a885cd000] > > It doesn't looks like GC and IO is the problem. > > Thanks >
