Hi, I'm running a 9 node Kafka 0.10.2.1 cluster in AWS and 3-4 of the nodes experience high IO wait but others don't. In terms read/write load, the high IO nodes look identical to others. All nodes were at around 40% disk space usage. No errors or warns in logs.
Here is part of the output running iostat -kx 2 5 avg-cpu: %user %nice %system %iowait %steal %idle 17.99 0.00 8.87 18.38 0.26 54.50 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util xvda 0.00 5.00 0.00 2.50 0.00 30.00 24.00 0.00 0.00 0.00 0.00 xvdf 0.00 8.00 1.00 50.00 4.00 736.75 29.05 149.10 3128.82 19.61 100.00 The cluster is composed of r4.xlarge nodes with single 1TB st1 volume used for Kafka data (xvdf). According AWS docs a 1 TB st1 EBS volume should have 40 MiB/s base throughput. As you can see from above, its only writing at ~736 kB/s. The average write size is also very small at ~15 kB. Running iotop shows the following most of the time: TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 14321 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [kworker/u30:1] With flashes of the Kafka java process also taking up 99.99% just like kworker. Additional info: - There are about 60 topics total, 60 partitions each, all with replication factor 3. Topic/partition distribution is very uniform across nodes. - Total message ingest rate for cluster is only 2.2 MB/s message at ~8000 msg/s. Load is very uniform across nodes. - Some possibly relevant configs in server.properties: log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 num.io.threads=1 - I'm using XFS for file system with default configs on Amazon Linux. I have only tried to restart the high IO nodes so far but it doesn't seem to help. Also tried increasing num.io.threads, did not help. I'm a little skeptical of just increasing EBS volume size for more IOPS given the IO stats above. I'm fairly new to using Kafka so perhaps something is very wrong with my setup. If anyone has any suggestions it would be greatly appreciated. Thanks, Xiaochuan Yu