Hi all, I've been attempting to benchmark Kafka KRaft version for an ApacheCon talk and have identified 2 problems:
1 - it's still impossible to create large number of partitions/topics - I can create more than the comparable Zookeeper version but still not "millions" - this is with RF=1 (as anything higher needs huge clusters to cope with the replication CPU overhead) only, and no load on the clusters yet (i.e. purely a topic/partition creation experiment). 2 - eventually the topic/partition creation command causes the Kafka process to fail - looks like a memory error - java.lang.OutOfMemoryError: Metaspace OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f4f554f9000, 65536, 1) failed; error='Not enough space' (errno=12) or similar error seems to happen consistently around 30,000+ partitions - this is on a test EC2 instance with 32GB Ram, 500,000 file descriptors (increased from default) and 64GB disk (plenty spare). I'm not an OS expert, but the kafka process and the OS both seem to have plenty of RAM when this error occurs. So there's 3 questions really: What's going wrong exactly? How to achieve more partitions? And should the topic create command (just using the CLI at present to create topics) really be capable of killing the Kafka instance, or should it fail and throw an error, and the Kafka instance still continue working... Regards, Paul Brebner
