Hi All,
We have setup our storm cluster to process 2500 messaged per second.
Unfortunately, we are not to able the throughput as expected.
Most of time either we get "GC overhead limit exceeded" or "Too many
failed tuples".
Below the server configuration and Storm configuration.
1. One Nimbus server.
2. 5 supervisor with 2 slots (workers) each.
*Storm Configuration:*
Config conf = new Config();
conf.setNumWorkers(10);
conf.setMaxSpoutPending(80000);
conf.setMaxTaskParallelism(6);
conf.put(RichSpoutBatchExecutor.MAX_BATCH_SIZE_CONF, 64 * 1024);
*Kafka spout configuration:*
kafkaConfig.bufferSizeBytes = 1024*1024*4;
kafkaConfig.fetchSizeBytes = 1024*1024*4;
kafkaConfig.forceFromStart = false;
Kafka cluster is running with partition of 2.
*Topology Configuration:*
Spout: With parallelism_hint 5
Bolt1: With parallelism_hint 6
Bolt2: With parallelism_hint 5
Bolt3, Bolt4, Bolt5 and Bolt5: With parallelism_hint 3
Storm.yaml:
supervisor.slots.ports:
- 6700
- 6701
supervisor.childopts: "-Xmx1024m"
worker.childopts: "-Xmx2048m"
topology.message.timeout.secs: 30
Please help me to solve this problem.
Thanks in advance.
--
Regards,
Riyaz
