Hi,
We are running a 5 node cluster(2x large ec2) with below config. Topology
is consuming message from SQS and writing to RDS and S3. Even if there are
no bolt failures , we are seeing many spout failures.
Can you please help in checking the config.Also i am setting tasks as
parallelism count * 2. Is this fine?
TOPOLOGY_NAME=MDP_STORM_PRD
MARIA_BOLT_PARALLELISM=50
S3_BOLT_PARALLELISM=50
SQS_DELETE_BOLT_PARALLELISM=100
SPOUT_PARALLELISM=50
NUMBER_OF_WORKERS=5
NUMBER_OF_ACKERS=5
SPOUT_MAX_PENDING=5000
MESSAGE_TIMEOUT_SECONDS=240
Topology Code
=============
Config config = new Config();
config.setNumWorkers(numWorkers);
config.setDebug(false);
config.setNumAckers(numAckers);
config.setMaxSpoutPending(maxSpoutPending);
config.setMessageTimeoutSecs(messageTimeoutSecs);
topologyBuilder.setSpout(spoutId, new
SQSMessageReaderSpout(sqsUtils.getSQSUrl(dataQueue), properties),
spoutParallelism).*setNumTasks(spoutParallelism * TWO);*
topologyBuilder.setBolt(mariaBoltId, new MariaDbBolt(properties),
mariaBoltParallelism)
.*setNumTasks(mariaBoltParallelism * TWO)*.fieldsGrouping(spoutId,
new Fields(MESSAGE_ID));
topologyBuilder.setBolt(s3BoltId, new S3WriteBolt(properties,
s3Properties), s3BoltParallelism)
.*setNumTasks(s3BoltParallelism * TWO).*
shuffleGrouping(mariaBoltId);
topologyBuilder
.setBolt(sqsDeleteBoltId, new
SQSMessageDeleteBolt(sqsUtils.getSQSUrl(dataQueue)), sqsBoltParallelism)
.*setNumTasks(sqsBoltParallelism * TWO)*.shuffleGrouping(s3BoltId);
StormSubmitter.submitTopology(topologyName, config,
topologyBuilder.createTopology());
Regards
Pradeep S
