Hi I have a Realtime data crunching pipeline which wherein storm consumes data from kafka, we do some processing on this data (basically counts and aggregations) and store the results into a MySql database.( I use the default Kafka spout.)
Over the weekend some network related issues led to MySQL server crash, as a result storm kept re-sending the messages, the processing in the storm supervisors increased drastically and CPU usage hit 100%. This in turn led to some supervisors falling out of the cluster and trying to restart. How would I be able to *handle* such an unforeseen situation? Is there any way through which storm would understand that there is a MySql server issue and stop re-sending data? Is there anyway that this exception can be caught and in *topology be deactivated* for a while? Regards, Chitra
