Hi guys, I’m a new user of storm and I need help for an issue I have with storm.
At a glance, I have a topology with three spouts and bolts ; - Spout -> get messages from Kafka - Bolt -> transform the message from Kafka to a well formed son - Bolt -> send the JSON to a kairosDB server All of these bolts/spouts are written in go by using the library https://github.com/jsgilmore/gostorm Unfortunately after using the topology for a while, the CPU usage is just growing and growing and never goes down. For instance after sending 1 million messages, the CPU stays at 200% of cpu usage. I tried to play with the configuration and get better result ; with the following configuration : conf.put(Config.TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS, 1000); conf.put(Config.TOPOLOGY_DISRUPTOR_BATCH_SIZE, 10); conf.put(Config.TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS, 500); conf.put(Config.TOPOLOGY_DISRUPTOR_WAIT_TIMEOUT_MILLIS, 500); conf.setMaxSpoutPending(1000000); conf.put(Config.TASK_HEARTBEAT_FREQUENCY_SECS , 600); My CPU usage after 1 million messages will be 50%, but it never goes down... What could be the issue(s) and how can I debug that ? I’m pretty new to the java/clojure world as well... Im running on storm 1.0.1 fyi Thanks in advance, -- Alexandre Wilhelm
