Yes, and I did not set the max message size properly on the Spout. From: Nathan Leung [mailto:[email protected]] Sent: Tuesday, May 20, 2014 10:43 AM To: user Subject: Re: Kafka Spout 0.8-plus stops consuming messages after a while
Hi Jing, Was message.max.bytes changed in your Kafka server config to be higher than the default value (1000000 bytes)? -Nathan On Mon, May 19, 2014 at 5:54 PM, Tao, Jing <[email protected]<mailto:[email protected]>> wrote: I finally found the root cause. Turns out the spout was reading a message exceeded the max message size. After increasing the message size in SpoutConfig, it worked. It would have been nice if the Spout threw an error or exception in such cases. Jing On May 16, 2014, at 12:09 PM, "Tao, Jing" <[email protected]<mailto:[email protected]>> wrote: Hi, I am using storm 0.9.0.1, kafka 0.8, and storm-kafka-0.8-plus. In LocalCluster, after fetching and processing about 18,000 messages, the spout stops fetching from Kafka even though there are new messages in the queue. There are no errors in the storm or kafka logs. Any idea why this is happening? Thanks, Jing
