Hi Junfeng： Thanks for your help. Yes, I use a kafka-spout written by myself. I use storm-kafka-client because I want to implement the SSL and ACL with Kafka topic. Now, I just test the kafka-spout, and the volume is small.
Joshua From: Junfeng Chen Date: 2016-09-29 15:27 To: user@storm apache. org Subject: Re: Re: Urgent help! kafka-spout stops fetching data after running for a while I have ever met the similar problem before, but if I only use kafka spout and print bolt, it runs smoothly without any error. i notice you are using storm-kafka-client, so you are using the kafka spout written by yourself? if yes, try kafka spout provided by Storm. And I am not sure how much of your data volume. if your data volume is large, increase the maxspoutpending value and try to use localOrShuffleGrouping between kafka spout and print bolt to reduce network pressure. Regard, Junfeng Chen On Thu, Sep 29, 2016 at 3:17 PM, [email protected] <[email protected]> wrote: Hi Junfeng, Actually, the topology just has two components, one is kafkaspout, and the other is the a print-bolt. the bolt does nothing but print the tuple message itself got. Can you suggest what kind of config will raise suche a problem? I will check the kafka-spout config. Joshua From: Junfeng Chen Date: 2016-09-29 15:05 To: user@storm apache. org Subject: Re: Re: Urgent help! kafka-spout stops fetching data after running for a while I suggest you to run the topology containing kafka spout only, aiming to find out if it is the kafka config problem. If kafka spout is running well, it means your bolts drag the whole topology. Regard, Junfeng Chen On Thu, Sep 29, 2016 at 2:52 PM, [email protected] <[email protected]> wrote: Hi Junfeng： The usage of the worker seems ok too, the %cpu is about 10, and the %MEM is about 5. Joshua From: Junfeng Chen Date: 2016-09-29 13:41 To: user@storm apache. org Subject: Re: Re: Urgent help! kafka-spout stops fetching data after running for a while Did you check the cpu usage of processes which run the kafkaspout. Regard, Junfeng Chen On Thu, Sep 29, 2016 at 12:46 PM, [email protected] <[email protected]> wrote: Hi Junfeng ： Yes, I checked the worker log, it seems ok, no error or warning is printed. Joshua From: Junfeng Chen Date: 2016-09-29 11:29 To: user@storm apache. org Subject: Re: Urgent help! kafka-spout stops fetching data after running for a while Have you checked your worker log file? Frequent worker accident may cause kafka spout not working. Regard, Junfeng Chen On Thu, Sep 29, 2016 at 10:54 AM, [email protected] <[email protected]> wrote: Hi user, My kafka-spout is based on storm-kafka-client. It can read existed data from kafka. It also can read data that I sent to kafka when the topology just runs for a while. But when there is no data, the topology remains processing nothing for a long while, like 1 hour, I begin to send data to the topic, the topology just can not read the data. I read the mails and find a way to set topology.backpressure.enable to false. But the problem remains. Great apprication for any advice. Joshua 2016-09-29 10:43:11
