I have ever met the similar problem before, but if I only use kafka spout and print bolt, it runs smoothly without any error. i notice you are using storm-kafka-client, so you are using the kafka spout written by yourself? if yes, try kafka spout provided by Storm. And I am not sure how much of your data volume. if your data volume is large, increase the maxspoutpending value and try to use localOrShuffleGrouping between kafka spout and print bolt to reduce network pressure.
Regard, Junfeng Chen On Thu, Sep 29, 2016 at 3:17 PM, [email protected] < [email protected]> wrote: > Hi Junfeng, > > Actually, the topology just has two components, one is kafkaspout, and the > other is the a print-bolt. > > the bolt does nothing but print the tuple message itself got. > > Can you suggest what kind of config will raise suche a problem? I will > check the kafka-spout config. > > ------------------------------ > Joshua > > > > *From:* Junfeng Chen <[email protected]> > *Date:* 2016-09-29 15:05 > *To:* user@storm apache. org <[email protected]> > *Subject:* Re: Re: Urgent help! kafka-spout stops fetching data after > running for a while > I suggest you to run the topology containing kafka spout only, aiming to > find out if it is the kafka config problem. If kafka spout is running well, > it means your bolts drag the whole topology. > > Regard, > Junfeng Chen > > On Thu, Sep 29, 2016 at 2:52 PM, [email protected] < > [email protected]> wrote: > >> Hi Junfeng： >> >> The usage of the worker seems ok too, the %cpu is about 10, and the %MEM >> is about 5. >> >> ------------------------------ >> Joshua >> >> >> >> *From:* Junfeng Chen <[email protected]> >> *Date:* 2016-09-29 13:41 >> *To:* user@storm apache. org <[email protected]> >> *Subject:* Re: Re: Urgent help! kafka-spout stops fetching data after >> running for a while >> Did you check the cpu usage of processes which run the kafkaspout. >> >> Regard, >> Junfeng Chen >> >> On Thu, Sep 29, 2016 at 12:46 PM, [email protected] < >> [email protected]> wrote: >> >>> Hi Junfeng ： >>> >>> Yes, I checked the worker log, it seems ok, no error or warning is >>> printed. >>> >>> ------------------------------ >>> Joshua >>> >>> >>> >>> *From:* Junfeng Chen <[email protected]> >>> *Date:* 2016-09-29 11:29 >>> *To:* user@storm apache. org <[email protected]> >>> *Subject:* Re: Urgent help! kafka-spout stops fetching data after >>> running for a while >>> Have you checked your worker log file? Frequent worker accident may >>> cause kafka spout not working. >>> >>> Regard, >>> Junfeng Chen >>> >>> On Thu, Sep 29, 2016 at 10:54 AM, [email protected] < >>> [email protected]> wrote: >>> >>>> Hi user, >>>> >>>> My kafka-spout is based on storm-kafka-client. >>>> >>>> It can read existed data from kafka. >>>> >>>> It also can read data that I sent to kafka when the topology just runs >>>> for a while. >>>> >>>> But when there is no data, the topology remains processing nothing for >>>> a long while, like 1 hour, I begin to send data to the topic, the topology >>>> just can not read the data. >>>> >>>> I read the mails and find a way to set topology.backpressure.enable to >>>> false. >>>> >>>> But the problem remains. >>>> >>>> Great apprication for any advice. >>>> >>>> ------------------------------ >>>> Joshua >>>> 2016-09-29 10:43:11 >>>> >>>> >>>> >>> >> >
