Please find the storm-UI screenshot as follows. http://imgur.com/FhIrGFd
On Sat, Apr 22, 2017 at 1:41 AM, Ali Nazemian <[email protected]> wrote: > Hi Casey, > > - topology.message.timeout: It was 30s at first. I have increased it to > 300s, no changes! > - It is a very basic geo-enrichment and simple rule for threat triage! > - No, not at all. > - I have changed that to find the best value. it is 5000 which is about to > 5MB. > - I have changed the number of executors for the Storm acker thread, and I > have also changed the value of topology.max.spout.pending, still no changes! > > On Sat, Apr 22, 2017 at 1:24 AM, Casey Stella <[email protected]> wrote: > >> Also, >> * what's your setting for topology.message.timeout? >> * You said you're seeing this in indexing and enrichment, what >> enrichments do you have in place? >> * Is ES being taxed heavily? >> * What's your ES batch size for the sensor? >> >> On Fri, Apr 21, 2017 at 10:46 AM, Casey Stella <[email protected]> >> wrote: >> >>> So you're seeing failures in the storm topology but no errors in the >>> logs. Would you mind sending over a screenshot of the indexing topology >>> from the storm UI? You might not be able to paste the image on the mailing >>> list, so maybe an imgur link would be in order. >>> >>> Thanks, >>> >>> Casey >>> >>> On Fri, Apr 21, 2017 at 10:34 AM, Ali Nazemian <[email protected]> >>> wrote: >>> >>>> Hi Ryan, >>>> >>>> No, I cannot see any error inside the indexing error topic. Also, the >>>> number of tuples is emitted and transferred to the error indexing bolt is >>>> zero! >>>> >>>> On Sat, Apr 22, 2017 at 12:29 AM, Ryan Merriman <[email protected]> >>>> wrote: >>>> >>>>> Do you see any errors in the error* index in Elasticsearch? There are >>>>> several catch blocks across the different topologies that transform errors >>>>> into json objects and forward them on to the indexing topology. If you're >>>>> not seeing anything in the worker logs it's likely the errors were >>>>> captured >>>>> there instead. >>>>> >>>>> Ryan >>>>> >>>>> On Fri, Apr 21, 2017 at 9:19 AM, Ali Nazemian <[email protected]> >>>>> wrote: >>>>> >>>>>> No everything is fine at the log level. Also, when I checked resource >>>>>> consumption at the workers, there had been plenty resources still >>>>>> available! >>>>>> >>>>>> On Fri, Apr 21, 2017 at 10:04 PM, Casey Stella <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Seeing anything in the storm logs for the workers? >>>>>>> >>>>>>> On Fri, Apr 21, 2017 at 07:41 Ali Nazemian <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> After I tried to tune the Metron performance I have noticed the >>>>>>>> rate of failure for the indexing/enrichment topologies are very high >>>>>>>> (about >>>>>>>> 95%). However, I can see the messages in Elasticsearch. I have tried to >>>>>>>> increase the timeout value for the acknowledgement. It didn't fix the >>>>>>>> problem. I can set the number of acker executors to 0 to temporarily >>>>>>>> fix >>>>>>>> the problem which is not a good idea at all. Do you have any idea what >>>>>>>> have >>>>>>>> caused such issue? The percentage of failure decreases by reducing the >>>>>>>> number of parallelism, but even without any parallelism, it is still >>>>>>>> high! >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Ali >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> A.Nazemian >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> A.Nazemian >>>> >>> >>> >> > > > -- > A.Nazemian > -- A.Nazemian
