Can you check on UI that the bolt is sending tuples on the metric stream? You have to click on "show system stats" on UI. On Oct 6, 2015 2:34 PM, "Stepan Urban" <[email protected]> wrote:
> Hi, > I tried to increase parallelism of bolts and create some performance > optimizations with no effect. Metrics stop logging after few hours. Is > there some mechanism that Storm stops logging when bolts are overloaded? > Any other sugesstions? > > Thanks > Stepan > > 2015-10-01 19:39 GMT+02:00 Stepan Urban <[email protected]>: > >> 1. All metrics are not being logged >> 2. Yes topology is still prgressing, everything other works as normal, >> worker logs are still running >> 3. Bolt is slower, it writes data to disk, do some simple preprocessing >> and send them to another bolt, spout only receive it and send it to bolt. >> >> Thanks >> Stepan >> >> 2015-10-01 13:19 GMT+02:00 Abhishek Agarwal <[email protected]>: >> >>> Please check the following - >>> 1. Are all the metrics not being logged or is it just the custom >>> metrics? Are the in-built system metrics being logged? >>> 2. When the metrics are not getting logged, is the topology still >>> progressing? >>> 3. Is your bolt too slow compared to the spout? >>> >>> On Thu, Oct 1, 2015 at 3:14 PM, Stepan Urban <[email protected]> >>> wrote: >>> >>>> Hi Ahishek, >>>> thanks for the response. Registring metrics is normal: >>>> transient MultiCountMetric countMetric; >>>> transient MultiReducedMetric meanMetric; >>>> >>>> countMetric = new MultiCountMetric(); >>>> meanMetric = new MultiReducedMetric(new MeanReducer()); >>>> context.registerMetric("count", countMetric, 1); >>>> context.registerMetric("mean", meanMetric, 1); >>>> >>>> and >>>> Config conf = new Config(); >>>> conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 1); >>>> >>>> even with default cluster.xml it stops logging on production computer. >>>> >>>> Yesterday I found that everything work on dev computer with data loaded >>>> from file. But on production computer, when data are receided from net it >>>> stops logging after few hours. Is it possible to find if some thread is not >>>> working properly using jstack? Or any other way how to debug it? >>>> >>>> Thanks >>>> Stepan >>>> >>>> >>>> >>>> 2015-10-01 6:26 GMT+02:00 Anishek Agarwal <[email protected]>: >>>> >>>>> From what i have found dont remove/add appenders in cluster.xml. not >>>>> sure about why it works for a few hours and then stops. any specific way >>>>> you are registering log metrics, is it possible to put some code out ? >>>>> >>>>> On Tue, Sep 29, 2015 at 9:14 PM, Stepan Urban <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> I am using LoggingMetricsConsumer and MultiReducedMetric to count >>>>>> several metrics in Storm topology. After few hours of running it stops >>>>>> logging these metrics into metrics.log file without an error in any log >>>>>> file. Any sugestions how to find where is the problem? First I used >>>>>> default >>>>>> cluster.xml with increased size of files, now I use only >>>>>> ch.qos.logback.core.FileAppender with the same results. After few mb of >>>>>> logs it stops logging. >>>>>> >>>>>> Thanks >>>>>> Stepan >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Abhishek Agarwal >>> >>> >> >
