I get the log freezing even if I run the topology with 1 worker or if I run the topology in local cluster node.
On Tue, May 5, 2015 at 11:00 AM, Jeff Maass <[email protected]> wrote: > Here is one possibility: > > A worker will stop writing to its log if the supervisor kills it. As I > know it, this happens when one of 2 things occur: > * nimbus orders the worker shut down - after not hearing from an executor > or after a manual topology kill is run > * the supervisor doesn’t receive a heartbeat message from the worker > process. This appears to happen when the worker is resource constrained - > namely - CPU. > > In the second situation, very quickly after the worker is shutdown, > nimbus tells a supervisor ( maybe the same one as previously ) to start up > a worker, doing the same tasks. So, if you are watching the worker process > running, you could say, walk away for a cup of coffee, and the worker could > change from writing to worker_6702 to worker_6703. > > How to tell this has happened: > * the worker pid will have changed > * the worker’s communication port / node may have changed > * nimbus logs - if nimbus initiated the action and the logging level is > set to info > * the supervisor logs - if the logging level is set to info > * count and record the # of worker processes on your nodes > * the storm metric - new worker events - has not to me proved to be > accurate. I do not know why. > > From: clay teahouse <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: 2015,Tuesday, May 5 at 10:31 > To: "[email protected]" <[email protected]>, Bobby Evans < > [email protected]> > Subject: Re: storm logback freezing > > Hello, > I am sorry for not being specific. The worker log is freezing. There is > plenty of space on disk. I have made changes to logback config file, as > follows. > <configuration scan="true" scanPeriod="30 seconds"> > <appender name="A1" class="ch.qos.logback.core.ConsoleAppender"> > <encoder> > <pattern>%-4r [%t] %-5p %c - %m%n</pattern> > </encoder> > </appender> > > <appender name="A2" > class="ch.qos.logback.core.rolling.RollingFileAppender"> > <file>/usr/local/storm/log/storm.log</file> > <rollingPolicy > class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"> > > <fileNamePattern>/usr/local/storm/log/storm.log.%i.zip</fileNamePattern> > <minIndex>1</minIndex> > <maxIndex>5</maxIndex> > </rollingPolicy> > > <triggeringPolicy > class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> > <maxFileSize>10MB</maxFileSize> > </triggeringPolicy> > <append>true</append> > <encoder> > <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - > %msg%n</pattern> > </encoder> > </appender> > > <logger name="org.apache.zookeeper" level="WARN"/> > <root level="INFO"> > <appender-ref ref="A2"/> > </root> > > On Tue, May 5, 2015 at 8:34 AM, Bobby Evans <[email protected]> wrote: > >> The only time I have seen logback "freeze" was when the disk it was >> writing to filled up. And in that case some log messages started to block, >> and it was very painful. >> >> - Bobby >> >> >> >> On Tuesday, May 5, 2015 8:18 AM, Jeff Maass <[email protected]> wrote: >> >> >> there are at least 3 different logs everyone should have: >> * nimbus >> * worker_{port} >> * supervisor >> >> Which of these is “freezing”? >> >> What version storm? >> >> When a worker starts, does writing to its log work? >> >> Have you made any changes to your logback configuration? >> >> >> >> From: clay teahouse <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: 2015,Monday, May 4 at 20:21 >> To: "[email protected]" <[email protected]>, >> "[email protected]" <[email protected]> >> Subject: storm logback freezing >> >> Hi all, >> Has anyone experienced a case where storm logback freezes? The topology >> seems to be functioning without an issue (I can see the results, in the >> destination consumers), but the storm log shows no progress. This usually >> happens a couple of hours after the topology starts, and not right away. >> I'd appreciate any feedback that what may be causing this issue. >> >> thank you >> >> Clay >> >> >> >> >
