[
https://issues.apache.org/jira/browse/STORM-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009462#comment-15009462
]
ASF GitHub Bot commented on STORM-1215:
---------------------------------------
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/888#issuecomment-157501592
OK I got a heap dump and when debug is enabled we are logging faster then
the logger can write. This is resulting in the async logger disruptor queue
starting to fill up, and making lots of GC problems. Can we disable this for
the worker process or find a way to configure it so the memory usage is not
unbounded?
> Use Async Loggers to avoid locking and logging overhead
> --------------------------------------------------------
>
> Key: STORM-1215
> URL: https://issues.apache.org/jira/browse/STORM-1215
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Kishor Patil
> Assignee: Kishor Patil
>
> The loggers are synchronous with immediateFlush to disk, making some of the
> daemons slow down. In some other cases, nimbus is slow too with submit-lock.
> Making loggers asynchronous with no necessity to write to disk on every
> logger event would improve cpu resource usage for logging.
> {code}
> "pool-7-thread-986" #1025 prio=5 os_prio=0 tid=0x00007f0f9628c800 nid=0x1b84
> runnable [0x00007f0f0fa2a000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> - locked <0x00000003c00ae520> (a java.io.BufferedOutputStream)
> at java.io.PrintStream.write(PrintStream.java:482)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
> at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
> - locked <0x00000003c00ae640> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
> at java.io.PrintStream.write(PrintStream.java:527)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at java.io.PrintStream.print(PrintStream.java:669)
> at java.io.PrintStream.println(PrintStream.java:806)
> - locked <0x00000003c00ae500> (a java.io.PrintStream)
> at
> org.apache.logging.log4j.status.StatusConsoleListener.log(StatusConsoleListener.java:81)
> at
> org.apache.logging.log4j.status.StatusLogger.logMessage(StatusLogger.java:218)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:727)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:716)
> at
> org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:344)
> at
> org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:59)
> at
> org.apache.logging.log4j.core.appender.AbstractAppender.error(AbstractAppender.java:86)
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:99)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:430)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:409)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:367)
> at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:727)
> at
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:716)
> at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:198)
> at clojure.tools.logging$eval1$fn__7.invoke(NO_SOURCE_FILE:0)
> at clojure.tools.logging.impl$fn__28$G__8__39.invoke(impl.clj:16)
> at clojure.tools.logging$log_STAR_.invoke(logging.clj:59)
> at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:781)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
