Denny, How to do use direct memory ?
Thanks, Senthil On Wed, Oct 10, 2012 at 7:25 PM, Denny Ye <[email protected]> wrote: > It might be caused by : > 1. Less heap memory. Increasing '-Xms -Xmx' options > 2. Disable that prompt. It's not recommended. Using > '-XX:-UseGCOverheadLimit' option > 3. There always have too many GC limitation with memory channel. I have > met some case with you before. I have tried to use direct memory for > storing event body out of heap in the past, that's effective method and no > any GC happened. I think you can test it by yourself. > > -Regards > Denny Ye > > 2012/10/11 Brock Noland <[email protected]> > >> Yep, sounds like: agent heap size < (capacity * avg event size) >> >> Brock >> >> On Wed, Oct 10, 2012 at 8:15 PM, Harish Mandala <[email protected]> >> wrote: >> > Hi, >> > >> > In flume-env.sh, please add >> > >> > JAVA_OPTS="-Xms128m -Xmx256m" >> > >> > (Or whatever amount of memory works for you. I have Xmx set to 4g) >> > >> > Regards, >> > Harish >> > >> > >> > On Wed, Oct 10, 2012 at 9:08 PM, Camp, Roy <[email protected]> wrote: >> >> >> >> I ran into the following error – I had to restart flume to recover. >> Do I >> >> just need to adjust a GC setting of some sort or is there a larger >> issue >> >> here? >> >> >> >> >> >> >> >> Source: thriftLegacy >> >> >> >> Channel: memory (capacity: 1,000,000) >> >> >> >> Sink: avro >> >> >> >> >> >> >> >> >> >> >> >> Exception in thread "pool-4-thread-2" java.lang.OutOfMemoryError: GC >> >> overhead limit exceeded >> >> >> >> at >> >> >> java.util.concurrent.LinkedBlockingDeque.<init>(LinkedBlockingDeque.java:108) >> >> >> >> at >> >> >> org.apache.flume.channel.MemoryChannel$MemoryTransaction.<init>(MemoryChannel.java:49) >> >> >> >> at >> >> >> org.apache.flume.channel.MemoryChannel.createTransaction(MemoryChannel.java:264) >> >> >> >> at >> >> >> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) >> >> >> >> at >> >> >> org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:260) >> >> >> >> at >> >> >> org.apache.flume.source.thriftLegacy.ThriftLegacySource$ThriftFlumeEventServerImpl.append(ThriftLegacySource.java:96) >> >> >> >> at >> >> >> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:276) >> >> >> >> at >> >> >> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:256) >> >> >> >> at >> >> >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >> >> >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> >> >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >> >> >> at java.lang.Thread.run(Thread.java:679) >> >> >> >> Exception in thread >> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Exception in thread "Avro NettyTransceiver I/O Worker 1" >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Oct 10, 2012 5:51:39 PM >> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker >> >> >> >> WARNING: Unexpected exception in the selector loop. >> >> >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Oct 10, 2012 5:52:35 PM >> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker >> >> >> >> WARNING: Unexpected exception in the selector loop. >> >> >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-2" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "pool-4-thread-4" java.lang.OutOfMemoryError: GC >> >> overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-0" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-10" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Oct 10, 2012 5:52:39 PM >> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker >> >> >> >> WARNING: Unexpected exception in the selector loop. >> >> >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Oct 10, 2012 5:52:39 PM >> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker >> >> >> >> WARNING: Unexpected exception in the selector loop. >> >> >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-9" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-1" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-5" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-3" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-13" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-6" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-4" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-5" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Exception in thread "lifecycleSupervisor-1-8" >> java.lang.OutOfMemoryError: >> >> GC overhead limit exceeded >> >> >> >> Oct 10, 2012 5:52:48 PM >> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker >> >> >> >> WARNING: Unexpected exception in the selector loop. >> >> >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> Roy >> >> >> >> >> >> >> >> >> > >> > >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - >> http://incubator.apache.org/mrunit/ >> > > -- ..Senthil "If there's anything more important than my ego around, I want it caught and shot now." - Douglas Adams.
