It's always usable with controlled 'close' method. Memory out of heap is more and more important in big-data era. Thus, Apache DirectMemory is kind of perfect solution to manage direct memory with user defined ways.
I will test the performance and stress both of creation and reclaim for direct memory, thanks Brock for your tips -Denny Ye 2012/10/11 Brock Noland <[email protected]> > The only issue with using Direct Memory is that it is not reclaimed > until the object is garbage collected. So it's possible to run out of > Direct Memory without actually even garbage collecting. DirectMemory > buffers have an undocumented cleaner object which can be called when > the object is no longer used to avoid running out of space. > > Example: > https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java#L62 > > Brock > > On Wed, Oct 10, 2012 at 9:40 PM, Denny Ye <[email protected]> wrote: > > A simple method like this : using special DirectEvent, it stores event > body > > from Source into direct bytebuffer with fixed body size. Thus, when this > > event has been consumed by Sink, the reference of that direct block out > of > > heap might have chance to be reclaimed(only have chance). > > > > This method may be limited by maximum direct memory, by default, this > size > > is equals with max heap size in HotSpot. Another focus is when that > direct > > bytebuffer storing event body can be reclaimed. While full GC is > running, it > > will check the direct memory usage and clear un-referenced direct block. > > > > We can tuning the direct memory allocation with pre-allocated direct > memory > > blocks. This is more complex than method mentioned above. > > Did I explain my thought clearly? > > > > -Regards > > Denny Ye > > > > 2012/10/11 Senthilvel Rangaswamy <[email protected]> > >> > >> Denny, > >> > >> How to do use direct memory ? > >> > >> Thanks, > >> Senthil > >> > >> > >> On Wed, Oct 10, 2012 at 7:25 PM, Denny Ye <[email protected]> wrote: > >>> > >>> It might be caused by : > >>> 1. Less heap memory. Increasing '-Xms -Xmx' options > >>> 2. Disable that prompt. It's not recommended. Using > >>> '-XX:-UseGCOverheadLimit' option > >>> 3. There always have too many GC limitation with memory channel. I have > >>> met some case with you before. I have tried to use direct memory for > storing > >>> event body out of heap in the past, that's effective method and no any > GC > >>> happened. I think you can test it by yourself. > >>> > >>> -Regards > >>> Denny Ye > >>> > >>> 2012/10/11 Brock Noland <[email protected]> > >>>> > >>>> Yep, sounds like: agent heap size < (capacity * avg event size) > >>>> > >>>> Brock > >>>> > >>>> On Wed, Oct 10, 2012 at 8:15 PM, Harish Mandala > >>>> <[email protected]> wrote: > >>>> > Hi, > >>>> > > >>>> > In flume-env.sh, please add > >>>> > > >>>> > JAVA_OPTS="-Xms128m -Xmx256m" > >>>> > > >>>> > (Or whatever amount of memory works for you. I have Xmx set to 4g) > >>>> > > >>>> > Regards, > >>>> > Harish > >>>> > > >>>> > > >>>> > On Wed, Oct 10, 2012 at 9:08 PM, Camp, Roy <[email protected]> wrote: > >>>> >> > >>>> >> I ran into the following error – I had to restart flume to recover. > >>>> >> Do I > >>>> >> just need to adjust a GC setting of some sort or is there a larger > >>>> >> issue > >>>> >> here? > >>>> >> > >>>> >> > >>>> >> > >>>> >> Source: thriftLegacy > >>>> >> > >>>> >> Channel: memory (capacity: 1,000,000) > >>>> >> > >>>> >> Sink: avro > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> Exception in thread "pool-4-thread-2" java.lang.OutOfMemoryError: > GC > >>>> >> overhead limit exceeded > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > java.util.concurrent.LinkedBlockingDeque.<init>(LinkedBlockingDeque.java:108) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.flume.channel.MemoryChannel$MemoryTransaction.<init>(MemoryChannel.java:49) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.flume.channel.MemoryChannel.createTransaction(MemoryChannel.java:264) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:260) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.flume.source.thriftLegacy.ThriftLegacySource$ThriftFlumeEventServerImpl.append(ThriftLegacySource.java:96) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:276) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:256) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >>>> >> > >>>> >> at > >>>> >> > >>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >>>> >> > >>>> >> at java.lang.Thread.run(Thread.java:679) > >>>> >> > >>>> >> Exception in thread > >>>> >> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "Avro NettyTransceiver I/O Worker 1" > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Oct 10, 2012 5:51:39 PM > >>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker > >>>> >> > >>>> >> WARNING: Unexpected exception in the selector loop. > >>>> >> > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Oct 10, 2012 5:52:35 PM > >>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker > >>>> >> > >>>> >> WARNING: Unexpected exception in the selector loop. > >>>> >> > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-2" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "pool-4-thread-4" java.lang.OutOfMemoryError: > GC > >>>> >> overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-0" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-10" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Oct 10, 2012 5:52:39 PM > >>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker > >>>> >> > >>>> >> WARNING: Unexpected exception in the selector loop. > >>>> >> > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Oct 10, 2012 5:52:39 PM > >>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker > >>>> >> > >>>> >> WARNING: Unexpected exception in the selector loop. > >>>> >> > >>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-9" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-1" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-5" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-3" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-13" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-6" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-4" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-5" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Exception in thread "lifecycleSupervisor-1-8" > >>>> >> java.lang.OutOfMemoryError: > >>>> >> GC overhead limit exceeded > >>>> >> > >>>> >> Oct 10, 2012 5:52:48 PM > >>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker > >>>> >> > >>>> >> WARNING: Unexpected exception in the selector loop. > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> Thanks, > >>>> >> > >>>> >> > >>>> >> > >>>> >> Roy > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> > > >>>> > > >>>> > >>>> > >>>> > >>>> -- > >>>> Apache MRUnit - Unit testing MapReduce - > >>>> http://incubator.apache.org/mrunit/ > >>> > >>> > >> > >> > >> > >> -- > >> ..Senthil > >> > >> "If there's anything more important than my ego around, I want it > >> caught and shot now." > >> - Douglas Adams. > >> > > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ >
