Hey Denny, thanks for the report.
Can you please try to rerun with: JAVA_OPTS="-Xms200m -Xmx200m -Xmn32m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -Xss128k -XX:+UseMembar -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/flume/gc.log" Plese attach the gc.log after. cheers, Alex Am 25.07.2012 10:35, schrieb Denny Ye: > hi all, > > I tested Flume in last week with ScribeSource( > https://issues.apache.org/jira/browse/FLUME-1382) and HDFS Sink. More > detailed conditions and deployment cases listed below. Too many 'Full GC' > impact the throughput and amount of events promoted into old generation. I > have applied some tuning methods, no much effect. > > Could someone give me your feedback or tip to reduce the GC problem? > Wish your attention. > > > PS: Using Mike's report template at > https://cwiki.apache.org/FLUME/flume-ng-performance-measurements.html > > * > * > > *Flume Performance Test 2012-07-25* > > *Overview* > > The Flume agent was run on its own physical machine in a single JVM. A > separate client machine generated load against the Flume box in > List<LogEntry> format. Flume stored data onto a 4-node HDFS cluster > configured on its own separate hardware. No virtual machines were used in > this test. > > *Hardware specs* > > CPU: Inter Xeon L5640 2 x quad-core @ 2.27 GHz (12 physical cores) > > Memory: 16 GB > > OS: CentOS release 5.3 (Final) > > *Flume configuration* > > JAVA Version: 1.6.0_20 (Java HotSpot 64-Bit Server VM) > > JAVA OPTS: -Xms1024m -Xmx4096m -XX:PermSize=256m -XX:NewRatio=1 > -XX:SurvivorRatio=5 -XX:InitialTenuringThreshold=15 > -XX:MaxTenuringThreshold=31 -XX:PretenureSizeThreshold=4096 > > Num. agents: 1 > > Num. parallel flows: 5 > > Source: ScribeSource > > Channel: MemoryChannel > > Sink: HDFSEventSink > > Selector: RandomSelector > > *Config-file* > > # list sources, channels, sinks for the agent > > agent.sources = seqGenSrc > > agent.channels = mc1 mc2 mc3 mc4 mc5 > > agent.sinks = hdfsSin1 hdfsSin2 hdfsSin3 hdfsSin4 hdfsSin5 > > > > # define sources > > agent.sources.seqGenSrc.type = org.apache.flume.source.scribe.ScribeSource > > agent.sources.seqGenSrc.selector.type = io.flume.RandomSelector > > > > # define sinks > > agent.sinks.hdfsSin1.type = hdfs > > agent.sinks.hdfsSin1.hdfs.path = /flume_test/data1/ > > agent.sinks.hdfsSin1.hdfs.rollInterval = 300 > > agent.sinks.hdfsSin1.hdfs.rollSize = 0 > > agent.sinks.hdfsSin1.hdfs.rollCount = 1000000 > > agent.sinks.hdfsSin1.hdfs.batchSize = 10000 > > agent.sinks.hdfsSin1.hdfs.fileType = DataStream > > agent.sinks.hdfsSin1.hdfs.txnEventMax = 1000 > > # ... define sink #2 #3 #4 #5 ... > > > > # define channels > > agent.channels.mc1.type = memory > > agent.channels.mc1.capacity = 1000000 > > agent.channels.mc1.transactionCapacity = 1000 > > # ... define channel #2 #3 #4 #5 ... > > > > # specify the channel each sink and source should use > > agent.sources.seqGenSrc.channels = mc1 mc2 mc3 mc4 mc5 > > agent.sinks.hdfsSin1.channel = mc1 > > # ... specify sink #2 #3 #4 #5 ... > > *Hadoop configuration* > > The HDFS sink was connected to a 4-node Hadoop cluster running CDH3u1. For > different HDFS sink, HDFS wrote data into different path. > > *Visualization of test setup* > > https://lh3.googleusercontent.com/dGumq1pu1Wr3Bj8WJmRHOoLWmUlGqxC4wW7_XCNO9R1wuh15LRXaKKxGoccpjBXtgqcdSVW-vtg > > There are 10 Scribe Clients and each client send 20 million LogEntry > objects to ScribleSource. > > *Data description* > > List<LogEntry> entries containing a string category and a ByteArray body. > The ByteArray body size is 500 bytes. > > *Results* > > Throughput: > > Average: Source: 46.4 MB/s, Sink: 45.2 MB/s > > Maximum: Source: 67.1 MB/s, Sink: 88.3 MB/s > > > > CPU: Average: 196%, Maximum: 440% > > > > GC: Young GC: 1636 times, Full GC: 384 times > > > No data loss. > > *Heap and GC* > > By analyzing JVM Heap, we found that there are many LogEntry objects in > OldGen. We have tried to carry out some optimizations, but the results are > not satisfactory. We will continue to track this limitation. > > > > FullGC Log examples: > > [Full GC [PSYoungGen: 1497984K->0K(1797568K)] [PSOldGen: > 1720643K->1693741K(2097152K)] 3218627K->1693741K(3894720K) [PSPermGen: > 14566K->14566K(262144K)], 5.0027700 secs] [Times: user=5.01 sys=0.00, > real=5.00 secs] > > [Full GC [PSYoungGen: 1497960K->0K(1797568K)] [PSOldGen: > 1693805K->1752540K(2097152K)] 3191765K->1752540K(3894720K) [PSPermGen: > 14571K->14571K(262144K)], 5.0732570 secs] [Times: user=5.07 sys=0.00, > real=5.07 secs] > > [Full GC [PSYoungGen: 1497984K->0K(1797568K)] [PSOldGen: > 1752540K->1642553K(2097152K)] 3250524K->1642553K(3894720K) [PSPermGen: > 14572K->14568K(262144K)], 5.0710730 secs] [Times: user=5.07 sys=0.01, > real=5.08 secs] > > > > －Regards > > Denny Ye >
