Hello, I wrote a test java file to put local files to hdfs, it put 1.1GB files to hdfs spent about 35 seconds. while using flume, 100MB files it should spend about 3-5 minutes. why flume is so slow and always report error? how to enlarge the flume hdfs sink throughput?
Andy 2013/3/1 周梦想 <[email protected]> > hello, > I saw lot's of flume-ng 1.3.1 agent Exception said Unable to put batch on > required channel,and then disconnected. > what is the effect? lost data or duplicate data? > What caused this error? > How to avoid this error? > > some of my conf: > agent46.channels.memch1.type = memory > agent46.channels.memch1.capacity = 100000 > agent46.channels.memch1.transactionCapactiy = 1000 > > agent46.sources.userlogsrc.type = avro > > agent46.sinks.myhdfssink.type = hdfs > agent46.sinks.myhdfssink.hdfs.rollInterval = 620 > agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000 > agent46.sinks.myhdfssink.hdfs.rollSize = 209715200 > agent46.sinks.myhdfssink.hdfs.rollCount = 600000 > agent46.sinks.myhdfssink.hdfs.batchSize = 50 > agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000 > agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100 > > ... > 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3] > (org.apache.flume.source.AvroSource.appendBatch:261) - Avro source > gamelogsrc: Unable to process event batch. Exception follows. > org.apache.flume.ChannelException: Unable to put batch on required > channel: org.apache.flume.channel.MemoryChannel{name: memch1} > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) > at > org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88) > at org.apache.avro.ipc.Responder.respond(Responder.java:149) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.apache.flume.ChannelException: Space for commit to queue > couldn't be acquired Sinks are likely not keeping up with sources, or the > buffer size is too tight > at > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126) > at > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192) > ... 28 more > 01 Mar 2013 03:00:20,503 DEBUG [conf-file-poller-0] > (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:188) > - Checking file:conf/hdfs.conf for changes > 01 Mar 2013 03:00:21,193 INFO [pool-9-thread-1] > (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171) > - [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140] > DISCONNECTED > dleUpstream:171) - [id: 0x40044674, /192.168.133.44:2404 :> / > 192.168.133.47:5140] UNBOUND > 01 Mar 2013 03:00:21,193 INFO [pool-9-thread-1] > (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171) > - [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140] CLOSED > 01 Mar 2013 03:00:21,194 INFO [pool-9-thread-1] > (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209) > - Connection to /192.168.133.44:2404 disconnected. > > Thanks, > Andy >
