the error indicates that source is pumping in data faster than the sink is draining the memory channel. causing the channel to fill up. it does not appear to be a MemCh issue.
On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian < [email protected]> wrote: > Hi > I am getting error while load testing a flume agent with HTTSrc and Avro > Sink. > > Is there an optimum configuration for memory channel for get flume working > on high load. Thanks in advance for any inputs. > > > tier1.sources = httpsrc1 > tier1.channels = c1 c2 > tier1.sinks = avrosink1 > > tier1.sources.httpsrc1.bind = host1 > tier1.sources.httpsrc1.type = http > tier1.sources.httpsrc1.port = 443 > tier1.sources.httpsrc1.channels = c1 c2 > tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler > tier1.sources.httpsrc1.handler.nickname = HTTPTesting > > tier1.channels.c1.type = memory > tier1.channels.c1.capacity = 10000 > tier1.channels.c1.transactionCapacity = 1000 > tier1.sinks.sink1.channel = c1 > tier1.sinks.avrosink1.channel = c1 > > tier1.channels.c2.type = memory > tier1.channels.c2.capacity = 100000 > tier1.channels.c2.transactionCapacity = 10000 > > #tier1.channels.c1.type = file > #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint > #tier1.channels.c1.dataDirs = /tmp/flume/data > #tier1.sinks.avrosink1.channel = c1 > > > tier1.sinks.avrosink1.type = avro > tier1.sinks.avrosink1.hostname = ipaddr > tier1.sinks.avrosink1.port = 4444 > tier1.sinks.avrosink1.avro.fileType = DataStream > tier1.sinks.avrosink1.serializer = avro_event > tier1.sinks.avrosink1.serializer.compressionCodec = snappy > > > > tier1.sinks = filesink1 > tier1.sinks.filesink1.type = file_roll > tier1.sinks.filesink1.channel = c2 > tier1.sinks.filesink1.batchSize = 10000 > #tier1.sinks.filesink1.rollInterval = 86400 > tier1.sinks.filesink1.rollInterval =0 > tier1.sinks.filesink1.sink.directory = /projects/logs > > > > WARN http.HTTPSource: Error appending event to channel. Channel might be > full. Consider increasing the channel capacity or make sure the sinks > perform faster. > org.apache.flume.ChannelException: Unable to put batch on required > channel: org.apache.flume.channel.MemoryChannel{name: c1} > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) > at > org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:814) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Caused by: org.apache.flume.ChannelException: Space for commit to queue > couldn't be acquired Sinks are likely not keeping up with sources, or the > buffer size is too tight > at > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128) > at > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192) > > > -- > Deepak Subhramanian > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
