Thanks Roshan. I increased the heapsize and it worked fine.
On Fri, Oct 18, 2013 at 11:55 PM, Roshan Naik <[email protected]>wrote: > the error indicates that source is pumping in data faster than the sink is > draining the memory channel. causing the channel to fill up. it does not > appear to be a MemCh issue. > > > On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian < > [email protected]> wrote: > >> Hi >> I am getting error while load testing a flume agent with HTTSrc and Avro >> Sink. >> >> Is there an optimum configuration for memory channel for get flume >> working on high load. Thanks in advance for any inputs. >> >> >> tier1.sources = httpsrc1 >> tier1.channels = c1 c2 >> tier1.sinks = avrosink1 >> >> tier1.sources.httpsrc1.bind = host1 >> tier1.sources.httpsrc1.type = http >> tier1.sources.httpsrc1.port = 443 >> tier1.sources.httpsrc1.channels = c1 c2 >> tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler >> tier1.sources.httpsrc1.handler.nickname = HTTPTesting >> >> tier1.channels.c1.type = memory >> tier1.channels.c1.capacity = 10000 >> tier1.channels.c1.transactionCapacity = 1000 >> tier1.sinks.sink1.channel = c1 >> tier1.sinks.avrosink1.channel = c1 >> >> tier1.channels.c2.type = memory >> tier1.channels.c2.capacity = 100000 >> tier1.channels.c2.transactionCapacity = 10000 >> >> #tier1.channels.c1.type = file >> #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint >> #tier1.channels.c1.dataDirs = /tmp/flume/data >> #tier1.sinks.avrosink1.channel = c1 >> >> >> tier1.sinks.avrosink1.type = avro >> tier1.sinks.avrosink1.hostname = ipaddr >> tier1.sinks.avrosink1.port = 4444 >> tier1.sinks.avrosink1.avro.fileType = DataStream >> tier1.sinks.avrosink1.serializer = avro_event >> tier1.sinks.avrosink1.serializer.compressionCodec = snappy >> >> >> >> tier1.sinks = filesink1 >> tier1.sinks.filesink1.type = file_roll >> tier1.sinks.filesink1.channel = c2 >> tier1.sinks.filesink1.batchSize = 10000 >> #tier1.sinks.filesink1.rollInterval = 86400 >> tier1.sinks.filesink1.rollInterval =0 >> tier1.sinks.filesink1.sink.directory = /projects/logs >> >> >> >> WARN http.HTTPSource: Error appending event to channel. Channel might be >> full. Consider increasing the channel capacity or make sure the sinks >> perform faster. >> org.apache.flume.ChannelException: Unable to put batch on required >> channel: org.apache.flume.channel.MemoryChannel{name: c1} >> at >> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) >> at >> org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:814) >> at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) >> at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> at org.mortbay.jetty.Server.handle(Server.java:326) >> at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> at >> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) >> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) >> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) >> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> at >> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >> Caused by: org.apache.flume.ChannelException: Space for commit to queue >> couldn't be acquired Sinks are likely not keeping up with sources, or the >> buffer size is too tight >> at >> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128) >> at >> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) >> at >> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192) >> >> >> -- >> Deepak Subhramanian >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. -- Deepak Subhramanian
