Yes, or simply give more memory (flume-env.sh), and add -Xms 400m in front of (JAVA_OPTS="-Xms400m -Xmx600m", as example).
best, Alex -- Alexander Lorenz http://mapredit.blogspot.com On Apr 26, 2012, at 11:13 AM, shekhar sharma wrote: > Do you mean something like this: > host2.sinks.avroSink.batch-size = 1000 > host2.sinks.avroSink.runner.type = polling > host2.sinks.avroSink.runner.polling.interval = 1 > > Regards, > Som Shekhar > > On Thu, Apr 26, 2012 at 2:07 PM, alo alt <wget.n...@googlemail.com> wrote: > Oh, 60k Events are indeed a lot to much for 20mb max heap size. If I > calculate right you need around 600MB Heap. A memory channel with 20mb heap > can handle around 2.3K events / second, an JDBC around 326. Did you batch the > lines or read them sequential? Batching could help you too (catch 1k events > and send out, next 1k and so on) > > cheers > - Alex > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Apr 26, 2012, at 10:28 AM, shekhar sharma wrote: > > > Hello Alex, > > One more bit of information is that out of 60000 records, previously i > > could process all the 60000 records, but not Esper is logging only 5000 > > records.. > > > > > > Regards, > > Som > > > > On Thu, Apr 26, 2012 at 1:54 PM, shekhar sharma <shekhar2...@gmail.com> > > wrote: > > HI Alex > > Everything is working fine, Esper is able to detect the events and can > > write them to separate log file. > > Well i have written a python script which generates the log file and i am > > creating almost 60000 records, and if you can see that my exec source tail > > command is working on that log file... > > After the events are detected and processed by Esper, and this error does > > not comes instantenously, after some time it is thrown... > > > > Could it me because of -Xmx20 options ?..because the system is my virtual > > linux box with 1GB RAM and 20GB HD.. > > Can the size of the event can also effect? the reason why i am saying this > > is: > > > > Event read by exec source is of SimpleEvent which has 2 member variable MAP > > for headers and byte[] for body and in Esper if you want to employ filter > > "IN" or "LIKE" then it requires string..so what i did, i have created > > another member variable of type string to convert the byte array to string. > > SO while sending this event to the channel, is this bloating up the channel > > capacity? > > > > Regards, > > Som > > > > > > > > On Thu, Apr 26, 2012 at 1:41 PM, alo alt <wget.n...@googlemail.com> wrote: > > Thanks, > > > > org.apache.flume.sink.esper.EsperSink is availabe in the CLASSPATH and > > flume can load them? Or did you write the sink? > > > > best, > > Alex > > > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > On Apr 26, 2012, at 10:03 AM, shekhar sharma wrote: > > > > > Hello Alex, > > > My configuration file is as follows: > > > > > > host1.properties file > > > > > > host1.sources = avroSource > > > host1.channels = memoryChannel > > > host1.sinks = esper > > > > > > #avroSource configuration > > > > > > host1.sources.avroSource.type = avro > > > host1.sources.avroSource.bind = localhost > > > host1.sources.avroSource.port = 41414 > > > host1.sources.avroSource.channels = memoryChannel > > > > > > #Channels > > > > > > host1.channels.memoryChannel.type = memory > > > > > > > > > #Sinks > > > host1.sinks.esper.type = org.apache.flume.sink.esper.EsperSink > > > host1.sinks.esper.channel = memoryChannel > > > > > > host2.properties file: > > > > > > host2.sources = execSource > > > host2.channels = memoryChannel > > > host2.sinks = avroSink > > > > > > #execSource configuration > > > > > > host2.sources.execSource.type = exec > > > host2.sources.execSource.command = /usr/bin/tail -F > > > /home/dev/LogScripts/json.csv > > > host2.sources.execSource.channels = memoryChannel > > > > > > #Channels > > > > > > host2.channels.memoryChannel.type = memory > > > > > > > > > #Sinks > > > host2.sinks.avroSink.type = avro > > > host2.sinks.avroSink.hostname=localhost > > > host2.sinks.avroSink.port=41414 > > > host2.sinks.avroSink.batch-size = 10 > > > host2.sinks.avroSink.runner.type = polling > > > host2.sinks.avroSink.runner.polling.interval = 1 > > > host2.sinks.avroSink.channel = memoryChannel > > > > > > Regards, > > > Som Shekhar Sharma > > > > > > > > > On Thu, Apr 26, 2012 at 12:35 PM, shekhar sharma <shekhar2...@gmail.com> > > > wrote: > > > Hello, > > > While using memory channel i am getting the following error, what could > > > be the reason for this: > > > org.apache.flume.ChannelException: Space for commit to queue couldn't be > > > acquired Sinks are likely not keeping up with sources, or the buffer size > > > is too tight > > > > > > > > > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:84) > > > at > > > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > > > at > > > org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:178) > > > at > > > org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:267) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > at > > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > Regards, > > > Som > > > > > > > > > > >
