Oh, 60k Events are indeed a lot to much for 20mb max heap size. If I calculate right you need around 600MB Heap. A memory channel with 20mb heap can handle around 2.3K events / second, an JDBC around 326. Did you batch the lines or read them sequential? Batching could help you too (catch 1k events and send out, next 1k and so on)
cheers - Alex -- Alexander Lorenz http://mapredit.blogspot.com On Apr 26, 2012, at 10:28 AM, shekhar sharma wrote: > Hello Alex, > One more bit of information is that out of 60000 records, previously i could > process all the 60000 records, but not Esper is logging only 5000 records.. > > > Regards, > Som > > On Thu, Apr 26, 2012 at 1:54 PM, shekhar sharma <shekhar2...@gmail.com> wrote: > HI Alex > Everything is working fine, Esper is able to detect the events and can write > them to separate log file. > Well i have written a python script which generates the log file and i am > creating almost 60000 records, and if you can see that my exec source tail > command is working on that log file... > After the events are detected and processed by Esper, and this error does not > comes instantenously, after some time it is thrown... > > Could it me because of -Xmx20 options ?..because the system is my virtual > linux box with 1GB RAM and 20GB HD.. > Can the size of the event can also effect? the reason why i am saying this is: > > Event read by exec source is of SimpleEvent which has 2 member variable MAP > for headers and byte[] for body and in Esper if you want to employ filter > "IN" or "LIKE" then it requires string..so what i did, i have created another > member variable of type string to convert the byte array to string. SO while > sending this event to the channel, is this bloating up the channel capacity? > > Regards, > Som > > > > On Thu, Apr 26, 2012 at 1:41 PM, alo alt <wget.n...@googlemail.com> wrote: > Thanks, > > org.apache.flume.sink.esper.EsperSink is availabe in the CLASSPATH and flume > can load them? Or did you write the sink? > > best, > Alex > > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Apr 26, 2012, at 10:03 AM, shekhar sharma wrote: > > > Hello Alex, > > My configuration file is as follows: > > > > host1.properties file > > > > host1.sources = avroSource > > host1.channels = memoryChannel > > host1.sinks = esper > > > > #avroSource configuration > > > > host1.sources.avroSource.type = avro > > host1.sources.avroSource.bind = localhost > > host1.sources.avroSource.port = 41414 > > host1.sources.avroSource.channels = memoryChannel > > > > #Channels > > > > host1.channels.memoryChannel.type = memory > > > > > > #Sinks > > host1.sinks.esper.type = org.apache.flume.sink.esper.EsperSink > > host1.sinks.esper.channel = memoryChannel > > > > host2.properties file: > > > > host2.sources = execSource > > host2.channels = memoryChannel > > host2.sinks = avroSink > > > > #execSource configuration > > > > host2.sources.execSource.type = exec > > host2.sources.execSource.command = /usr/bin/tail -F > > /home/dev/LogScripts/json.csv > > host2.sources.execSource.channels = memoryChannel > > > > #Channels > > > > host2.channels.memoryChannel.type = memory > > > > > > #Sinks > > host2.sinks.avroSink.type = avro > > host2.sinks.avroSink.hostname=localhost > > host2.sinks.avroSink.port=41414 > > host2.sinks.avroSink.batch-size = 10 > > host2.sinks.avroSink.runner.type = polling > > host2.sinks.avroSink.runner.polling.interval = 1 > > host2.sinks.avroSink.channel = memoryChannel > > > > Regards, > > Som Shekhar Sharma > > > > > > On Thu, Apr 26, 2012 at 12:35 PM, shekhar sharma <shekhar2...@gmail.com> > > wrote: > > Hello, > > While using memory channel i am getting the following error, what could be > > the reason for this: > > org.apache.flume.ChannelException: Space for commit to queue couldn't be > > acquired Sinks are likely not keeping up with sources, or the buffer size > > is too tight > > > > > > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:84) > > at > > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > > at > > org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:178) > > at > > org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:267) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > > > Regards, > > Som > > > > >
