Hi Mohan, I checked output by CEP which I forwarded to BAM [1]. This is for 8.59 minute and it has 95 requests per 1 minute. Since at this time there are less crowd in office, this can be increased for about 10 times larger than this value. But yesterday I got the error when I tested it at about 11.00 p.m. which is a time with very low traffic. Do you think 1000 per min is a traffic that BAM's cassandra can't bear? And also when I test this in my local machine at the same time, I don't receive this error.
Thank You. 1. https://gist.github.com/gists On Tue, Feb 11, 2014 at 7:57 AM, Mohanadarshan Vivekanandalingam < [email protected]> wrote: > Hi Chamila, > > On Mon, Feb 10, 2014 at 11:19 PM, Chamila Wijayarathna > <[email protected]>wrote: > >> Hi, >> I again ran BAM with my custom producer sending events to it. For more >> than 2 hours it didn't gave me above mentioned exception. Then I started >> CEP and sent netflow records to BAM through CEP. >> After about 5 minutes it began to give above mentioned error >> continuously.(nohup.out[1]). >> I also noticed that at several places I got message saying " INFO >> {org.apache.cassandra.net.MessagingService} - 1732 MUTATION messages >> dropped in last 5000ms >> " with number of messages increasing. In morning I discussed with Mohan >> and in several resources at internet I read this error can occur because of >> high message frequency to cassandra. I have done the performance tuning >> recommendations mentioned in [2] also. >> And also I didn't receive openjdk exception on BAM this time.Cassandra >> Can someone please give me some clue on this? >> > > Then, I think as Prabath mentioned you have to scale the BAM where you > need a fully distributed BAM cluster.. Do you have any clear idea about how > many events are coming to BAM?? > > Thanks, > Mohan > > > > >> 1. https://gist.github.com/cdwijayarathna/8920593 >> 2. >> http://docs.wso2.org/display/BAM240/Performance+Tuning+Recommendations<http://www.google.com/url?q=http%3A%2F%2Fdocs.wso2.org%2Fdisplay%2FBAM240%2FPerformance%2BTuning%2BRecommendations&sa=D&sntz=1&usg=AFQjCNFEiVzw4i1-kHAke3YxSC7WHw2qOg> >> >> Thank you. >> >> >> On Mon, Feb 10, 2014 at 10:56 PM, Chamila Wijayarathna >> <[email protected]>wrote: >> >>> Hi Mohan, >>> I'll check that. >>> Thank you >>> >>> >>> On Mon, Feb 10, 2014 at 9:30 PM, Mohanadarshan Vivekanandalingam < >>> [email protected]> wrote: >>> >>>> Hi Chamila, >>>> >>>> I can see below exception before the timeoutException, where openjdk is >>>> used but we normally recommend to use oracle jdk.. Please see link [1].. I >>>> am doubt whether below issue leads to unstable situation.. Can you please >>>> use the recommended jdk and see.. >>>> >>>> BAM team, any more input on this?? >>>> >>>> java.io.IOException: Cannot run program >>>> "/usr/lib/jvm/java-6-openjdk/bin/java" (in directory >>>> "/home/wso2/product/wso2bam-2.4.0"): java.io.IOException: error=12, Cannot >>>> allocate memory >>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:488) >>>> at java.lang.Runtime.exec(Runtime.java:610) >>>> >>>> at java.lang.Runtime.exec(Runtime.java:448) >>>> at >>>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:317) >>>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129) >>>> at >>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62) >>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351) >>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126) >>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934) >>>> >>>> at >>>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201) >>>> at >>>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187) >>>> at >>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:569) >>>> at >>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:282) >>>> >>>> >>>> >>>> at >>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:189) >>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:679) >>>> >>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot >>>> allocate memory >>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) >>>> >>>> at java.lang.ProcessImpl.start(ProcessImpl.java:81) >>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:470) >>>> ... 18 more >>>> >>>> >>>> [1] http://docs.wso2.org/display/BAM240/Installation+Prerequisites >>>> >>>> >>>> >>>> >>>> Thanks, >>>> Mohan >>>> >>>> >>>> >>>> On Mon, Feb 10, 2014 at 7:47 PM, Chamila Wijayarathna <[email protected] >>>> > wrote: >>>> >>>>> I ran BAM without sending events to it and then it don't give any >>>>> exeption. >>>>> >>>>> >>>>> On Mon, Feb 10, 2014 at 6:53 PM, Chamila Wijayarathna < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello all, >>>>>> I have a BAM instance running on a remote server with port offset of >>>>>> 1. >>>>>> I'm sending a stream of wso2 events to it using an external >>>>>> publisher[1]. >>>>>> After about 1.5 hours, I am getting the following error. >>>>>> >>>>>> *[2014-02-10 18:12:08,721] ERROR >>>>>> {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber} >>>>>> - Error processing event. * >>>>>> *me.prettyprint.hector.api.exceptions.HTimedOutException: >>>>>> TimedOutException()* >>>>>> * at >>>>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)* >>>>>> * at >>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)* >>>>>> * at >>>>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)* >>>>>> * at >>>>>> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)* >>>>>> * at >>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:185)* >>>>>> * at >>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:676)* >>>>>> * at >>>>>> org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:75)* >>>>>> * at >>>>>> org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)* >>>>>> * at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)* >>>>>> * at >>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)* >>>>>> * at java.util.concurrent.FutureTask.run(FutureTask.java:166)* >>>>>> * at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)* >>>>>> * at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)* >>>>>> * at java.lang.Thread.run(Thread.java:679)* >>>>>> *Caused by: TimedOutException()* >>>>>> * at >>>>>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)* >>>>>> * at >>>>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)* >>>>>> * at >>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)* >>>>>> * at >>>>>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)* >>>>>> * at >>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)* >>>>>> * at >>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)* >>>>>> * at >>>>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)* >>>>>> * at >>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)* >>>>>> * ... 12 more* >>>>>> >>>>>> Full command line output can be found at [2]. Events I am sending >>>>>> has been written to cassandra properly. I have a hive script which run >>>>>> at >>>>>> each minute on BAM too. >>>>>> What is the reason for this error? How can I solve it? >>>>>> >>>>>> >>>>>> 1. https://gist.github.com/cdwijayarathna/8915500 >>>>>> 2. https://gist.github.com/cdwijayarathna/8915782 >>>>>> >>>>>> Thank You! >>>>>> >>>>>> -- >>>>>> *Chamila Wijayarathna* >>>>>> Engineering Intern, >>>>>> WSO2 Inc. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chamila Wijayarathna* >>>>> Engineering Intern, >>>>> WSO2 Inc. >>>>> >>>>> >>>> >>>> >>>> -- >>>> *V. Mohanadarshan* >>>> *Software Engineer,* >>>> *Data Technologies Team,* >>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>> *lean.enterprise.middleware.* >>>> >>>> email: [email protected] >>>> phone:(+94) 771117673 >>>> >>> >>> >>> >>> -- >>> *Chamila Wijayarathna* >>> Engineering Intern, >>> WSO2 Inc. >>> >>> >> >> >> -- >> *Chamila Wijayarathna* >> Engineering Intern, >> WSO2 Inc. >> >> > > > -- > *V. Mohanadarshan* > *Software Engineer,* > *Data Technologies Team,* > *WSO2, Inc. http://wso2.com <http://wso2.com> * > *lean.enterprise.middleware.* > > email: [email protected] > phone:(+94) 771117673 > -- *Chamila Wijayarathna* Engineering Intern, WSO2 Inc.
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
