Hi, We must first find out, why Cassandra takes that much of time to insert data, normally it should support very high write speeds, please check if Cassandra is tuned properly. Check here [1] for more information.
[1] http://docs.wso2.org/display/BAM240/Performance+Tuning+Recommendations Cheers, Anjana. On Tue, Feb 11, 2014 at 10:54 AM, Dunith Dhanushka <[email protected]> wrote: > Hi Chamila, > > This usually happens when BAM tries to write large number of records into > Cassandra and waits to read the result back. If Cassandra write operation > takes long time, client request will be timed out before writing is > completed. There can be many reasons for the write latency such as high > server load, ongoing compaction and GC swaps which can not be predicted > exactly. > > In order to avoid this situation, try below steps > > 1. Increase the 'rpc_timeout_in_ms' property in cassandra.yaml. Default > value is 10 seconds. > 2. In <BAM_HOME>/repository/conf/data-bridge/data-bridge-config.xml, > increase 'clientTimeoutMS' property to something like 40000. > > Regards, > Dunith > > > On Tue, Feb 11, 2014 at 9:18 AM, Chamila Wijayarathna <[email protected]>wrote: > >> Sorry, wrong Link. Correct one at [1]. >> >> 1. https://gist.github.com/cdwijayarathna/8928949 >> >> Thank You. >> >> >> On Tue, Feb 11, 2014 at 9:14 AM, Chamila Wijayarathna >> <[email protected]>wrote: >> >>> Hi Mohan, >>> I checked output by CEP which I forwarded to BAM [1]. This is for 8.59 >>> minute and it has 95 requests per 1 minute. >>> Since at this time there are less crowd in office, this can be increased >>> for about 10 times larger than this value. >>> But yesterday I got the error when I tested it at about 11.00 p.m. which >>> is a time with very low traffic. >>> Do you think 1000 per min is a traffic that BAM's cassandra can't bear? >>> And also when I test this in my local machine at the same time, I don't >>> receive this error. >>> >>> Thank You. >>> >>> >>> 1. https://gist.github.com/gists >>> >>> >>> On Tue, Feb 11, 2014 at 7:57 AM, Mohanadarshan Vivekanandalingam < >>> [email protected]> wrote: >>> >>>> Hi Chamila, >>>> >>>> On Mon, Feb 10, 2014 at 11:19 PM, Chamila Wijayarathna < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> I again ran BAM with my custom producer sending events to it. For more >>>>> than 2 hours it didn't gave me above mentioned exception. Then I started >>>>> CEP and sent netflow records to BAM through CEP. >>>>> After about 5 minutes it began to give above mentioned error >>>>> continuously.(nohup.out[1]). >>>>> I also noticed that at several places I got message saying " INFO >>>>> {org.apache.cassandra.net.MessagingService} - 1732 MUTATION messages >>>>> dropped in last 5000ms >>>>> " with number of messages increasing. In morning I discussed with >>>>> Mohan and in several resources at internet I read this error can occur >>>>> because of high message frequency to cassandra. I have done the >>>>> performance >>>>> tuning recommendations mentioned in [2] also. >>>>> And also I didn't receive openjdk exception on BAM this time.Cassandra >>>>> Can someone please give me some clue on this? >>>>> >>>> >>>> Then, I think as Prabath mentioned you have to scale the BAM where you >>>> need a fully distributed BAM cluster.. Do you have any clear idea about how >>>> many events are coming to BAM?? >>>> >>>> Thanks, >>>> Mohan >>>> >>>> >>>> >>>> >>>>> 1. https://gist.github.com/cdwijayarathna/8920593 >>>>> 2. >>>>> http://docs.wso2.org/display/BAM240/Performance+Tuning+Recommendations<http://www.google.com/url?q=http%3A%2F%2Fdocs.wso2.org%2Fdisplay%2FBAM240%2FPerformance%2BTuning%2BRecommendations&sa=D&sntz=1&usg=AFQjCNFEiVzw4i1-kHAke3YxSC7WHw2qOg> >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> On Mon, Feb 10, 2014 at 10:56 PM, Chamila Wijayarathna < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Mohan, >>>>>> I'll check that. >>>>>> Thank you >>>>>> >>>>>> >>>>>> On Mon, Feb 10, 2014 at 9:30 PM, Mohanadarshan Vivekanandalingam < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chamila, >>>>>>> >>>>>>> I can see below exception before the timeoutException, where openjdk >>>>>>> is used but we normally recommend to use oracle jdk.. Please see link >>>>>>> [1].. >>>>>>> I am doubt whether below issue leads to unstable situation.. Can you >>>>>>> please >>>>>>> use the recommended jdk and see.. >>>>>>> >>>>>>> BAM team, any more input on this?? >>>>>>> >>>>>>> java.io.IOException: Cannot run program >>>>>>> "/usr/lib/jvm/java-6-openjdk/bin/java" (in directory >>>>>>> "/home/wso2/product/wso2bam-2.4.0"): java.io.IOException: error=12, >>>>>>> Cannot allocate memory >>>>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:488) >>>>>>> at java.lang.Runtime.exec(Runtime.java:610) >>>>>>> >>>>>>> at java.lang.Runtime.exec(Runtime.java:448) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:317) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62) >>>>>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351) >>>>>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126) >>>>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201) >>>>>>> at >>>>>>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187) >>>>>>> at >>>>>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:569) >>>>>>> at >>>>>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:282) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> at >>>>>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:189) >>>>>>> at >>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>>> >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>> >>>>>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot >>>>>>> allocate memory >>>>>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) >>>>>>> >>>>>>> at java.lang.ProcessImpl.start(ProcessImpl.java:81) >>>>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:470) >>>>>>> ... 18 more >>>>>>> >>>>>>> >>>>>>> [1] http://docs.wso2.org/display/BAM240/Installation+Prerequisites >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Mohan >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 10, 2014 at 7:47 PM, Chamila Wijayarathna < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I ran BAM without sending events to it and then it don't give any >>>>>>>> exeption. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 10, 2014 at 6:53 PM, Chamila Wijayarathna < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hello all, >>>>>>>>> I have a BAM instance running on a remote server with port offset >>>>>>>>> of 1. >>>>>>>>> I'm sending a stream of wso2 events to it using an external >>>>>>>>> publisher[1]. >>>>>>>>> After about 1.5 hours, I am getting the following error. >>>>>>>>> >>>>>>>>> *[2014-02-10 18:12:08,721] ERROR >>>>>>>>> {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber} >>>>>>>>> - Error processing event. * >>>>>>>>> *me.prettyprint.hector.api.exceptions.HTimedOutException: >>>>>>>>> TimedOutException()* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)* >>>>>>>>> * at >>>>>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:185)* >>>>>>>>> * at >>>>>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:676)* >>>>>>>>> * at >>>>>>>>> org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:75)* >>>>>>>>> * at >>>>>>>>> org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)* >>>>>>>>> * at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)* >>>>>>>>> * at >>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)* >>>>>>>>> * at java.util.concurrent.FutureTask.run(FutureTask.java:166)* >>>>>>>>> * at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)* >>>>>>>>> * at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)* >>>>>>>>> * at java.lang.Thread.run(Thread.java:679)* >>>>>>>>> *Caused by: TimedOutException()* >>>>>>>>> * at >>>>>>>>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)* >>>>>>>>> * at >>>>>>>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)* >>>>>>>>> * at >>>>>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)* >>>>>>>>> * at >>>>>>>>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)* >>>>>>>>> * at >>>>>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)* >>>>>>>>> * ... 12 more* >>>>>>>>> >>>>>>>>> Full command line output can be found at [2]. Events I am sending >>>>>>>>> has been written to cassandra properly. I have a hive script which >>>>>>>>> run at >>>>>>>>> each minute on BAM too. >>>>>>>>> What is the reason for this error? How can I solve it? >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. https://gist.github.com/cdwijayarathna/8915500 >>>>>>>>> 2. https://gist.github.com/cdwijayarathna/8915782 >>>>>>>>> >>>>>>>>> Thank You! >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Chamila Wijayarathna* >>>>>>>>> Engineering Intern, >>>>>>>>> WSO2 Inc. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Chamila Wijayarathna* >>>>>>>> Engineering Intern, >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *V. Mohanadarshan* >>>>>>> *Software Engineer,* >>>>>>> *Data Technologies Team,* >>>>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>>>>> *lean.enterprise.middleware.* >>>>>>> >>>>>>> email: [email protected] >>>>>>> phone:(+94) 771117673 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chamila Wijayarathna* >>>>>> Engineering Intern, >>>>>> WSO2 Inc. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chamila Wijayarathna* >>>>> Engineering Intern, >>>>> WSO2 Inc. >>>>> >>>>> >>>> >>>> >>>> -- >>>> *V. Mohanadarshan* >>>> *Software Engineer,* >>>> *Data Technologies Team,* >>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>> *lean.enterprise.middleware.* >>>> >>>> email: [email protected] >>>> phone:(+94) 771117673 >>>> >>> >>> >>> >>> -- >>> *Chamila Wijayarathna* >>> Engineering Intern, >>> WSO2 Inc. >>> >>> >> >> >> -- >> *Chamila Wijayarathna* >> Engineering Intern, >> WSO2 Inc. >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Regards, > > Dunith Dhanushka, > Senior Software Engineer - BAM, > WSO2 Inc, > > Mobile - +94 71 8615744 > Blog - dunithd.wordpress.com <http://blog.dunith.com> > Twitter - @dunithd <http://twitter.com/dunithd> > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Anjana Fernando* Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
