Hi Chamila,
On Mon, Feb 10, 2014 at 11:19 PM, Chamila Wijayarathna <[email protected]>wrote:
> Hi,
> I again ran BAM with my custom producer sending events to it. For more
> than 2 hours it didn't gave me above mentioned exception. Then I started
> CEP and sent netflow records to BAM through CEP.
> After about 5 minutes it began to give above mentioned error
> continuously.(nohup.out[1]).
> I also noticed that at several places I got message saying " INFO
> {org.apache.cassandra.net.MessagingService} - 1732 MUTATION messages
> dropped in last 5000ms
> " with number of messages increasing. In morning I discussed with Mohan
> and in several resources at internet I read this error can occur because of
> high message frequency to cassandra. I have done the performance tuning
> recommendations mentioned in [2] also.
> And also I didn't receive openjdk exception on BAM this time.Cassandra
> Can someone please give me some clue on this?
>
Then, I think as Prabath mentioned you have to scale the BAM where you need
a fully distributed BAM cluster.. Do you have any clear idea about how many
events are coming to BAM??
Thanks,
Mohan
> 1. https://gist.github.com/cdwijayarathna/8920593
> 2.
> http://docs.wso2.org/display/BAM240/Performance+Tuning+Recommendations<http://www.google.com/url?q=http%3A%2F%2Fdocs.wso2.org%2Fdisplay%2FBAM240%2FPerformance%2BTuning%2BRecommendations&sa=D&sntz=1&usg=AFQjCNFEiVzw4i1-kHAke3YxSC7WHw2qOg>
>
> Thank you.
>
>
> On Mon, Feb 10, 2014 at 10:56 PM, Chamila Wijayarathna
> <[email protected]>wrote:
>
>> Hi Mohan,
>> I'll check that.
>> Thank you
>>
>>
>> On Mon, Feb 10, 2014 at 9:30 PM, Mohanadarshan Vivekanandalingam <
>> [email protected]> wrote:
>>
>>> Hi Chamila,
>>>
>>> I can see below exception before the timeoutException, where openjdk is
>>> used but we normally recommend to use oracle jdk.. Please see link [1].. I
>>> am doubt whether below issue leads to unstable situation.. Can you please
>>> use the recommended jdk and see..
>>>
>>> BAM team, any more input on this??
>>>
>>> java.io.IOException: Cannot run program
>>> "/usr/lib/jvm/java-6-openjdk/bin/java" (in directory
>>> "/home/wso2/product/wso2bam-2.4.0"): java.io.IOException: error=12, Cannot
>>> allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
>>> at java.lang.Runtime.exec(Runtime.java:610)
>>>
>>> at java.lang.Runtime.exec(Runtime.java:448)
>>> at
>>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:317)
>>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>>> at
>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>>>
>>> at
>>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>>> at
>>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>>> at
>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:569)
>>> at
>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:282)
>>>
>>>
>>> at
>>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:189)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:679)
>>>
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>>> allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>>>
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
>>> ... 18 more
>>>
>>>
>>> [1] http://docs.wso2.org/display/BAM240/Installation+Prerequisites
>>>
>>>
>>>
>>> Thanks,
>>> Mohan
>>>
>>>
>>>
>>> On Mon, Feb 10, 2014 at 7:47 PM, Chamila Wijayarathna
>>> <[email protected]>wrote:
>>>
>>>> I ran BAM without sending events to it and then it don't give any
>>>> exeption.
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 6:53 PM, Chamila Wijayarathna <[email protected]
>>>> > wrote:
>>>>
>>>>> Hello all,
>>>>> I have a BAM instance running on a remote server with port offset of 1.
>>>>> I'm sending a stream of wso2 events to it using an external
>>>>> publisher[1].
>>>>> After about 1.5 hours, I am getting the following error.
>>>>>
>>>>> *[2014-02-10 18:12:08,721] ERROR
>>>>> {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber}
>>>>> - Error processing event. *
>>>>> *me.prettyprint.hector.api.exceptions.HTimedOutException:
>>>>> TimedOutException()*
>>>>> * at
>>>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)*
>>>>> * at
>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)*
>>>>> * at
>>>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)*
>>>>> * at
>>>>> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)*
>>>>> * at
>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:185)*
>>>>> * at
>>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:676)*
>>>>> * at
>>>>> org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:75)*
>>>>> * at
>>>>> org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)*
>>>>> * at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)*
>>>>> * at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)*
>>>>> * at java.util.concurrent.FutureTask.run(FutureTask.java:166)*
>>>>> * at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)*
>>>>> * at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*
>>>>> * at java.lang.Thread.run(Thread.java:679)*
>>>>> *Caused by: TimedOutException()*
>>>>> * at
>>>>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)*
>>>>> * at
>>>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)*
>>>>> * at
>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)*
>>>>> * at
>>>>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)*
>>>>> * at
>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)*
>>>>> * at
>>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)*
>>>>> * at
>>>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)*
>>>>> * at
>>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)*
>>>>> * ... 12 more*
>>>>>
>>>>> Full command line output can be found at [2]. Events I am sending has
>>>>> been written to cassandra properly. I have a hive script which run at each
>>>>> minute on BAM too.
>>>>> What is the reason for this error? How can I solve it?
>>>>>
>>>>>
>>>>> 1. https://gist.github.com/cdwijayarathna/8915500
>>>>> 2. https://gist.github.com/cdwijayarathna/8915782
>>>>>
>>>>> Thank You!
>>>>>
>>>>> --
>>>>> *Chamila Wijayarathna*
>>>>> Engineering Intern,
>>>>> WSO2 Inc.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Chamila Wijayarathna*
>>>> Engineering Intern,
>>>> WSO2 Inc.
>>>>
>>>>
>>>
>>>
>>> --
>>> *V. Mohanadarshan*
>>> *Software Engineer,*
>>> *Data Technologies Team,*
>>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>>> *lean.enterprise.middleware.*
>>>
>>> email: [email protected]
>>> phone:(+94) 771117673
>>>
>>
>>
>>
>> --
>> *Chamila Wijayarathna*
>> Engineering Intern,
>> WSO2 Inc.
>>
>>
>
>
> --
> *Chamila Wijayarathna*
> Engineering Intern,
> WSO2 Inc.
>
>
--
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*
email: [email protected]
phone:(+94) 771117673
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
