Hi,
I again ran BAM with my custom producer sending events to it. For more than
2 hours it didn't gave me above mentioned exception. Then I started CEP and
sent netflow records to BAM through CEP.
After about 5 minutes it began to give above mentioned error
continuously.(nohup.out[1]).
I also noticed that at several places I got message saying " INFO
{org.apache.cassandra.net.MessagingService} - 1732 MUTATION messages
dropped in last 5000ms
" with number of messages increasing. In morning I discussed with Mohan and
in several resources at internet I read this error can occur because of
high message frequency to cassandra. I have done the performance tuning
recommendations mentioned in [2] also.
And also I didn't receive openjdk exception on BAM this time.
Can someone please give me some clue on this?
1. https://gist.github.com/cdwijayarathna/8920593
2.
http://docs.wso2.org/display/BAM240/Performance+Tuning+Recommendations<http://www.google.com/url?q=http%3A%2F%2Fdocs.wso2.org%2Fdisplay%2FBAM240%2FPerformance%2BTuning%2BRecommendations&sa=D&sntz=1&usg=AFQjCNFEiVzw4i1-kHAke3YxSC7WHw2qOg>
Thank you.
On Mon, Feb 10, 2014 at 10:56 PM, Chamila Wijayarathna <[email protected]>wrote:
> Hi Mohan,
> I'll check that.
> Thank you
>
>
> On Mon, Feb 10, 2014 at 9:30 PM, Mohanadarshan Vivekanandalingam <
> [email protected]> wrote:
>
>> Hi Chamila,
>>
>> I can see below exception before the timeoutException, where openjdk is
>> used but we normally recommend to use oracle jdk.. Please see link [1].. I
>> am doubt whether below issue leads to unstable situation.. Can you please
>> use the recommended jdk and see..
>>
>> BAM team, any more input on this??
>>
>> java.io.IOException: Cannot run program
>> "/usr/lib/jvm/java-6-openjdk/bin/java" (in directory
>> "/home/wso2/product/wso2bam-2.4.0"): java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
>> at java.lang.Runtime.exec(Runtime.java:610)
>>
>> at java.lang.Runtime.exec(Runtime.java:448)
>> at
>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:317)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>>
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> at
>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:569)
>> at
>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:282)
>>
>> at
>> org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:189)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>>
>> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
>> ... 18 more
>>
>>
>> [1] http://docs.wso2.org/display/BAM240/Installation+Prerequisites
>>
>>
>> Thanks,
>> Mohan
>>
>>
>>
>> On Mon, Feb 10, 2014 at 7:47 PM, Chamila Wijayarathna
>> <[email protected]>wrote:
>>
>>> I ran BAM without sending events to it and then it don't give any
>>> exeption.
>>>
>>>
>>> On Mon, Feb 10, 2014 at 6:53 PM, Chamila Wijayarathna
>>> <[email protected]>wrote:
>>>
>>>> Hello all,
>>>> I have a BAM instance running on a remote server with port offset of 1.
>>>> I'm sending a stream of wso2 events to it using an external
>>>> publisher[1].
>>>> After about 1.5 hours, I am getting the following error.
>>>>
>>>> *[2014-02-10 18:12:08,721] ERROR
>>>> {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber}
>>>> - Error processing event. *
>>>> *me.prettyprint.hector.api.exceptions.HTimedOutException:
>>>> TimedOutException()*
>>>> * at
>>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)*
>>>> * at
>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)*
>>>> * at
>>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)*
>>>> * at
>>>> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)*
>>>> * at
>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.commit(CassandraConnector.java:185)*
>>>> * at
>>>> org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.insertEventList(CassandraConnector.java:676)*
>>>> * at
>>>> org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:75)*
>>>> * at
>>>> org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:80)*
>>>> * at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)*
>>>> * at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)*
>>>> * at java.util.concurrent.FutureTask.run(FutureTask.java:166)*
>>>> * at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)*
>>>> * at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*
>>>> * at java.lang.Thread.run(Thread.java:679)*
>>>> *Caused by: TimedOutException()*
>>>> * at
>>>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)*
>>>> * at
>>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)*
>>>> * at
>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)*
>>>> * at
>>>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)*
>>>> * at
>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)*
>>>> * at
>>>> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)*
>>>> * at
>>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)*
>>>> * at
>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)*
>>>> * ... 12 more*
>>>>
>>>> Full command line output can be found at [2]. Events I am sending has
>>>> been written to cassandra properly. I have a hive script which run at each
>>>> minute on BAM too.
>>>> What is the reason for this error? How can I solve it?
>>>>
>>>>
>>>> 1. https://gist.github.com/cdwijayarathna/8915500
>>>> 2. https://gist.github.com/cdwijayarathna/8915782
>>>>
>>>> Thank You!
>>>>
>>>> --
>>>> *Chamila Wijayarathna*
>>>> Engineering Intern,
>>>> WSO2 Inc.
>>>>
>>>>
>>>
>>>
>>> --
>>> *Chamila Wijayarathna*
>>> Engineering Intern,
>>> WSO2 Inc.
>>>
>>>
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: [email protected]
>> phone:(+94) 771117673
>>
>
>
>
> --
> *Chamila Wijayarathna*
> Engineering Intern,
> WSO2 Inc.
>
>
--
*Chamila Wijayarathna*
Engineering Intern,
WSO2 Inc.
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
