Hi, We found the cause for this error. Will fix this and run the tests again.
Thank you, Sajini On Wed, Mar 18, 2015 at 5:36 PM, Akalanka Pagoda Arachchi < [email protected]> wrote: > Hi, > > Maybe the error is not really in this tread, but has been occurred after > accumulating data from a memory leak. Better analyse the memory allocations. > > Thanks, > Akalanka. > > On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe <[email protected]> > wrote: > >> Lets analyse the heap dump and see what caused the issue... >> also check the metrics recorded (via matric library) when server went OOM. >> >> On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage <[email protected]> wrote: >> >>> Hi, >>> >>> I came across out of memory exception while doing long running test on >>> mb cluster with oracle database as message store. Following was the >>> scenario i tested. >>> >>> 1. Start two clustered mb nodes with serializers (nodeA(coordinator), >>> nodeB). >>> 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" >>> messages to nodeA with average 1500 TPS. >>> 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" >>> messages to nodeB with average 1000 TPS. >>> 4. After around 14 hours, om exception has thrown from nodeA and all >>> operations has halted. >>> >>> After analysing heap dump i found which thread has thrown the error, and >>> it's as follows. >>> >>> "pool-25-thread-1" prio=5 tid=174 RUNNABLE >>> at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) >>> at java.lang.Long.toString(Long.java:267) >>> at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) >>> Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 >>> Local Variable: com.google.gson.JsonArray#1 >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) >>> at com.google.gson.internal.Streams.parse(Streams.java:44) >>> at com.google.gson.JsonParser.parse(JsonParser.java:84) >>> at com.google.gson.JsonParser.parse(JsonParser.java:59) >>> Local Variable: com.google.gson.stream.JsonReader#1 >>> at com.google.gson.JsonParser.parse(JsonParser.java:45) >>> at >>> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) >>> at >>> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) >>> at >>> com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >>> Local Variable: >>> com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 >>> at >>> com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) >>> at >>> com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) >>> at com.hazelcast.map.MapService.toObject(MapService.java:872) >>> at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) >>> at >>> org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) >>> Local Variable: java.util.TreeSet#1 >>> at >>> org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) >>> Local Variable: java.lang.String#2380 >>> Local Variable: java.lang.String#2379 >>> at >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) >>> Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 >>> at >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) >>> Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 >>> Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 >>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) >>> Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 >>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >>> at >>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >>> Local Variable: org.apache.thrift.transport.TSocket#2 >>> Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 >>> Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 >>> Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> Local Variable: >>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 >>> Local Variable: java.util.concurrent.ThreadPoolExecutor#26 >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Have anyone faced similar issue ? What is the best method to handle this >>> out of memory issue? >>> >>> >>> Thanks, >>> -- >>> Pumudu Ruhunage >>> Associate Software Engineer | WSO2 Inc >>> M: +94 779 664493 | http://wso2.com >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Darshana Akalanka Pagoda Arachchi,* > *Software Engineer* > *078-4721791* > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
