Hi, Maybe the error is not really in this tread, but has been occurred after accumulating data from a memory leak. Better analyse the memory allocations.
Thanks, Akalanka. On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe <[email protected]> wrote: > Lets analyse the heap dump and see what caused the issue... > also check the metrics recorded (via matric library) when server went OOM. > > On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage <[email protected]> wrote: > >> Hi, >> >> I came across out of memory exception while doing long running test on mb >> cluster with oracle database as message store. Following was the scenario i >> tested. >> >> 1. Start two clustered mb nodes with serializers (nodeA(coordinator), >> nodeB). >> 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" >> messages to nodeA with average 1500 TPS. >> 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" >> messages to nodeB with average 1000 TPS. >> 4. After around 14 hours, om exception has thrown from nodeA and all >> operations has halted. >> >> After analysing heap dump i found which thread has thrown the error, and >> it's as follows. >> >> "pool-25-thread-1" prio=5 tid=174 RUNNABLE >> at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) >> at java.lang.Long.toString(Long.java:267) >> at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) >> Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 >> Local Variable: com.google.gson.JsonArray#1 >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) >> at com.google.gson.internal.Streams.parse(Streams.java:44) >> at com.google.gson.JsonParser.parse(JsonParser.java:84) >> at com.google.gson.JsonParser.parse(JsonParser.java:59) >> Local Variable: com.google.gson.stream.JsonReader#1 >> at com.google.gson.JsonParser.parse(JsonParser.java:45) >> at >> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) >> at >> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) >> at >> com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >> Local Variable: >> com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 >> at >> com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) >> at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) >> at com.hazelcast.map.MapService.toObject(MapService.java:872) >> at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) >> at >> org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) >> Local Variable: java.util.TreeSet#1 >> at >> org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) >> Local Variable: java.lang.String#2380 >> Local Variable: java.lang.String#2379 >> at >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) >> Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 >> at >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) >> Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 >> Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 >> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) >> Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >> at >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >> Local Variable: org.apache.thrift.transport.TSocket#2 >> Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 >> Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 >> Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> Local Variable: >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 >> Local Variable: java.util.concurrent.ThreadPoolExecutor#26 >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 >> at java.lang.Thread.run(Thread.java:745) >> >> Have anyone faced similar issue ? What is the best method to handle this >> out of memory issue? >> >> >> Thanks, >> -- >> Pumudu Ruhunage >> Associate Software Engineer | WSO2 Inc >> M: +94 779 664493 | http://wso2.com >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Ramith Jayasinghe > Technical Lead > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > E: [email protected] > P: +94 777542851 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Darshana Akalanka Pagoda Arachchi,* *Software Engineer* *078-4721791*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
