Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested.
1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" messages to nodeA with average 1500 TPS. 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. "pool-25-thread-1" prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
