[Dev] [MB] Out of memory error in long running mb cluster
Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested. 1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber MyQueue1, to nodeB and publish MyQueue1 messages to nodeA with average 1500 TPS. 3. Add queue subscriber MyQueue2 to nodeA and publish MyQueue2 messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. pool-25-thread-1 prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.init(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] Out of memory error in long running mb cluster
Lets analyse the heap dump and see what caused the issue... also check the metrics recorded (via matric library) when server went OOM. On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage pum...@wso2.com wrote: Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested. 1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber MyQueue1, to nodeB and publish MyQueue1 messages to nodeA with average 1500 TPS. 3. Add queue subscriber MyQueue2 to nodeA and publish MyQueue2 messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. pool-25-thread-1 prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.init(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] Out of memory error in long running mb cluster
Hi, Maybe the error is not really in this tread, but has been occurred after accumulating data from a memory leak. Better analyse the memory allocations. Thanks, Akalanka. On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe ram...@wso2.com wrote: Lets analyse the heap dump and see what caused the issue... also check the metrics recorded (via matric library) when server went OOM. On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage pum...@wso2.com wrote: Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested. 1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber MyQueue1, to nodeB and publish MyQueue1 messages to nodeA with average 1500 TPS. 3. Add queue subscriber MyQueue2 to nodeA and publish MyQueue2 messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. pool-25-thread-1 prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.init(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Ramith
Re: [Dev] [MB] Out of memory error in long running mb cluster
Hi, We found the cause for this error. Will fix this and run the tests again. Thank you, Sajini On Wed, Mar 18, 2015 at 5:36 PM, Akalanka Pagoda Arachchi darsha...@wso2.com wrote: Hi, Maybe the error is not really in this tread, but has been occurred after accumulating data from a memory leak. Better analyse the memory allocations. Thanks, Akalanka. On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe ram...@wso2.com wrote: Lets analyse the heap dump and see what caused the issue... also check the metrics recorded (via matric library) when server went OOM. On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage pum...@wso2.com wrote: Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested. 1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber MyQueue1, to nodeB and publish MyQueue1 messages to nodeA with average 1500 TPS. 3. Add queue subscriber MyQueue2 to nodeA and publish MyQueue2 messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. pool-25-thread-1 prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.init(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2
