Hi All,
Observed following error on AS 520 manager Node when restarting the AS
worker Nodes. Issue seams Intermittent.
Have we observed this issue before ? Any solutions/explanations?
*Cluster Details:*
niginx fronted 1 Manager/ 2 worker node AS 520 (with Service
Pack1-hazelcast_3.0.1.wso2v1.jar) cluster, each node runs on separate
machines.
Local registry mounted to H2
Config and Governance registry mounted on a Mysql Database
Manager Node: 192.168.48.90 Wrk Node 1: 192.168.48.91 Wrk Node 2:
192.168.48.92
*Steps followed:*
1. Cluster was running nearly 2 weeks continuously
2. AS cluster used to host back-end services required for ESB
3. AS Services invoked thorough ESB continously as a long running/load test
4. Due to a OOM issue on worker Node2 , restarted the cluster
*Issue:*
TID: [0] [AS] [2015-03-17 02:06:34,207] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member left [a3ee426a-153e-4ebc-95d5-95292370ec23]: /192.168.48.92:4000
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [0] [AS] [2015-03-17 04:49:47,567] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member joined [a5d7cd86-fc94-409b-9720-e68210e6906c]: /192.168.48.92:4000
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [0] [AS] [2015-03-17 04:50:18,641] ERROR
{com.hazelcast.spi.impl.PartitionIteratingOperation} -
[192.168.48.90]:4000 [wso2.as.domain] [unknown]
java.lang.NullPointerException
{com.hazelcast.spi.impl.PartitionIteratingOperation}
com.hazelcast.nio.serialization.HazelcastSerializationException:
java.lang.NullPointerException
at
com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:298)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:254)
at
com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:262)
at
com.hazelcast.spi.impl.PartitionIteratingOperation$PartitionResponse.writeData(PartitionIteratingOperation.java:130)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36)
at
com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:36)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:252)
at
com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:262)
at
com.hazelcast.spi.impl.NormalResponse.writeData(NormalResponse.java:79)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36)
at
com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:48)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:176)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:157)
at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:151)
at
com.hazelcast.spi.impl.BasicOperationService.send(BasicOperationService.java:639)
at
com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:120)
at
com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:385)
at
com.hazelcast.spi.impl.BasicOperationService.processPacket(BasicOperationService.java:309)
at
com.hazelcast.spi.impl.BasicOperationService.access$400(BasicOperationService.java:102)
at
com.hazelcast.spi.impl.BasicOperationService$BasicOperationProcessorImpl.process(BasicOperationService.java:756)
at
com.hazelcast.spi.impl.BasicOperationScheduler$ProcessTask.run(BasicOperationScheduler.java:186)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at
com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
Caused by: java.lang.NullPointerException
at com.hazelcast.map.MapEntrySet.writeData(MapEntrySet.java:60)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129)
at
com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36)
at
com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:36)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:252)
... 25 more
TID: [0] [AS] [2015-03-17 04:52:41,699] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member joined [83db16fc-65be-41de-9750-36fb5dd6f6d7]: /192.168.48.91:4000
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
--
Asanka Vithanage
Senior Software Engineer -QA
Mobile: +94 0716286708
Email: asan...@wso2.com
WSO2 Inc. www.wso2.com
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
