Hi all,
I have an ESB cluster setup with 1 manager and 3 workers. I have deployed a
task that prints a log and the task is successfully running on one of the
workers at any given time.
However, if all 3 workers are stopped and one worker is restarted, I am
getting the following error in the worker :
ERROR - TaskServiceImpl Error initializing task managers: Error in cluster
message send-receive: com.hazelcast.spi.exception.TargetNotMemberException:
Not Member! target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
org.wso2.carbon.ntask.common.TaskException: Error in cluster message
send-receive: com.hazelcast.spi.exception.TargetNotMemberException: Not
Member! target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
Noticed that this port 4400 is the port of the last worker (out of the 3) I
stopped while stopping the workers. Refer the full stack trace at [1].
After this happens non of the workers run the task.
In this setup, manager node does not have any well known members and worker
nodes have manager as a well known member.
In another setup where each of the 4 nodes have other 3 nodes as well known
members, this error again came up when I kept stopping and restarting the
nodes.
What could be the cause of this error?
[1]
ERROR - TaskServiceImpl Error initializing task managers: Error in cluster
message send-receive: com.hazelcast.spi.exception.TargetNotMemberException:
Not Member! target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
org.wso2.carbon.ntask.common.TaskException: Error in cluster message
send-receive: com.hazelcast.spi.exception.TargetNotMemberException: Not
Member! target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
at
org.wso2.carbon.ntask.core.impl.clustered.ClusterGroupCommunicator.sendReceive(ClusterGroupCommunicator.java:202)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.sendReceive(ClusteredTaskManager.java:255)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.getRunningTasksInServer(ClusteredTaskManager.java:285)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.getAllRunningTasksInServers(ClusteredTaskManager.java:239)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.scheduleMissingTasks(ClusteredTaskManager.java:60)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusteredTaskManager.initStartupTasks(ClusteredTaskManager.java:55)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagersForType(TaskServiceImpl.java:137)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.serverInitialized(TaskServiceImpl.java:175)
at
org.wso2.carbon.ntask.core.TaskStartupHandler.completedServerStartup(TaskStartupHandler.java:44)
at
org.wso2.carbon.core.internal.CarbonCoreServiceComponent.notifyAfter(CarbonCoreServiceComponent.java:238)
at
org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:208)
at
org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:285)
at
org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at
org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at
org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at
org.wso2.carbon.throttling.agent.internal.ThrottlingAgentServiceComponent.registerThrottlingAgent(ThrottlingAgentServiceComponent.java:123)
at
org.wso2.carbon.throttling.agent.internal.ThrottlingAgentServiceComponent.activate(ThrottlingAgentServiceComponent.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at
org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at
org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
at
org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at
org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at
org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at
org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at
org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at
org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at
org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
at
org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:517)
at
org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
at
org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at
org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at
org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
at
org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at
org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at
org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at
org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at
org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at
org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at
org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at
org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
at
org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
at
org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
at
org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
at
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1279)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1192)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5210)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5493)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.util.concurrent.ExecutionException:
com.hazelcast.spi.exception.TargetNotMemberException: Not Member!
target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:825)
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698)
at
com.hazelcast.util.executor.DelegatingFuture.get(DelegatingFuture.java:68)
at
com.hazelcast.util.executor.DelegatingFuture.get(DelegatingFuture.java:56)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusterGroupCommunicator.sendReceive(ClusterGroupCommunicator.java:200)
... 84 more
Caused by: com.hazelcast.spi.exception.TargetNotMemberException: Not
Member! target:Address[10.100.7.10]:4400, partitionId: -1, operation:
com.hazelcast.executor.MemberCallableTaskOperation, service:
hz:impl:executorService
at
com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:329)
at
com.hazelcast.spi.impl.BasicInvocation.invoke(BasicInvocation.java:190)
at
com.hazelcast.spi.impl.BasicOperationService.invokeOnTarget(BasicOperationService.java:285)
at
com.hazelcast.executor.ExecutorServiceProxy.submitToMember(ExecutorServiceProxy.java:284)
at
org.wso2.carbon.ntask.core.impl.clustered.ClusterGroupCommunicator.sendReceive(ClusterGroupCommunicator.java:198)
... 84 more
[2015-02-12 16:50:29,480] INFO - CarbonUIServiceComponent Mgt Console URL
: https://10.100.7.10:9444/carbon/
Thanks,
Maheeka Jayasuriya
Software Engineer
Mobile : +94777750661
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
