Chandan Purushothama created CLOUDSTACK-7726:
------------------------------------------------
Summary: [Automation][XenServer] Failed to Remove LoadBalancer
Rule due to callHostPlugin failure
Key: CLOUDSTACK-7726
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7726
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Automation
Affects Versions: 4.5.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.5.0
=======================================================
CallHost Plugin Failure while trying to remove the Load Balancing Rule:
=======================================================
2014-10-13 19:20:06,408 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-45:ctx-fb2667f4) Seq 1-122723089845846136: Executing request
2014-10-13 19:20:06,410 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-89:ctx-0a5fbdf2) callHostPlugin failed for cmd: createFileInDomr
with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip:
169.254.2.250, filecontents: global
log 127.0.0.1:3914 local0 warning
maxconn 4096
maxpipes 1024
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option dontlognull
retries 3
option redispatch
option forwardfor
option forceclose
timeout connect 5000
timeout client 50000
timeout server 50000
listen stats_on_public 10.220.166.19:8081
mode http
option httpclose
stats enable
stats uri /admin?stats
stats realm Haproxy\ Statistics
stats auth admin1:AdMiN123
listen 10_220_166_28-2222 10.220.166.28:2222
balance roundrobin
, due to There was a failure communicating with the plugin.
2014-10-13 19:20:06,410 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Throwable caught while
executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
createFileInDomr with args filepath:
/etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250,
filecontents: global
log 127.0.0.1:3914 local0 warning
maxconn 4096
maxpipes 1024
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option dontlognull
retries 3
option redispatch
option forwardfor
option forceclose
timeout connect 5000
timeout client 50000
timeout server 50000
listen stats_on_public 10.220.166.19:8081
mode http
option httpclose
stats enable
stats uri /admin?stats
stats realm Haproxy\ Statistics
stats auth admin1:AdMiN123
listen 10_220_166_28-2222 10.220.166.28:2222
balance roundrobin
, due to There was a failure communicating with the plugin.
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3670)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:575)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:161)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:155)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:179)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:125)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:430)
at
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64)
at
com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
at
com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-10-13 19:20:06,411 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Response Received:
2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (DirectAgent-89:ctx-0a5fbdf2)
Seq 1-122723089845846135: Processing: { Ans: , MgmtId: 59408290029589, via: 1,
Ver: v1, Flags: 0,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
callHostPlugin failed for cmd: createFileInDomr with args filepath:
/etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250,
filecontents: global\n\tlog 127.0.0.1:3914 local0 warning\n\tmaxconn
4096\n\tmaxpipes 1024\n\tchroot /var/lib/haproxy\n\tuser haproxy\n\tgroup
haproxy\n\tdaemon\n\t \ndefaults\n\tlog global\n\tmode tcp\n\toption
dontlognull\n\tretries 3\n\toption redispatch\n\toption forwardfor\n\toption
forceclose\n\ttimeout connect 5000\n\ttimeout client 50000\n\ttimeout
server 50000\n\nlisten stats_on_public 10.220.166.19:8081\n\tmode
http\n\toption httpclose\n\tstats enable\n\tstats uri /admin?stats\n\tstats
realm Haproxy\\ Statistics\n\tstats auth admin1:AdMiN123\n\n\t \nlisten
10_220_166_28-2222 10.220.166.28:2222\n\tbalance roundrobin\n\t \n\t \n, due
to There was a failure communicating with the plugin.","wait":0}}] }
2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request]
(API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Seq
1-122723089845846135: Received: { Ans: , MgmtId: 59408290029589, via: 1, Ver:
v1, Flags: 0, { Answer } }
2014-10-13 19:20:06,424 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id:
11,while removing vmId 4
2014-10-13 19:20:06,427 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id: 11
while removing vm instances
2014-10-13 19:20:06,427 WARN [c.c.n.l.LoadBalancingRulesManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Unable to apply the load
balancer config because resource is unavaliable.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to apply loadbalancing rules on router
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4050)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3901)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy190.applyLoadBalancingRules(Unknown Source)
at
com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:384)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1845)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2433)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1881)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1805)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1377)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy138.removeFromLoadBalancer(Unknown Source)
at
org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-10-13 19:20:06,444 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-23:ctx-238092a2) Created a vif
825c5404-4bc8-3a35-8961-db3d95bde69d on 1
2014-10-13 19:20:06,452 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Unexpected exception while executing
org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to remove specified
load balancer rule id for vms [4]
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1413)
at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy138.removeFromLoadBalancer(Unknown Source)
at
org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-10-13 19:20:06,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Complete async job-84, jobStatus:
FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to remove specified load balancer rule id for vms [4]"}
2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Publish async job-84 complete on
message bus
2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs related to job- 84
2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Update db status for job- 84
2014-10-13 19:20:06,458 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs joined with job- 84 and
disjoin all subjobs created from job- 84
2014-10-13 19:20:06,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-41:ctx-9ee60aa8 job-84) Done executing
org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
for job-84
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
