[
https://issues.apache.org/jira/browse/CLOUDSTACK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandan Purushothama updated CLOUDSTACK-7726:
---------------------------------------------
Attachment: management-server.zip
> [Automation][XenServer] Failed to Remove LoadBalancer Rule due to
> callHostPlugin failure
> ----------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7726
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7726
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation
> Affects Versions: 4.5.0
> Reporter: Chandan Purushothama
> Priority: Critical
> Fix For: 4.5.0
>
> Attachments: management-server.zip
>
>
> =======================================================
> CallHost Plugin Failure while trying to remove the Load Balancing Rule:
> =======================================================
> 2014-10-13 19:20:06,408 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-45:ctx-fb2667f4) Seq 1-122723089845846136: Executing request
> 2014-10-13 19:20:06,410 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-89:ctx-0a5fbdf2) callHostPlugin failed for cmd: createFileInDomr
> with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip:
> 169.254.2.250, filecontents: global
> log 127.0.0.1:3914 local0 warning
> maxconn 4096
> maxpipes 1024
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
>
> defaults
> log global
> mode tcp
> option dontlognull
> retries 3
> option redispatch
> option forwardfor
> option forceclose
> timeout connect 5000
> timeout client 50000
> timeout server 50000
> listen stats_on_public 10.220.166.19:8081
> mode http
> option httpclose
> stats enable
> stats uri /admin?stats
> stats realm Haproxy\ Statistics
> stats auth admin1:AdMiN123
>
> listen 10_220_166_28-2222 10.220.166.28:2222
> balance roundrobin
>
>
> , due to There was a failure communicating with the plugin.
> 2014-10-13 19:20:06,410 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Throwable caught
> while executing command
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
> cmd: createFileInDomr with args filepath:
> /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250,
> filecontents: global
> log 127.0.0.1:3914 local0 warning
> maxconn 4096
> maxpipes 1024
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
>
> defaults
> log global
> mode tcp
> option dontlognull
> retries 3
> option redispatch
> option forwardfor
> option forceclose
> timeout connect 5000
> timeout client 50000
> timeout server 50000
> listen stats_on_public 10.220.166.19:8081
> mode http
> option httpclose
> stats enable
> stats uri /admin?stats
> stats realm Haproxy\ Statistics
> stats auth admin1:AdMiN123
>
> listen 10_220_166_28-2222 10.220.166.28:2222
> balance roundrobin
>
>
> , due to There was a failure communicating with the plugin.
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3670)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:575)
> at
> com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:161)
> at
> com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:155)
> at
> com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:179)
> at
> com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:125)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:430)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at
> com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,411 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Response Received:
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (DirectAgent-89:ctx-0a5fbdf2)
> Seq 1-122723089845846135: Processing: { Ans: , MgmtId: 59408290029589, via:
> 1, Ver: v1, Flags: 0,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> callHostPlugin failed for cmd: createFileInDomr with args filepath:
> /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250,
> filecontents: global\n\tlog 127.0.0.1:3914 local0 warning\n\tmaxconn
> 4096\n\tmaxpipes 1024\n\tchroot /var/lib/haproxy\n\tuser haproxy\n\tgroup
> haproxy\n\tdaemon\n\t \ndefaults\n\tlog global\n\tmode tcp\n\toption
> dontlognull\n\tretries 3\n\toption redispatch\n\toption forwardfor\n\toption
> forceclose\n\ttimeout connect 5000\n\ttimeout client 50000\n\ttimeout
> server 50000\n\nlisten stats_on_public 10.220.166.19:8081\n\tmode
> http\n\toption httpclose\n\tstats enable\n\tstats uri
> /admin?stats\n\tstats realm Haproxy\\ Statistics\n\tstats auth
> admin1:AdMiN123\n\n\t \nlisten 10_220_166_28-2222
> 10.220.166.28:2222\n\tbalance roundrobin\n\t \n\t \n, due to There was a
> failure communicating with the plugin.","wait":0}}] }
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Seq
> 1-122723089845846135: Received: { Ans: , MgmtId: 59408290029589, via: 1,
> Ver: v1, Flags: 0, { Answer } }
> 2014-10-13 19:20:06,424 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id:
> 11,while removing vmId 4
> 2014-10-13 19:20:06,427 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id:
> 11 while removing vm instances
> 2014-10-13 19:20:06,427 WARN [c.c.n.l.LoadBalancingRulesManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Unable to apply the
> load balancer config because resource is unavaliable.
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
> unreachable: Unable to apply loadbalancing rules on router
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4050)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3901)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy190.applyLoadBalancingRules(Unknown Source)
> at
> com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:384)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1845)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2433)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1881)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1805)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1377)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy138.removeFromLoadBalancer(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,444 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-23:ctx-238092a2) Created a vif
> 825c5404-4bc8-3a35-8961-db3d95bde69d on 1
> 2014-10-13 19:20:06,452 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Unexpected exception while
> executing
> org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to remove specified
> load balancer rule id for vms [4]
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1413)
> at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy138.removeFromLoadBalancer(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Complete async job-84, jobStatus:
> FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to remove specified load balancer rule id for vms [4]"}
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Publish async job-84 complete on
> message bus
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs related to job- 84
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Update db status for job- 84
> 2014-10-13 19:20:06,458 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs joined with job- 84
> and disjoin all subjobs created from job- 84
> 2014-10-13 19:20:06,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-41:ctx-9ee60aa8 job-84) Done executing
> org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
> for job-84
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
