I am running xenserver 6.2, I have 1 cluster with 4 Hosts. /Sonali
-----Original Message----- From: Daan Hoogland [mailto:[email protected]] Sent: Tuesday, February 24, 2015 1:22 PM To: [email protected] Subject: Re: unable to remove NIC from instance Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet. What type of hypervisors are you running? On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <[email protected]> wrote: > Hi Daan, > > Super thanks for reply. I am running version 4.4.2 > > /Sonali > > -----Original Message----- > From: Daan Hoogland [mailto:[email protected]] > Sent: Tuesday, February 24, 2015 1:05 PM > To: [email protected] > Subject: Re: unable to remove NIC from instance > > It seems to me you are a victim of overzealous exception handling. > I've seen that happen in the async job handling. The job dispatching or > construction has an exception that is not handled but caught and ignored, it > end and the async job is not removed from the queue. I'd like to look at the > code near the stack trace but for that I need the exact version of Cloudstack > you are running. > > On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <[email protected]> > wrote: >> Hi Daan, >> >> I have checked logs form starting of api call, and I have not found >> anything special, But apart from error, I found one warning, >> >> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >> to find lock for the key vm_instance24 and thread id 137380473 >> >> Don’t know meaning of this. >> >> Here are my logs, >> >> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response >> Received: >> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request] >> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: >> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: >> 10, >> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1," >> _ >> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx: >> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","r- >> 6 >> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"vi >> r >> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":" >> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"viridia >> n >> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"viri >> d >> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true;a >> c >> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;ap >> i >> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"resu >> l >> t":true,"wait":0}}] } >> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing >> request >> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization 0.11499999999999999 >> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received: >> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId >> 249373720971888: Resp: Routing to peer >> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing >> request >> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588 >> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999 >> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received: >> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId >> 249373720971888: Resp: Routing to peer >> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing >> request >> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization 0.13343749999999996 >> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received: >> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId >> 249373720971888: Resp: Routing to peer >> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET >> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iEl >> w >> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df0 >> 7 >> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526, >> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType: >> None, instanceId: null, cmd: >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, >> cmdInfo: >> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >> c >> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d9 >> 7 >> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df075 >> 3 >> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >> c >> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d97 >> 9 >> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccount >> I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >> IN_PROGRESS, >> processStatus: 0, resultCode: 0, result: null, initMsid: >> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >> null, created: null} >> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job >> monitoring >> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO >> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId: >> null, cmd: >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, >> cmdInfo: >> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >> c >> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d9 >> 7 >> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df075 >> 3 >> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >> c >> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d97 >> 9 >> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccount >> I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >> IN_PROGRESS, >> processStatus: 0, resultCode: 0, result: null, initMsid: >> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >> null, created: null} >> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END=== 192.168.45.32 >> -- GET >> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iEl >> w >> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df0 >> 7 >> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl] >> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console proxy >> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync job-527 >> execution on object VmWorkJobQueue.24 >> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >> to find lock for the key vm_instance24 and thread id 137380473 >> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] >> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary >> storage VM >> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] >> (StatsCollector-4:ctx-4a027738) HostStatsCollector is running... >> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing >> request >> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received: >> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request] >> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received: >> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, { >> GetHostStatsAnswer } } >> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing >> request >> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received: >> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request] >> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received: >> { >> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, { >> GetHostStatsAnswer } } >> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing >> request >> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received: >> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request] >> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received: >> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { >> GetHostStatsAnswer } } >> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET >> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8 >> a >> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=14246934 >> 0 >> 6327 >> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing >> request >> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] >> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received: >> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request] >> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received: >> { >> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, { >> GetHostStatsAnswer } } >> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END=== 192.168.45.32 >> -- GET >> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8 >> a >> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=14246934 >> 0 >> 6327 >> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception >> while executing >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >> java.lang.RuntimeException: Job failed due to exception null >> at >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526, >> jobStatus: FAILED, resultCode: 530, result: >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >> [],"errorcode":530,"errortext":"Job failed due to exception null"} >> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >> for job-526 >> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job >> monitoring >> >> >> /Sonali >> >> -----Original Message----- >> From: Daan Hoogland [mailto:[email protected]] >> Sent: Monday, February 23, 2015 2:42 PM >> To: [email protected] >> Subject: Re: unable to remove NIC from instance >> >> Sonali, >> >> this looks like a consequential error. You need to look further back into >> the log to find the root cause. The start job is the a query for results, >> please find the start for the actual api command. >> >> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <[email protected]> >> wrote: >>> I am trying to remove "Shared guest network". These are new logs, >>> >>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >>> (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status. >>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 -- GET >>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>> c >>> f >>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>> 7 >>> 3 >>> 8796 >>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END=== >>> 115.114.134.114 -- GET >>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>> c >>> f >>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>> 7 >>> 3 >>> 8796 >>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher] >>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception >>> while executing >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>> java.lang.RuntimeException: Job failed due to exception null >>> at >>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async job-499, >>> jobStatus: FAILED, resultCode: 530, result: >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>> for job-499 >>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from job >>> monitoring >>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] >>> (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running... >>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 -- GET >>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>> c >>> f >>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>> 7 >>> 4 >>> 1797 >>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END=== >>> 115.114.134.114 -- GET >>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>> c >>> f >>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>> 7 >>> 4 >>> 1797 >>> >>> >>> /Sonali >>> >>> -----Original Message----- >>> From: Sonali Jadhav [mailto:[email protected]] >>> Sent: Monday, February 23, 2015 10:25 AM >>> To: [email protected] >>> Subject: RE: unable to remove NIC from instance >>> >>> Hi, >>> >>> Yes I have tried that, and issue is still same. Same logs. >>> >>> /Sonali >>> >>> -----Original Message----- >>> From: Daan Hoogland [mailto:[email protected]] >>> Sent: Sunday, February 22, 2015 10:57 PM >>> To: [email protected] >>> Subject: Re: unable to remove NIC from instance >>> >>> Sonil, Did you try what Gopalakrishnan suggested? >>> >>> Also can you expand on the version of cloudstack you are running? >>> >>> thanks, >>> >>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <[email protected]> >>> wrote: >>>> Anyone help? >>>> /Sonali >>>> >>>> >>>> ---- Sonali Jadhav wrote ---- >>>> >>>> Anyone? >>>> >>>> Sent from my Sony Xperia™ smartphone >>>> >>>> >>>> ---- Sonali Jadhav wrote ---- >>>> >>>> Here are logs, >>>> >>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl] >>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495: >>>> { >>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010, >>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0} >>>> } >>>> ] >>>> } >>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl] >>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work >>>> job-483 >>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction] >>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction: >>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by >>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-Tr >>>> a >>>> n >>>> s >>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Refl >>>> e >>>> c >>>> t >>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke: >>>> 9 >>>> 1 >>>> - >>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:20 >>>> 4 >>>> - >>>> $ >>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Clea >>>> n >>>> u >>>> p >>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultManag >>>> e >>>> d >>>> C >>>> ontext$1.call:56 >>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl] >>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due to >>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: >>>> com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE >>>> async_job.id= 483 >>>> at >>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178) >>>> at >>>> org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149) >>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >>>> at >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >>>> at >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >>>> at >>>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >>>> at >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >>>> at >>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >>>> at >>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >>>> at >>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source) >>>> at >>>> com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> Caused by: >>>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: >>>> Cannot delete or update a parent row: a foreign key constraint fails >>>> (`cloud`.`async_job_join_map`, CONSTRAINT >>>> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) >>>> REFERENCES `async_job` (`id`)) >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> Method) >>>> at >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040) >>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) >>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) >>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) >>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) >>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625) >>>> at >>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119) >>>> at >>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415) >>>> at >>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333) >>>> at >>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>> at >>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169) >>>> ... 26 more >>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET >>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>> 3 >>>> f >>>> 1 >>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>> 9 >>>> 6 >>>> 3 >>>> 84 >>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl] >>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to >>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END=== >>>> 192.168.34.1 >>>> -- GET >>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>> 3 >>>> f >>>> 1 >>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>> 9 >>>> 6 >>>> 3 >>>> 84 >>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception >>>> while executing >>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd >>>> java.lang.RuntimeException: Job failed due to exception null >>>> at >>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async job-482, >>>> jobStatus: FAILED, resultCode: 530, result: >>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing >>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for >>>> job-482 >>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts suitable >>>> for reconnect >>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from job >>>> monitoring >>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting >>>> hosts suitable for reconnect >>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>> clusters already owned by this management server >>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>> hosts for clusters already owned by this management server >>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>> clusters not owned by any management server >>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl] >>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>> hosts for clusters not owned by any management server >>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET >>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>> 3 >>>> f >>>> 1 >>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>> 9 >>>> 9 >>>> 3 >>>> 85 >>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl] >>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to >>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END=== >>>> 192.168.34.1 >>>> -- GET >>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>> 3 >>>> f >>>> 1 >>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>> 9 >>>> 9 >>>> 3 >>>> 85 >>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl] >>>> (AgentManager-Handler-2:null) Ping from 2 >>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl] >>>> (AgentManager-Handler-7:null) Ping from 9 >>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache] >>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current status >>>> on >>>> 4(SeSolXS02) >>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache] >>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current status >>>> on >>>> 1(SeSolXS01) >>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache] >>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current status >>>> on >>>> 5(SeSolXS03) >>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache] >>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status >>>> on >>>> 6(SeSolXS04) >>>> >>>> >>>> >>>> >>>> >>>> /Sonali >>>> >>>> -----Original Message----- >>>> From: Gopalakrishnan S [mailto:[email protected]] >>>> Sent: Friday, February 20, 2015 7:12 PM >>>> To: [email protected] >>>> Subject: Re: unable to remove NIC from instance >>>> >>>> Hi Sonali, >>>> >>>> It may be still routing that shared guest network. Try to restart your >>>> virutal router and virtual machine or shutdown your VM before removing >>>> that NIC. let me know the logs when you remove that NIC. >>>> >>>> Thank You. >>>> Gopalakrishnan.S >>>> Cloud Engineer - Fogpanel >>>> >>>> ----- Original Message ----- >>>> From: Sonali Jadhav >>>> To: [email protected] >>>> Sent: Friday, February 20, 2015 7:06 PM >>>> Subject: unable to remove NIC from instance >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> I have one linux instance to which I had added "Shared guest network" >>>> NIC. Then I added "isolated guest network" nic to it. Made "isolated guest >>>> network" NIC as default and tried to remove "Shared guest network" NIC, >>>> but its failing with null exception. I have attached logs. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> /sonali >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Daan >>> >>> >>> >>> >> >> >> >> -- >> Daan >> >> > > > > -- > Daan > > -- Daan
