Hi Daan, Super thanks for reply. I am running version 4.4.2
/Sonali -----Original Message----- From: Daan Hoogland [mailto:[email protected]] Sent: Tuesday, February 24, 2015 1:05 PM To: [email protected] Subject: Re: unable to remove NIC from instance It seems to me you are a victim of overzealous exception handling. I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running. On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <[email protected]> wrote: > Hi Daan, > > I have checked logs form starting of api call, and I have not found > anything special, But apart from error, I found one warning, > > 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] > (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable to > find lock for the key vm_instance24 and thread id 137380473 > > Don’t know meaning of this. > > Here are my logs, > > 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response > Received: > 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request] > (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: > Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: > 10, > [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_ > vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx: > true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","r-6 > -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"vir > idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":" > viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"viridian > :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"virid > ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true;ac > pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;api > c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"resul > t":true,"wait":0}}] } > 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing > request > 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization 0.11499999999999999 > 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received: > 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId > 249373720971888: Resp: Routing to peer > 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing > request > 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588 > 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999 > 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received: > 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId > 249373720971888: Resp: Routing to peer > 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing > request > 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-192:ctx-61e81d25) Vm cpu utilization 0.13343749999999996 > 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received: > 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId > 249373720971888: Resp: Routing to peer > 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet] > (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET > command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iElw > 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df07 > 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 > 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526, > details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType: > None, instanceId: null, cmd: > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, > cmdInfo: > {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","c > txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d97 > 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0753 > 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","c > mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d979 > f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccountI > d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: IN_PROGRESS, > processStatus: 0, resultCode: 0, result: null, initMsid: > 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: > null, created: null} > 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job > monitoring > 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO > {id:526, userId: 2, accountId: 2, instanceType: None, instanceId: > null, cmd: > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, > cmdInfo: > {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","c > txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d97 > 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0753 > 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","c > mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d979 > f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccountI > d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: IN_PROGRESS, > processStatus: 0, resultCode: 0, result: null, initMsid: > 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: > null, created: null} > 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet] > (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END=== 192.168.45.32 > -- GET > command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iElw > 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df07 > 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 > 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console proxy > 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync job-527 > execution on object VmWorkJobQueue.24 > 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] > (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable to > find lock for the key vm_instance24 and thread id 137380473 > 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary > storage VM > 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] > (StatsCollector-4:ctx-4a027738) HostStatsCollector is running... > 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing > request > 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received: > 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request] > (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received: > { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, { > GetHostStatsAnswer } } > 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing > request > 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received: > 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request] > (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received: { > Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, { > GetHostStatsAnswer } } > 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing > request > 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received: > 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request] > (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received: > { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { > GetHostStatsAnswer } } > 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet] > (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET > command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8a > &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469340 > 6327 > 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing > request > 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received: > 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request] > (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received: { > Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, { > GetHostStatsAnswer } } > 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet] > (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END=== 192.168.45.32 > -- GET > command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8a > &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469340 > 6327 > 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception while > executing > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin > java.lang.RuntimeException: Job failed due to exception null > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526, > jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > [],"errorcode":530,"errortext":"Job failed due to exception null"} > 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-118:ctx-adab6031 job-526) Done executing > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin > for job-526 > 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job > monitoring > > > /Sonali > > -----Original Message----- > From: Daan Hoogland [mailto:[email protected]] > Sent: Monday, February 23, 2015 2:42 PM > To: [email protected] > Subject: Re: unable to remove NIC from instance > > Sonali, > > this looks like a consequential error. You need to look further back into the > log to find the root cause. The start job is the a query for results, please > find the start for the actual api command. > > On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <[email protected]> > wrote: >> I am trying to remove "Shared guest network". These are new logs, >> >> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >> (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status. >> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 -- GET >> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94c >> f >> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246707 >> 3 >> 8796 >> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END=== >> 115.114.134.114 -- GET >> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94c >> f >> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246707 >> 3 >> 8796 >> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception >> while executing >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >> java.lang.RuntimeException: Job failed due to exception null >> at >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async job-499, >> jobStatus: FAILED, resultCode: 530, result: >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >> [],"errorcode":530,"errortext":"Job failed due to exception null"} >> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing >> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >> for job-499 >> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from job >> monitoring >> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] >> (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running... >> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 -- GET >> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94c >> f >> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246707 >> 4 >> 1797 >> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet] >> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END=== >> 115.114.134.114 -- GET >> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94c >> f >> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246707 >> 4 >> 1797 >> >> >> /Sonali >> >> -----Original Message----- >> From: Sonali Jadhav [mailto:[email protected]] >> Sent: Monday, February 23, 2015 10:25 AM >> To: [email protected] >> Subject: RE: unable to remove NIC from instance >> >> Hi, >> >> Yes I have tried that, and issue is still same. Same logs. >> >> /Sonali >> >> -----Original Message----- >> From: Daan Hoogland [mailto:[email protected]] >> Sent: Sunday, February 22, 2015 10:57 PM >> To: [email protected] >> Subject: Re: unable to remove NIC from instance >> >> Sonil, Did you try what Gopalakrishnan suggested? >> >> Also can you expand on the version of cloudstack you are running? >> >> thanks, >> >> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <[email protected]> >> wrote: >>> Anyone help? >>> /Sonali >>> >>> >>> ---- Sonali Jadhav wrote ---- >>> >>> Anyone? >>> >>> Sent from my Sony Xperia™ smartphone >>> >>> >>> ---- Sonali Jadhav wrote ---- >>> >>> Here are logs, >>> >>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl] >>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495: >>> { >>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010, >>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}} >>> ] >>> } >>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl] >>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work >>> job-483 >>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction] >>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction: >>> Time = 2 Name = Vm-Operations-Cleanup-1; called by >>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-Tra >>> n >>> s >>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Refle >>> c >>> t >>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:9 >>> 1 >>> - >>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204 >>> - >>> $ >>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Clean >>> u >>> p >>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultManage >>> d >>> C >>> ontext$1.call:56 >>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl] >>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due to >>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: >>> com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE >>> async_job.id= 483 >>> at >>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178) >>> at >>> org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149) >>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >>> at >>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >>> at >>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >>> at >>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source) >>> at >>> com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: >>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: >>> Cannot delete or update a parent row: a foreign key constraint fails >>> (`cloud`.`async_job_join_map`, CONSTRAINT >>> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES >>> `async_job` (`id`)) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) >>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) >>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) >>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625) >>> at >>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333) >>> at >>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318) >>> at >>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>> at >>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>> at >>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169) >>> ... 26 more >>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET >>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab3 >>> f >>> 1 >>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443869 >>> 6 >>> 3 >>> 84 >>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl] >>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to >>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END=== 192.168.34.1 >>> -- GET >>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab3 >>> f >>> 1 >>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443869 >>> 6 >>> 3 >>> 84 >>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher] >>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception >>> while executing >>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd >>> java.lang.RuntimeException: Job failed due to exception null >>> at >>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async job-482, >>> jobStatus: FAILED, resultCode: 530, result: >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing >>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for >>> job-482 >>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts suitable >>> for reconnect >>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from job >>> monitoring >>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting hosts >>> suitable for reconnect >>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>> clusters already owned by this management server >>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring hosts >>> for clusters already owned by this management server >>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>> clusters not owned by any management server >>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl] >>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring hosts >>> for clusters not owned by any management server >>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET >>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab3 >>> f >>> 1 >>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443869 >>> 9 >>> 3 >>> 85 >>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl] >>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to >>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END=== 192.168.34.1 >>> -- GET >>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab3 >>> f >>> 1 >>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443869 >>> 9 >>> 3 >>> 85 >>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl] >>> (AgentManager-Handler-2:null) Ping from 2 >>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl] >>> (AgentManager-Handler-7:null) Ping from 9 >>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache] >>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current status >>> on >>> 4(SeSolXS02) >>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache] >>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current status >>> on >>> 1(SeSolXS01) >>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache] >>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current status >>> on >>> 5(SeSolXS03) >>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache] >>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status on >>> 6(SeSolXS04) >>> >>> >>> >>> >>> >>> /Sonali >>> >>> -----Original Message----- >>> From: Gopalakrishnan S [mailto:[email protected]] >>> Sent: Friday, February 20, 2015 7:12 PM >>> To: [email protected] >>> Subject: Re: unable to remove NIC from instance >>> >>> Hi Sonali, >>> >>> It may be still routing that shared guest network. Try to restart your >>> virutal router and virtual machine or shutdown your VM before removing that >>> NIC. let me know the logs when you remove that NIC. >>> >>> Thank You. >>> Gopalakrishnan.S >>> Cloud Engineer - Fogpanel >>> >>> ----- Original Message ----- >>> From: Sonali Jadhav >>> To: [email protected] >>> Sent: Friday, February 20, 2015 7:06 PM >>> Subject: unable to remove NIC from instance >>> >>> >>> Hi, >>> >>> >>> >>> I have one linux instance to which I had added "Shared guest network" >>> NIC. Then I added "isolated guest network" nic to it. Made "isolated guest >>> network" NIC as default and tried to remove "Shared guest network" NIC, but >>> its failing with null exception. I have attached logs. >>> >>> >>> >>> Thanks, >>> >>> /sonali >>> >>> >>> >>> >>> >>> >> >> >> >> -- >> Daan >> >> >> >> > > > > -- > Daan > > -- Daan
