I have no experience with this. It could be fixed in the db probably but that is always a bad advice 1 - coming from a developer 2 - for application in a production system
If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head. On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <[email protected]> wrote: > Hi Daan, > > Is there any manual way to fix it temporary ? I have production customer who > is facing this issue. > > /Sonali > > -----Original Message----- > From: Sonali Jadhav [mailto:[email protected]] > Sent: Tuesday, February 24, 2015 1:27 PM > To: [email protected] > Subject: RE: unable to remove NIC from instance > > I am running xenserver 6.2, I have 1 cluster with 4 Hosts. > > /Sonali > > -----Original Message----- > From: Daan Hoogland [mailto:[email protected]] > Sent: Tuesday, February 24, 2015 1:22 PM > To: [email protected] > Subject: Re: unable to remove NIC from instance > > Ok, thanks, we are running a custom 4.4.2 version we had to make due to > problems with the stock 4.4.2. I will look into making another release > candidate for 4.4.3 and have a look at the code to see if I can find a cause > to your problem. The last 4.4.3 release candidate I made was largely ignored > and therefore never released. It also has a problem with creating new volumes > on vmware which is not solved yet. > What type of hypervisors are you running? > > On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <[email protected]> > wrote: >> Hi Daan, >> >> Super thanks for reply. I am running version 4.4.2 >> >> /Sonali >> >> -----Original Message----- >> From: Daan Hoogland [mailto:[email protected]] >> Sent: Tuesday, February 24, 2015 1:05 PM >> To: [email protected] >> Subject: Re: unable to remove NIC from instance >> >> It seems to me you are a victim of overzealous exception handling. >> I've seen that happen in the async job handling. The job dispatching or >> construction has an exception that is not handled but caught and ignored, it >> end and the async job is not removed from the queue. I'd like to look at the >> code near the stack trace but for that I need the exact version of >> Cloudstack you are running. >> >> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <[email protected]> >> wrote: >>> Hi Daan, >>> >>> I have checked logs form starting of api call, and I have not found >>> anything special, But apart from error, I found one warning, >>> >>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >>> to find lock for the key vm_instance24 and thread id 137380473 >>> >>> Don’t know meaning of this. >>> >>> Here are my logs, >>> >>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response >>> Received: >>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request] >>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: >>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: >>> 10, >>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1," >>> _ >>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx: >>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","r- >>> 6 >>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"vi >>> r >>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":" >>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"viridia >>> n >>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"viri >>> d >>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true;a >>> c >>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;ap >>> i >>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"resu >>> l >>> t":true,"wait":0}}] } >>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing >>> request >>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase] >>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization 0.11499999999999999 >>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received: >>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId >>> 249373720971888: Resp: Routing to peer >>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing >>> request >>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588 >>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999 >>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received: >>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId >>> 249373720971888: Resp: Routing to peer >>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing >>> request >>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase] >>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization 0.13343749999999996 >>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received: >>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId >>> 249373720971888: Resp: Routing to peer >>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET >>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iEl >>> w >>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df0 >>> 7 >>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526, >>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType: >>> None, instanceId: null, cmd: >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, >>> cmdInfo: >>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >>> c >>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d9 >>> 7 >>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df075 >>> 3 >>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >>> c >>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d97 >>> 9 >>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccount >>> I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >>> IN_PROGRESS, >>> processStatus: 0, resultCode: 0, result: null, initMsid: >>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >>> null, created: null} >>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job >>> monitoring >>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO >>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId: >>> null, cmd: >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin, >>> cmdInfo: >>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >>> c >>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977d9 >>> 7 >>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df075 >>> 3 >>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >>> c >>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d97 >>> 9 >>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccount >>> I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >>> IN_PROGRESS, >>> processStatus: 0, resultCode: 0, result: null, initMsid: >>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >>> null, created: null} >>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END=== 192.168.45.32 >>> -- GET >>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4iEl >>> w >>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6df0 >>> 7 >>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl] >>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console proxy >>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync job-527 >>> execution on object VmWorkJobQueue.24 >>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >>> to find lock for the key vm_instance24 and thread id 137380473 >>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] >>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary >>> storage VM >>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] >>> (StatsCollector-4:ctx-4a027738) HostStatsCollector is running... >>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing >>> request >>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received: >>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request] >>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received: >>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, { >>> GetHostStatsAnswer } } >>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing >>> request >>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received: >>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request] >>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received: >>> { >>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, { >>> GetHostStatsAnswer } } >>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing >>> request >>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received: >>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request] >>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received: >>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { >>> GetHostStatsAnswer } } >>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET >>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8 >>> a >>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=14246934 >>> 0 >>> 6327 >>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing >>> request >>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] >>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received: >>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request] >>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received: >>> { >>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, { >>> GetHostStatsAnswer } } >>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet] >>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END=== 192.168.45.32 >>> -- GET >>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadbb8 >>> a >>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=14246934 >>> 0 >>> 6327 >>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception >>> while executing >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>> java.lang.RuntimeException: Job failed due to exception null >>> at >>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526, >>> jobStatus: FAILED, resultCode: 530, result: >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing >>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>> for job-526 >>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job >>> monitoring >>> >>> >>> /Sonali >>> >>> -----Original Message----- >>> From: Daan Hoogland [mailto:[email protected]] >>> Sent: Monday, February 23, 2015 2:42 PM >>> To: [email protected] >>> Subject: Re: unable to remove NIC from instance >>> >>> Sonali, >>> >>> this looks like a consequential error. You need to look further back into >>> the log to find the root cause. The start job is the a query for results, >>> please find the start for the actual api command. >>> >>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <[email protected]> >>> wrote: >>>> I am trying to remove "Shared guest network". These are new logs, >>>> >>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] >>>> (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status. >>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 -- GET >>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>>> c >>>> f >>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>>> 7 >>>> 3 >>>> 8796 >>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END=== >>>> 115.114.134.114 -- GET >>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>>> c >>>> f >>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>>> 7 >>>> 3 >>>> 8796 >>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception >>>> while executing >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>>> java.lang.RuntimeException: Job failed due to exception null >>>> at >>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async job-499, >>>> jobStatus: FAILED, resultCode: 530, result: >>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >>>> for job-499 >>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from job >>>> monitoring >>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] >>>> (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running... >>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 -- GET >>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>>> c >>>> f >>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>>> 7 >>>> 4 >>>> 1797 >>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END=== >>>> 115.114.134.114 -- GET >>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b94 >>>> c >>>> f >>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=1424670 >>>> 7 >>>> 4 >>>> 1797 >>>> >>>> >>>> /Sonali >>>> >>>> -----Original Message----- >>>> From: Sonali Jadhav [mailto:[email protected]] >>>> Sent: Monday, February 23, 2015 10:25 AM >>>> To: [email protected] >>>> Subject: RE: unable to remove NIC from instance >>>> >>>> Hi, >>>> >>>> Yes I have tried that, and issue is still same. Same logs. >>>> >>>> /Sonali >>>> >>>> -----Original Message----- >>>> From: Daan Hoogland [mailto:[email protected]] >>>> Sent: Sunday, February 22, 2015 10:57 PM >>>> To: [email protected] >>>> Subject: Re: unable to remove NIC from instance >>>> >>>> Sonil, Did you try what Gopalakrishnan suggested? >>>> >>>> Also can you expand on the version of cloudstack you are running? >>>> >>>> thanks, >>>> >>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav >>>> <[email protected]> wrote: >>>>> Anyone help? >>>>> /Sonali >>>>> >>>>> >>>>> ---- Sonali Jadhav wrote ---- >>>>> >>>>> Anyone? >>>>> >>>>> Sent from my Sony Xperia™ smartphone >>>>> >>>>> >>>>> ---- Sonali Jadhav wrote ---- >>>>> >>>>> Here are logs, >>>>> >>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl] >>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495: >>>>> { >>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010, >>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0} >>>>> } >>>>> ] >>>>> } >>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl] >>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work >>>>> job-483 >>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction] >>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction: >>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by >>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-Tr >>>>> a >>>>> n >>>>> s >>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Refl >>>>> e >>>>> c >>>>> t >>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke: >>>>> 9 >>>>> 1 >>>>> - >>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:20 >>>>> 4 >>>>> - >>>>> $ >>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Clea >>>>> n >>>>> u >>>>> p >>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultManag >>>>> e >>>>> d >>>>> C >>>>> ontext$1.call:56 >>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl] >>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due to >>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: >>>>> com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE >>>>> async_job.id= 483 >>>>> at >>>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178) >>>>> at >>>>> org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149) >>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> at >>>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >>>>> at >>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >>>>> at >>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >>>>> at >>>>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >>>>> at >>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >>>>> at >>>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >>>>> at >>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >>>>> at >>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown >>>>> Source) >>>>> at >>>>> com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>> at >>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> Caused by: >>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: >>>>> Cannot delete or update a parent row: a foreign key constraint fails >>>>> (`cloud`.`async_job_join_map`, CONSTRAINT >>>>> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) >>>>> REFERENCES `async_job` (`id`)) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040) >>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) >>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) >>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) >>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) >>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625) >>>>> at >>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119) >>>>> at >>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415) >>>>> at >>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333) >>>>> at >>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318) >>>>> at >>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>>> at >>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>>> at >>>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169) >>>>> ... 26 more >>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET >>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>>> 3 >>>>> f >>>>> 1 >>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>>> 9 >>>>> 6 >>>>> 3 >>>>> 84 >>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl] >>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to >>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END=== >>>>> 192.168.34.1 >>>>> -- GET >>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>>> 3 >>>>> f >>>>> 1 >>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>>> 9 >>>>> 6 >>>>> 3 >>>>> 84 >>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception >>>>> while executing >>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd >>>>> java.lang.RuntimeException: Job failed due to exception null >>>>> at >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>>> at >>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>> at >>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async job-482, >>>>> jobStatus: FAILED, resultCode: 530, result: >>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing >>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for >>>>> job-482 >>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts suitable >>>>> for reconnect >>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from job >>>>> monitoring >>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting >>>>> hosts suitable for reconnect >>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>>> clusters already owned by this management server >>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>>> hosts for clusters already owned by this management server >>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>>> clusters not owned by any management server >>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl] >>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>>> hosts for clusters not owned by any management server >>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET >>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>>> 3 >>>>> f >>>>> 1 >>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>>> 9 >>>>> 9 >>>>> 3 >>>>> 85 >>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl] >>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to >>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END=== >>>>> 192.168.34.1 >>>>> -- GET >>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0ab >>>>> 3 >>>>> f >>>>> 1 >>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=14244386 >>>>> 9 >>>>> 9 >>>>> 3 >>>>> 85 >>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl] >>>>> (AgentManager-Handler-2:null) Ping from 2 >>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl] >>>>> (AgentManager-Handler-7:null) Ping from 9 >>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current status >>>>> on >>>>> 4(SeSolXS02) >>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current status >>>>> on >>>>> 1(SeSolXS01) >>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current status >>>>> on >>>>> 5(SeSolXS03) >>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status >>>>> on >>>>> 6(SeSolXS04) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> /Sonali >>>>> >>>>> -----Original Message----- >>>>> From: Gopalakrishnan S [mailto:[email protected]] >>>>> Sent: Friday, February 20, 2015 7:12 PM >>>>> To: [email protected] >>>>> Subject: Re: unable to remove NIC from instance >>>>> >>>>> Hi Sonali, >>>>> >>>>> It may be still routing that shared guest network. Try to restart your >>>>> virutal router and virtual machine or shutdown your VM before removing >>>>> that NIC. let me know the logs when you remove that NIC. >>>>> >>>>> Thank You. >>>>> Gopalakrishnan.S >>>>> Cloud Engineer - Fogpanel >>>>> >>>>> ----- Original Message ----- >>>>> From: Sonali Jadhav >>>>> To: [email protected] >>>>> Sent: Friday, February 20, 2015 7:06 PM >>>>> Subject: unable to remove NIC from instance >>>>> >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> I have one linux instance to which I had added "Shared guest network" >>>>> NIC. Then I added "isolated guest network" nic to it. Made "isolated >>>>> guest network" NIC as default and tried to remove "Shared guest network" >>>>> NIC, but its failing with null exception. I have attached logs. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> /sonali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Daan >>>> >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Daan >>> >>> >> >> >> >> -- >> Daan >> >> > > > > -- > Daan > > > > -- Daan
