@Somesh: added ticket https://issues.apache.org/jira/browse/CLOUDSTACK-8328 @Daan: should I add ticket for overzealous exception handling ?
/Sonali -----Original Message----- From: Somesh Naidu [mailto:[email protected]] Sent: Tuesday, March 10, 2015 8:00 PM To: [email protected] Subject: RE: unable to remove NIC from instance It could be a bug in that case. I guess you could file it on the tracker for further investigation. Regards, Somesh -----Original Message----- From: Sonali Jadhav [mailto:[email protected]] Sent: Tuesday, March 10, 2015 8:21 AM To: [email protected] Subject: RE: unable to remove NIC from instance Hi somesh, I just cross checked that, service_offering where id=14, is nothing but "Custom" compute offering I created. That’s why ram_size value is set to NULL. So basically I cant/don’t want to change value of ram_size service offering, Is this bug? /Sonali -----Original Message----- From: Sonali Jadhav [mailto:[email protected]] Sent: Tuesday, March 10, 2015 5:38 PM To: [email protected] Subject: RE: unable to remove NIC from instance Hi Somesh, I have very weird values in Service offering table, mysql> select * from service_offering; +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+ | id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner | +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+ | 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL | | 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL | | 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL | | 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL | | 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL | | 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL | | 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL | | 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | | 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL | +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+ 14 rows in set (0.00 sec) mysql> any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came. /Sonali -----Original Message----- From: Somesh Naidu [mailto:[email protected]] Sent: Thursday, March 5, 2015 8:49 PM To: [email protected] Subject: RE: unable to remove NIC from instance That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC. Regards, Somesh -----Original Message----- From: Sonali Jadhav [mailto:[email protected]] Sent: Thursday, March 05, 2015 12:20 AM To: [email protected] Subject: RE: unable to remove NIC from instance Hi Somesh, Here is output, mysql> select ram_size from service_offering where id=14; +----------+ | ram_size | +----------+ | NULL | +----------+ 1 row in set (0.00 sec) Thanks, /Sonali -----Original Message----- From: Somesh Naidu [mailto:[email protected]] Sent: Thursday, March 5, 2015 2:19 AM To: [email protected] Subject: RE: unable to remove NIC from instance Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM. Could you send me the output of the following SQL: select ram_size from service_offering where id=14; Somesh CloudPlatform Escalations Citrix Systems, Inc. -----Original Message----- From: Sonali Jadhav [mailto:[email protected]] Sent: Tuesday, March 03, 2015 6:39 AM To: [email protected] Subject: RE: unable to remove NIC from instance Hi Somesh, Have you looked at the logs and query results I sent, what do you think? /Sonali -----Original Message----- From: Daan Hoogland [mailto:[email protected]] Sent: Thursday, February 26, 2015 1:42 PM To: [email protected] Subject: Re: unable to remove NIC from instance As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise. If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed. On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <[email protected]> wrote: > Hi Somesh n Daan, > > Thanks for reply. > > Here are logs from point I logged to CloudStack till the error > occurred http://pastebin.com/j1rfEq5s > > And here is sql query output, > > mysql> select * from vm_instance where instance_name="i-9-24-VM" \G > *************************** 1. row *************************** > id: 24 > name: linuxvm > uuid: b5c535b4-10c4-44a9-b117-6df07537434c > instance_name: i-9-24-VM > state: Stopped > vm_template_id: 202 > guest_os_id: 201 > private_mac_address: 02:00:3e:c4:00:01 > private_ip_address: 10.1.1.14 > pod_id: 1 > data_center_id: 1 > host_id: NULL > last_host_id: 1 > proxy_id: 1 > proxy_assign_time: 2015-02-20 11:25:18 > vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI= > ha_enabled: 0 > limit_cpu_use: 0 > update_count: 25 > update_time: 2015-02-23 06:26:46 > created: 2015-02-20 11:24:26 > removed: NULL > type: User > vm_type: User > account_id: 9 > domain_id: 5 > service_offering_id: 14 > reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018 > hypervisor_type: XenServer > disk_offering_id: 6 > owner: 9 > host_name: linuxvm > display_name: linuxvm > desired_state: NULL > dynamically_scalable: 0 > display_vm: 1 > power_state: NULL > power_state_update_time: NULL > power_state_update_count: 0 > power_host: NULL > 1 row in set (0.00 sec) > > mysql> > > mysql> select * from nics where instance_id=24 \G > *************************** 1. row *************************** > id: 60 > uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f > instance_id: 24 > mac_address: 06:by:bg:00:01:33 > ip4_address: 1.1.1.1 > netmask: 255.255.0.0 > gateway: 1.1.1.1 > ip_type: Ip4 > broadcast_uri: vlan://201 > network_id: 207 > mode: Dhcp > state: Allocated > strategy: Create > reserver_name: DirectNetworkGuru > reservation_id: NULL > device_id: 1 > update_time: 2015-02-23 07:25:40 > isolation_uri: vlan://201 > ip6_address: NULL > default_nic: 0 > vm_type: User > created: 2015-02-20 11:24:26 > removed: NULL > ip6_gateway: NULL > ip6_cidr: NULL > secondary_ip: 0 > display_nic: 1 > *************************** 2. row *************************** > id: 61 > uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4 > instance_id: 24 > mac_address: 02:00:3f:a4:00:01 > ip4_address: 10.1.1.14 > netmask: 255.255.255.0 > gateway: 10.1.1.1 > ip_type: Ip4 > broadcast_uri: NULL > network_id: 211 > mode: Dhcp > state: Allocated > strategy: Start > reserver_name: ExternalGuestNetworkGuru > reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018 > device_id: 0 > update_time: 2015-02-23 07:25:41 > isolation_uri: NULL > ip6_address: NULL > default_nic: 1 > vm_type: User > created: 2015-02-20 13:21:07 > removed: NULL > ip6_gateway: NULL > ip6_cidr: NULL > secondary_ip: 0 > display_nic: 1 > 2 rows in set (0.00 sec) > > mysql> > > Let me know if you find something from this. > > /Sonali > > -----Original Message----- > From: Somesh Naidu [mailto:[email protected]] > Sent: Wednesday, February 25, 2015 11:07 PM > To: [email protected] > Subject: RE: unable to remove NIC from instance > > Sonali, > > While this could be fixed in the DB, as Daan mentioned, it is not > recommended. The primary table that needs to be update is the nics table but > in addition, relevant tables like user_ip_address, etc need to be updated. > There may also be impact on usage. Moreover, updating the DB won’t take care > of removing the nic on the hypervisor/VM. > > A better way to deal with this is to resolve the original error that is > preventing CCP from removing the nic. > > I have looked at the information (log snippet) you have shared. As Daan > pointed out, these logs contain consequential errors where > queryAsyncJobResult API is failing to get the result of a job (happens to be > the RemoveNicFromVMCmdByAdmin API call that you are interested in) that > relate to some trouble with the job framework. But this is not why the > original remove nic command failed. > > What you can do is, > 1. Note the time of the last line of log in mgmt. server (just do a tail on > mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs > starting from time noted in step #1 4. In addition provide details of the VM, > a. select * from vm_instance where id=<VM_ID>; > b. select * from nics where instance_id=<VM_ID>; > > If I have this information I can try and identify why the original operation > is failing. > > Somesh > CloudPlatform Escalations > Citrix Systems, Inc. > > -----Original Message----- > From: Daan Hoogland [mailto:[email protected]] > Sent: Wednesday, February 25, 2015 7:47 AM > To: [email protected] > Subject: Re: unable to remove NIC from instance > > I have no experience with this. It could be fixed in the db probably > but that is always a bad advice > 1 - coming from a developer > 2 - for application in a production system > > If you are very familiar with the datamodel you might try to, I would only > try to do it in a test environment (or at least first) Also I do not have the > db change that would help you by head. > > > On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <[email protected]> > wrote: >> Hi Daan, >> >> Is there any manual way to fix it temporary ? I have production customer who >> is facing this issue. >> >> /Sonali >> >> -----Original Message----- >> From: Sonali Jadhav [mailto:[email protected]] >> Sent: Tuesday, February 24, 2015 1:27 PM >> To: [email protected] >> Subject: RE: unable to remove NIC from instance >> >> I am running xenserver 6.2, I have 1 cluster with 4 Hosts. >> >> /Sonali >> >> -----Original Message----- >> From: Daan Hoogland [mailto:[email protected]] >> Sent: Tuesday, February 24, 2015 1:22 PM >> To: [email protected] >> Subject: Re: unable to remove NIC from instance >> >> Ok, thanks, we are running a custom 4.4.2 version we had to make due to >> problems with the stock 4.4.2. I will look into making another release >> candidate for 4.4.3 and have a look at the code to see if I can find a cause >> to your problem. The last 4.4.3 release candidate I made was largely ignored >> and therefore never released. It also has a problem with creating new >> volumes on vmware which is not solved yet. >> What type of hypervisors are you running? >> >> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <[email protected]> >> wrote: >>> Hi Daan, >>> >>> Super thanks for reply. I am running version 4.4.2 >>> >>> /Sonali >>> >>> -----Original Message----- >>> From: Daan Hoogland [mailto:[email protected]] >>> Sent: Tuesday, February 24, 2015 1:05 PM >>> To: [email protected] >>> Subject: Re: unable to remove NIC from instance >>> >>> It seems to me you are a victim of overzealous exception handling. >>> I've seen that happen in the async job handling. The job dispatching or >>> construction has an exception that is not handled but caught and ignored, >>> it end and the async job is not removed from the queue. I'd like to look at >>> the code near the stack trace but for that I need the exact version of >>> Cloudstack you are running. >>> >>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <[email protected]> >>> wrote: >>>> Hi Daan, >>>> >>>> I have checked logs form starting of api call, and I have not found >>>> anything special, But apart from error, I found one warning, >>>> >>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >>>> to find lock for the key vm_instance24 and thread id 137380473 >>>> >>>> Don’t know meaning of this. >>>> >>>> Here are my logs, >>>> >>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response >>>> Received: >>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request] >>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: >>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: >>>> 10, >>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1," >>>> _ >>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx: >>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true"," >>>> r >>>> - >>>> 6 >>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":" >>>> v >>>> i >>>> r >>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":" >>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid >>>> i >>>> a >>>> n >>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi >>>> r >>>> i >>>> d >>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true >>>> ; >>>> a >>>> c >>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1; >>>> a >>>> p >>>> i >>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re >>>> s >>>> u >>>> l >>>> t":true,"wait":0}}] } >>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing >>>> request >>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization >>>> 0.11499999999999999 >>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received: >>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId >>>> 249373720971888: Resp: Routing to peer >>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing >>>> request >>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588 >>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999 >>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received: >>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId >>>> 249373720971888: Resp: Routing to peer >>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing >>>> request >>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization >>>> 0.13343749999999996 >>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response >>>> Received: >>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId >>>> 249373720971888: Resp: Routing to peer >>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET >>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i >>>> E >>>> l >>>> w >>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d >>>> f >>>> 0 >>>> 7 >>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526, >>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType: >>>> None, instanceId: null, cmd: >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi >>>> n >>>> , >>>> cmdInfo: >>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >>>> c >>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977 >>>> d >>>> 9 >>>> 7 >>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0 >>>> 7 >>>> 5 >>>> 3 >>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >>>> c >>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d >>>> 9 >>>> 7 >>>> 9 >>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou >>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >>>> IN_PROGRESS, >>>> processStatus: 0, resultCode: 0, result: null, initMsid: >>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >>>> null, created: null} >>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job >>>> monitoring >>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO >>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId: >>>> null, cmd: >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi >>>> n >>>> , >>>> cmdInfo: >>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d"," >>>> c >>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977 >>>> d >>>> 9 >>>> 7 >>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0 >>>> 7 >>>> 5 >>>> 3 >>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c"," >>>> c >>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d >>>> 9 >>>> 7 >>>> 9 >>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou >>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status: >>>> IN_PROGRESS, >>>> processStatus: 0, resultCode: 0, result: null, initMsid: >>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: >>>> null, created: null} >>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END=== >>>> 192.168.45.32 >>>> -- GET >>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i >>>> E >>>> l >>>> w >>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d >>>> f >>>> 0 >>>> 7 >>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955 >>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl] >>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console >>>> proxy >>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync >>>> job-527 execution on object VmWorkJobQueue.24 >>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2] >>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable >>>> to find lock for the key vm_instance24 and thread id 137380473 >>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] >>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary >>>> storage VM >>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] >>>> (StatsCollector-4:ctx-4a027738) HostStatsCollector is running... >>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing >>>> request >>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received: >>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request] >>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received: >>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, { >>>> GetHostStatsAnswer } } >>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing >>>> request >>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received: >>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request] >>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received: >>>> { >>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, { >>>> GetHostStatsAnswer } } >>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing >>>> request >>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response >>>> Received: >>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request] >>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received: >>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, { >>>> GetHostStatsAnswer } } >>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET >>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb >>>> b >>>> 8 >>>> a >>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469 >>>> 3 >>>> 4 >>>> 0 >>>> 6327 >>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing >>>> request >>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] >>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received: >>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request] >>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received: >>>> { >>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, { >>>> GetHostStatsAnswer } } >>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet] >>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END=== >>>> 192.168.45.32 >>>> -- GET >>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb >>>> b >>>> 8 >>>> a >>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469 >>>> 3 >>>> 4 >>>> 0 >>>> 6327 >>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception >>>> while executing >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi >>>> n >>>> java.lang.RuntimeException: Job failed due to exception null >>>> at >>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>> at >>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>> at >>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526, >>>> jobStatus: FAILED, resultCode: 530, result: >>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing >>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi >>>> n >>>> for job-526 >>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job >>>> monitoring >>>> >>>> >>>> /Sonali >>>> >>>> -----Original Message----- >>>> From: Daan Hoogland [mailto:[email protected]] >>>> Sent: Monday, February 23, 2015 2:42 PM >>>> To: [email protected] >>>> Subject: Re: unable to remove NIC from instance >>>> >>>> Sonali, >>>> >>>> this looks like a consequential error. You need to look further back into >>>> the log to find the root cause. The start job is the a query for results, >>>> please find the start for the actual api command. >>>> >>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <[email protected]> >>>> wrote: >>>>> I am trying to remove "Shared guest network". These are new logs, >>>>> >>>>> 2015-02-23 06:53:59,737 DEBUG >>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl] >>>>> (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR >>>>> status. >>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 -- >>>>> GET >>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b >>>>> 9 >>>>> 4 >>>>> c >>>>> f >>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246 >>>>> 7 >>>>> 0 >>>>> 7 >>>>> 3 >>>>> 8796 >>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END=== >>>>> 115.114.134.114 -- GET >>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b >>>>> 9 >>>>> 4 >>>>> c >>>>> f >>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246 >>>>> 7 >>>>> 0 >>>>> 7 >>>>> 3 >>>>> 8796 >>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception >>>>> while executing >>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm >>>>> i >>>>> n >>>>> java.lang.RuntimeException: Job failed due to exception null >>>>> at >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>>> at >>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>> at >>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>> at >>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>> at >>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async >>>>> job-499, >>>>> jobStatus: FAILED, resultCode: 530, result: >>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"} >>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing >>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm >>>>> i >>>>> n >>>>> for job-499 >>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from >>>>> job monitoring >>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] >>>>> (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running... >>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 -- >>>>> GET >>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b >>>>> 9 >>>>> 4 >>>>> c >>>>> f >>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246 >>>>> 7 >>>>> 0 >>>>> 7 >>>>> 4 >>>>> 1797 >>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet] >>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END=== >>>>> 115.114.134.114 -- GET >>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b >>>>> 9 >>>>> 4 >>>>> c >>>>> f >>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246 >>>>> 7 >>>>> 0 >>>>> 7 >>>>> 4 >>>>> 1797 >>>>> >>>>> >>>>> /Sonali >>>>> >>>>> -----Original Message----- >>>>> From: Sonali Jadhav [mailto:[email protected]] >>>>> Sent: Monday, February 23, 2015 10:25 AM >>>>> To: [email protected] >>>>> Subject: RE: unable to remove NIC from instance >>>>> >>>>> Hi, >>>>> >>>>> Yes I have tried that, and issue is still same. Same logs. >>>>> >>>>> /Sonali >>>>> >>>>> -----Original Message----- >>>>> From: Daan Hoogland [mailto:[email protected]] >>>>> Sent: Sunday, February 22, 2015 10:57 PM >>>>> To: [email protected] >>>>> Subject: Re: unable to remove NIC from instance >>>>> >>>>> Sonil, Did you try what Gopalakrishnan suggested? >>>>> >>>>> Also can you expand on the version of cloudstack you are running? >>>>> >>>>> thanks, >>>>> >>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav >>>>> <[email protected]> wrote: >>>>>> Anyone help? >>>>>> /Sonali >>>>>> >>>>>> >>>>>> ---- Sonali Jadhav wrote ---- >>>>>> >>>>>> Anyone? >>>>>> >>>>>> Sent from my Sony Xperia™ smartphone >>>>>> >>>>>> >>>>>> ---- Sonali Jadhav wrote ---- >>>>>> >>>>>> Here are logs, >>>>>> >>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl] >>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495: >>>>>> { >>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010, >>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait": >>>>>> 0 >>>>>> } >>>>>> } >>>>>> ] >>>>>> } >>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl] >>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work >>>>>> job-483 >>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction] >>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction: >>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by >>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843- >>>>>> T >>>>>> r >>>>>> a >>>>>> n >>>>>> s >>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re >>>>>> f >>>>>> l >>>>>> e >>>>>> c >>>>>> t >>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke: >>>>>> 9 >>>>>> 1 >>>>>> - >>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke: >>>>>> 2 >>>>>> 0 >>>>>> 4 >>>>>> - >>>>>> $ >>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl >>>>>> e >>>>>> a >>>>>> n >>>>>> u >>>>>> p >>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan >>>>>> a >>>>>> g >>>>>> e >>>>>> d >>>>>> C >>>>>> ontext$1.call:56 >>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl] >>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due >>>>>> to >>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: >>>>>> com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE >>>>>> async_job.id= 483 >>>>>> at >>>>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178) >>>>>> at >>>>>> org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149) >>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> at >>>>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >>>>>> at >>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >>>>>> at >>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >>>>>> at >>>>>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >>>>>> at >>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >>>>>> at >>>>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >>>>>> at >>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >>>>>> at >>>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown >>>>>> Source) >>>>>> at >>>>>> com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>> Caused by: >>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: >>>>>> Cannot delete or update a parent row: a foreign key constraint fails >>>>>> (`cloud`.`async_job_join_map`, CONSTRAINT >>>>>> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) >>>>>> REFERENCES `async_job` (`id`)) >>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>>> Method) >>>>>> at >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>>> at >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>> at >>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) >>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) >>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) >>>>>> at >>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333) >>>>>> at >>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318) >>>>>> at >>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>>>> at >>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>>>> at >>>>>> com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169) >>>>>> ... 26 more >>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet] >>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET >>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0 >>>>>> a >>>>>> b >>>>>> 3 >>>>>> f >>>>>> 1 >>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443 >>>>>> 8 >>>>>> 6 >>>>>> 9 >>>>>> 6 >>>>>> 3 >>>>>> 84 >>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl] >>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to >>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet] >>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END=== >>>>>> 192.168.34.1 >>>>>> -- GET >>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0 >>>>>> a >>>>>> b >>>>>> 3 >>>>>> f >>>>>> 1 >>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443 >>>>>> 8 >>>>>> 6 >>>>>> 9 >>>>>> 6 >>>>>> 3 >>>>>> 84 >>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception >>>>>> while executing >>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd >>>>>> java.lang.RuntimeException: Job failed due to exception null >>>>>> at >>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) >>>>>> at >>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>> at >>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>> at >>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async >>>>>> job-482, >>>>>> jobStatus: FAILED, resultCode: 530, result: >>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": >>>>>> [],"errorcode":530,"errortext":"Job failed due to exception >>>>>> null"} >>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing >>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for >>>>>> job-482 >>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts >>>>>> suitable for reconnect >>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from >>>>>> job monitoring >>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting >>>>>> hosts suitable for reconnect >>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>>>> clusters already owned by this management server >>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>>>> hosts for clusters already owned by this management server >>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for >>>>>> clusters not owned by any management server >>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl] >>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring >>>>>> hosts for clusters not owned by any management server >>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet] >>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET >>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0 >>>>>> a >>>>>> b >>>>>> 3 >>>>>> f >>>>>> 1 >>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443 >>>>>> 8 >>>>>> 6 >>>>>> 9 >>>>>> 9 >>>>>> 3 >>>>>> 85 >>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl] >>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to >>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to >>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker >>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet] >>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END=== >>>>>> 192.168.34.1 >>>>>> -- GET >>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0 >>>>>> a >>>>>> b >>>>>> 3 >>>>>> f >>>>>> 1 >>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443 >>>>>> 8 >>>>>> 6 >>>>>> 9 >>>>>> 9 >>>>>> 3 >>>>>> 85 >>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl] >>>>>> (AgentManager-Handler-2:null) Ping from 2 >>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl] >>>>>> (AgentManager-Handler-7:null) Ping from 9 >>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache] >>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current >>>>>> status on >>>>>> 4(SeSolXS02) >>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache] >>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current >>>>>> status on >>>>>> 1(SeSolXS01) >>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache] >>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current >>>>>> status on >>>>>> 5(SeSolXS03) >>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache] >>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status >>>>>> on >>>>>> 6(SeSolXS04) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> /Sonali >>>>>> >>>>>> -----Original Message----- >>>>>> From: Gopalakrishnan S [mailto:[email protected]] >>>>>> Sent: Friday, February 20, 2015 7:12 PM >>>>>> To: [email protected] >>>>>> Subject: Re: unable to remove NIC from instance >>>>>> >>>>>> Hi Sonali, >>>>>> >>>>>> It may be still routing that shared guest network. Try to restart your >>>>>> virutal router and virtual machine or shutdown your VM before removing >>>>>> that NIC. let me know the logs when you remove that NIC. >>>>>> >>>>>> Thank You. >>>>>> Gopalakrishnan.S >>>>>> Cloud Engineer - Fogpanel >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: Sonali Jadhav >>>>>> To: [email protected] >>>>>> Sent: Friday, February 20, 2015 7:06 PM >>>>>> Subject: unable to remove NIC from instance >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> I have one linux instance to which I had added "Shared guest network" >>>>>> NIC. Then I added "isolated guest network" nic to it. Made "isolated >>>>>> guest network" NIC as default and tried to remove "Shared guest network" >>>>>> NIC, but its failing with null exception. I have attached logs. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> /sonali >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Daan >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Daan >>>> >>>> >>> >>> >>> >>> -- >>> Daan >>> >>> >> >> >> >> -- >> Daan >> >> >> >> > > > > -- > Daan > > -- Daan
