Diego, can you also provide output on mysql statements I've mentioned in my prev email?
-Alena. On 7/24/12 12:12 PM, "Diego Spinola Castro" <[email protected]> wrote: >OK: > >https://dl.dropbox.com/u/19653845/management-server.log.2012-07-19.gz > > >2012/7/24 Alena Prokharchyk <[email protected]> > >> Please Zip it, place it on some publicly available web host and send the >> link. >> >> Thank you, >> Alena. >> >> On 7/24/12 11:59 AM, "Diego Spinola Castro" <[email protected]> >> wrote: >> >> >What about management log. its 10mb , how do i send to you? >> > >> >mysql> select * from op_networks; >> >+-----+-----------------+------------+----+--------------+ >> >| id | mac_address_seq | nics_count | gc | check_for_gc | >> >+-----+-----------------+------------+----+--------------+ >> >| 200 | 2069 | 0 | 0 | 0 | >> >| 201 | 1 | 0 | 0 | 0 | >> >| 202 | 1062 | 0 | 0 | 0 | >> >| 203 | 1 | 0 | 0 | 0 | >> >| 204 | 224 | 5 | 1 | 1 | >> >| 205 | 13 | 0 | 1 | 0 | >> >| 206 | 8 | 3 | 1 | 1 | >> >| 207 | 12 | 0 | 1 | 0 | >> >| 208 | 7 | 2 | 1 | 1 | >> >| 209 | 10 | 0 | 1 | 0 | >> >| 210 | 5 | 0 | 1 | 0 | >> >| 211 | 10 | 1 | 1 | 1 | >> >| 212 | 14 | 0 | 1 | 0 | >> >| 213 | 8 | 6 | 1 | 1 | >> >| 214 | 5 | 0 | 1 | 0 | >> >| 215 | 4 | 0 | 1 | 0 | >> >| 216 | 3 | 0 | 1 | 0 | >> >| 217 | 3 | 0 | 1 | 0 | >> >| 218 | 4 | 3 | 1 | 1 | >> >| 219 | 3 | 0 | 1 | 0 | >> >| 220 | 10 | 5 | 1 | 1 | >> >| 221 | 6 | 6 | 1 | 1 | >> >| 222 | 16 | 5 | 1 | 1 | >> >| 223 | 14 | 1 | 1 | 1 | >> >| 224 | 30 | 1 | 1 | 1 | >> >| 225 | 3 | 0 | 1 | 0 | >> >| 226 | 8 | 11 | 1 | 1 | >> >| 227 | 3 | 0 | 1 | 0 | >> >| 228 | 3 | 2 | 1 | 1 | >> >| 229 | 4 | 3 | 1 | 1 | >> >| 230 | 3 | 0 | 1 | 0 | >> >| 231 | 6 | 7 | 1 | 1 | >> >| 232 | 26 | 24 | 1 | 1 | >> >| 233 | 4 | 2 | 1 | 1 | >> >| 234 | 7 | 9 | 1 | 1 | >> >| 235 | 3 | 0 | 1 | 0 | >> >| 236 | 3 | 2 | 1 | 1 | >> >| 237 | 4 | 0 | 1 | 0 | >> >| 238 | 9 | 7 | 1 | 1 | >> >| 240 | 4 | 3 | 1 | 1 | >> >| 241 | 5 | 3 | 1 | 1 | >> >| 242 | 3 | 3 | 1 | 1 | >> >| 243 | 30 | 4 | 1 | 1 | >> >| 244 | 3 | 0 | 1 | 0 | >> >| 245 | 4 | 3 | 1 | 1 | >> >| 246 | 4 | 3 | 1 | 1 | >> >| 247 | 4 | 3 | 1 | 1 | >> >| 248 | 3 | 2 | 1 | 1 | >> >| 249 | 3 | 2 | 1 | 1 | >> >| 250 | 3 | 3 | 1 | 1 | >> >| 251 | 7 | 0 | 1 | 0 | >> >| 252 | 4 | 6 | 1 | 1 | >> >| 253 | 3 | 2 | 1 | 1 | >> >| 254 | 13 | 6 | 1 | 1 | >> >| 255 | 7 | 3 | 1 | 1 | >> >| 256 | 3 | 1 | 1 | 1 | >> >| 257 | 6 | 4 | 1 | 1 | >> >| 258 | 1 | 6 | 1 | 1 | >> >| 259 | 4 | 3 | 1 | 1 | >> >| 260 | 5 | 2 | 1 | 1 | >> >| 261 | 3 | 3 | 1 | 1 | >> >| 262 | 5 | 0 | 1 | 0 | >> >| 263 | 5 | 1 | 1 | 1 | >> >| 264 | 3 | 0 | 1 | 0 | >> >| 265 | 3 | 1 | 1 | 1 | >> >| 266 | 4 | 1 | 1 | 1 | >> >| 267 | 3 | 1 | 1 | 1 | >> >| 268 | 6 | 2 | 1 | 1 | >> >| 269 | 3 | 1 | 1 | 1 | >> >| 270 | 4 | 0 | 1 | 0 | >> >| 271 | 3 | 1 | 1 | 1 | >> >| 272 | 3 | 1 | 1 | 1 | >> >| 273 | 3 | 1 | 1 | 1 | >> >| 274 | 10 | 2 | 1 | 1 | >> >| 275 | 3 | 1 | 1 | 1 | >> >+-----+-----------------+------------+----+--------------+ >> > >> >mysql> select * from op_networks where id=236; >> >+-----+-----------------+------------+----+--------------+ >> >| id | mac_address_seq | nics_count | gc | check_for_gc | >> >+-----+-----------------+------------+----+--------------+ >> >| 236 | 3 | 2 | 1 | 1 | >> >+-----+-----------------+------------+----+--------------+ >> > >> >mysql> select * from vm_instance where id in (select instance_id from >>nics >> >where removed is null and network_id=236)\G >> >*************************** 1. row *************************** >> > id: 348 >> > name: r-348-VM >> > instance_name: r-348-VM >> > state: Stopped >> > vm_template_id: 8 >> > guest_os_id: 15 >> >private_mac_address: 02:00:13:63:03:ed >> > private_ip_address: 10.16.140.70 >> > private_netmask: NULL >> > pod_id: 1 >> > data_center_id: 1 >> > host_id: NULL >> > last_host_id: 2 >> > proxy_id: 305 >> > proxy_assign_time: 2012-03-17 04:43:35 >> > vnc_password: d3cc123795551907 >> > ha_enabled: 1 >> > limit_cpu_use: 1 >> > update_count: 202 >> > update_time: 2012-07-17 15:54:54 >> > created: 2012-02-16 16:37:19 >> > removed: NULL >> > type: DomainRouter >> > vm_type: DomainRouter >> > account_id: 43 >> > domain_id: 1 >> >service_offering_id: 7 >> > reservation_id: 147062dc-3596-4799-a84e-667784810067 >> > hypervisor_type: VMware >> >1 row in set (0.00 sec) >> > >> > >> >2012/7/24 Alena Prokharchyk <[email protected]> >> > >> >> Error message: >> >> >> >> "Unable to remove the network id=236 as it has active Nics". >> >> >> >> >> >> Indicates that the User vm(s) (not router) fail to expunge during the >> >> account deletion, therefore the corresponding network failed to be >> >> removed. The account will be marked with Removed=someDate + >> >> cleanup_needed=1, and the recurring thread (runs every >> >> account.clenaup.interval defined in global config) will try to >>cleanup >> >>the >> >> account again and again, until it succeeds. >> >> >> >> >> >> >> >> To find out if this is the root cause, following is needed: >> >> >> >> * management server log >> >> * content of the op_networks table, the active nic info is stored >>there. >> >> Nics should be counted for non-expunged user vms only (select * from >> >> op_networks where id=236) >> >> * information about the user vms belonging to network id=236 (select >>* >> >> from vm_instance where id in (select instance_id from nics where >>removed >> >> is null and network_id=236)) >> >> >> >> -Alena. >> >> >> >> On 7/24/12 11:15 AM, "Edison Su" <[email protected]> wrote: >> >> >> >> >Sounds like a bug, the VR is not got destroyed during deleteAccount. >> >> >Could you help to fire a bug, and attach the whole mgt server log? >> >> > >> >> >> -----Original Message----- >> >> >> From: Diego Spinola Castro [mailto:[email protected]] >> >> >> Sent: Tuesday, July 24, 2012 10:03 AM >> >> >> To: [email protected] >> >> >> Subject: Orphaned VRouters >> >> >> >> >> >> Hi, i found some orphan vrouters and don't know why they still >> >>running >> >> >> even >> >> >> there's no account anymore (i deleted account). >> >> >> >> >> >> Bellow management.log >> >> >> >> >> >> >> >> >> 2012-07-19 12:39:06,437 DEBUG [cloud.async.AsyncJobManagerImpl] >> >> >> (http-12449-14:null) submit async job-6598, details: AsyncJobVO >> >> >> {id:6598, >> >> >> userId: 32, accountId: 2, sessionKey: null, instanceType: None, >> >> >> instanceId: >> >> >> null, cmd: com.cloud.api.commands.DeleteAccountCmd, cmdOriginator: >> >>null, >> >> >> cmdInfo: >> >> >> >> >>>>{"id":"43","response":"json","sessionkey":"7uBHMfOyDdSLdtSjPB9K0FcVloQ\ >> >> >> >> >>>>u003d","ctxUserId":"32","_":"1342707903173","ctxAccountId":"2","ctxStar >> >> >> tEventId":"35670"}, >> >> >> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, >> >> >> processStatus: 0, resultCode: 0, result: null, initMsid: >> >>144350996536, >> >> >> completeMsid: null, lastUpdated: null, lastPolled: null, created: >> >>null} >> >> >> 2012-07-19 12:39:06,438 DEBUG [cloud.async.AsyncJobManagerImpl] >> >> >> (Job-Executor-25:job-6598) Executing >> >> >> com.cloud.api.commands.DeleteAccountCmd for job-6598 >> >> >> 2012-07-19 12:39:06,443 DEBUG [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Access to Acct[43-300012_CUBOCC] >>granted >> >>to >> >> >> Acct[2-admin] by DomainChecker >> >> >> 2012-07-19 12:39:06,460 DEBUG [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Remove account 43 >> >> >> 2012-07-19 12:39:06,514 DEBUG [agent.transport.Request] >> >> >> (Job-Executor-25:job-6598) Seq 20-573703176: Sending { Cmd , >>MgmtId: >> >> >> 144350996536, via: 20, Ver: v1, Flags: 100011, >> >> >> >> >>>>[{"DeleteSnapshotsDirCommand":{"primaryStoragePoolNameLabel":"016b139d- >> >> >> a43e-32de-a330-e57a91388695","secondaryStoragePoolURL":"nfs:// >> >> >> >> >>>>10.16.140.11/export","dcId":1,"accountId":43,"volumeId":467,"volumePath >> >> >> ":"2c880c4443a44672971b964d0cfe725c","wait":0}}] >> >> >> } >> >> >> 2012-07-19 12:39:09,334 DEBUG [agent.transport.Request] >> >> >> (Job-Executor-25:job-6598) Seq 20-573703176: Received: { Ans: , >> >>MgmtId: >> >> >> 144350996536, via: 20, Ver: v1, Flags: 10, { Answer } } >> >> >> 2012-07-19 12:39:09,343 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details null >> >> >> 2012-07-19 12:39:09,343 DEBUG >>[storage.snapshot.SnapshotManagerImpl] >> >> >> (Job-Executor-25:job-6598) Deleted snapshotsDir for volume: 467 >>under >> >> >> account: 43 >> >> >> 2012-07-19 12:39:09,345 DEBUG [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Successfully deleted snapshots >>directories >> >> >> for >> >> >> all volumes under account 43 across all zones >> >> >> 2012-07-19 12:39:09,348 DEBUG [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Destroying # of vms (accountId=43): 1 >> >> >> 2012-07-19 12:39:09,348 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> >> (Job-Executor-25:job-6598) Stopped called on VM[User|i-43-347-VM] >>but >> >> >> the >> >> >> state is Destroyed >> >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.capacity.CapacityManagerImpl] >> >> >> (Job-Executor-25:job-6598) VM state transitted from :Destroyed to >> >> >> Expunging >> >> >> with event: ExpungeOperationvm's original host id: 4 new host id: >> >>null >> >> >> host >> >> >> id before state transition: null >> >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> >> (Job-Executor-25:job-6598) Destroying vm VM[User|i-43-347-VM] >> >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Cleaning network for vm: 347 >> >> >> 2012-07-19 12:39:09,369 DEBUG [cloud.storage.StorageManagerImpl] >> >> >> (Job-Executor-25:job-6598) Cleaning storage for vm: 347 >> >> >> 2012-07-19 12:39:09,401 DEBUG [cloud.storage.StorageManagerImpl] >> >> >> (Job-Executor-25:job-6598) Detaching Vol[467|vm=347|DATADISK] >> >> >> 2012-07-19 12:39:09,402 DEBUG [cloud.storage.StorageManagerImpl] >> >> >> (Job-Executor-25:job-6598) Detaching Vol[488|vm=347|DATADISK] >> >> >> 2012-07-19 12:39:09,404 DEBUG [cloud.storage.StorageManagerImpl] >> >> >> (Job-Executor-25:job-6598) Expunging Vol[392|vm=347|ROOT] >> >> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request] >> >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Sending { Cmd , >>MgmtId: >> >> >> 144350996536, via: 3, Ver: v1, Flags: 100111, >> >> >> [{"storage.DestroyCommand":{"vmName":"i-43-347- >> >> >> VM","volume":{"id":392,"name":"ROOT- >> >> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT- >> >> >> >> >>>>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage >> >> >> PoolUuid":"30b0d978-3291-3ce3-a06f- >> >> >> eea3ac2f4f51","deviceId":0},"wait":0}}] >> >> >> } >> >> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request] >> >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Executing: { Cmd , >> >>MgmtId: >> >> >> 144350996536, via: 3, Ver: v1, Flags: 100111, >> >> >> [{"storage.DestroyCommand":{"vmName":"i-43-347- >> >> >> VM","volume":{"id":392,"name":"ROOT- >> >> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT- >> >> >> >> >>>>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage >> >> >> PoolUuid":"30b0d978-3291-3ce3-a06f- >> >> >> eea3ac2f4f51","deviceId":0},"wait":0}}] >> >> >> } >> >> >> 2012-07-19 12:39:27,879 DEBUG [agent.transport.Request] >> >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Received: { Ans: , >> >>MgmtId: >> >> >> 144350996536, via: 3, Ver: v1, Flags: 110, { Answer } } >> >> >> 2012-07-19 12:39:27,879 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details Success >> >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.storage.StorageManagerImpl] >> >> >> (Job-Executor-25:job-6598) Volume successfully expunged from 206 >> >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> >> (Job-Executor-25:job-6598) Expunged VM[User|i-43-347-VM] >> >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.UserVmManagerImpl] >> >> >> (Job-Executor-25:job-6598) Starting cleaning up vm >> >>VM[User|i-43-347-VM] >> >> >> resources... >> >> >> 2012-07-19 12:39:27,947 DEBUG >>[network.firewall.FirewallManagerImpl] >> >> >> (Job-Executor-25:job-6598) No firewall rules are found for vm >>id=347 >> >> >> 2012-07-19 12:39:27,947 DEBUG [cloud.vm.UserVmManagerImpl] >> >> >> (Job-Executor-25:job-6598) Firewall rules are removed successfully >> >>as a >> >> >> part of vm id=347 expunge >> >> >> 2012-07-19 12:39:27,995 DEBUG [network.rules.RulesManagerImpl] >> >> >> (Job-Executor-25:job-6598) Applying port forwarding rules for ip >> >> >> address >> >> >> id=58 as a part of vm expunge >> >> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not >> >> >> handled >> >> >> by ExternalFirewall >> >> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not >> >> >> handled >> >> >> by ExternalLoadBalancer >> >> >> 2012-07-19 12:39:28,003 DEBUG >> >> >> [network.router.VirtualNetworkApplianceManagerImpl] >> >> >> (Job-Executor-25:job-6598) Router r-348-VM is in Stopped, so not >> >> >> sending >> >> >> apply firewall rules commands to the backend >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were >> >>handled >> >> >> by >> >> >> DomainRouter >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not >> >> >> handled >> >> >> by Dhcp >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were >> >>handled >> >> >> by >> >> >> Ovs >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were >> >>handled >> >> >> by >> >> >> ExternalDhcp >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were >> >>handled >> >> >> by >> >> >> BareMetal >> >> >> 2012-07-19 12:39:28,003 DEBUG >> >> >> [network.element.ElasticLoadBalancerElement] >> >> >> (Job-Executor-25:job-6598) Not handling network with guest Type >> >> >> Virtual >> >> >> and traffic type Guest >> >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not >> >> >> handled >> >> >> by ElasticLoadBalancer >> >> >> 2012-07-19 12:39:28,034 DEBUG [cloud.vm.UserVmManagerImpl] >> >> >> (Job-Executor-25:job-6598) Port forwarding rules are removed >> >> >> successfully >> >> >> as a part of vm id=347 expunge >> >> >> 2012-07-19 12:39:28,045 DEBUG [cloud.vm.UserVmManagerImpl] >> >> >> (Job-Executor-25:job-6598) Removed vm id=347 from all load >>balancers >> >>as >> >> >> a >> >> >> part of expunge process >> >> >> 2012-07-19 12:39:28,046 DEBUG [cloud.vm.UserVmManagerImpl] >> >> >> (Job-Executor-25:job-6598) Successfully cleaned up vm >> >>VM[User|i-43-347- >> >> >> VM] >> >> >> resources as a part of expunge process >> >> >> 2012-07-19 12:39:28,108 INFO [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) deleteAccount: Deleted 1 network groups >> >>for >> >> >> account 43 >> >> >> 2012-07-19 12:39:28,108 DEBUG [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Deleting networks for account 43 >> >> >> 2012-07-19 12:39:28,111 DEBUG [cloud.network.NetworkManagerImpl] >> >> >> (Job-Executor-25:job-6598) Unable to remove the network id=236 as >>it >> >> >> has >> >> >> active Nics. >> >> >> 2012-07-19 12:39:28,111 WARN [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Unable to destroy network >> >>Ntwk[236|Guest|6] >> >> >> as a >> >> >> part of account id=43 cleanup. >> >> >> 2012-07-19 12:39:28,114 INFO [cloud.user.AccountManagerImpl] >> >> >> (Job-Executor-25:job-6598) Cleanup for account 43 is needed. >> >> >> 2012-07-19 12:39:28,126 DEBUG [cloud.async.AsyncJobManagerImpl] >> >> >> (Job-Executor-25:job-6598) Complete async job-6598, jobStatus: 1, >> >> >> resultCode: 0, result: >> >>com.cloud.api.response.SuccessResponse@49ce1227 >> >> >> 2012-07-19 12:39:28,129 DEBUG [cloud.async.AsyncJobManagerImpl] >> >> >> (Job-Executor-25:job-6598) Done executing >> >> >> com.cloud.api.commands.DeleteAccountCmd for job-6598 >> >> >> 2012-07-19 12:39:36,657 DEBUG [cloud.async.AsyncJobManagerImpl] >> >> >> (http-12449-14:null) Async job-6598 completed >> >> >> >> >> >> >> >> >> PS: all vms, snapshots and volumes was vanished after account >> >>deleted. >> >> >> just >> >> >> vrouter wasn't. >> >> > >> >> >> >> >> >> >> > >> >> >> >
