OK: https://dl.dropbox.com/u/19653845/management-server.log.2012-07-19.gz
2012/7/24 Alena Prokharchyk <[email protected]> > Please Zip it, place it on some publicly available web host and send the > link. > > Thank you, > Alena. > > On 7/24/12 11:59 AM, "Diego Spinola Castro" <[email protected]> > wrote: > > >What about management log. its 10mb , how do i send to you? > > > >mysql> select * from op_networks; > >+-----+-----------------+------------+----+--------------+ > >| id | mac_address_seq | nics_count | gc | check_for_gc | > >+-----+-----------------+------------+----+--------------+ > >| 200 | 2069 | 0 | 0 | 0 | > >| 201 | 1 | 0 | 0 | 0 | > >| 202 | 1062 | 0 | 0 | 0 | > >| 203 | 1 | 0 | 0 | 0 | > >| 204 | 224 | 5 | 1 | 1 | > >| 205 | 13 | 0 | 1 | 0 | > >| 206 | 8 | 3 | 1 | 1 | > >| 207 | 12 | 0 | 1 | 0 | > >| 208 | 7 | 2 | 1 | 1 | > >| 209 | 10 | 0 | 1 | 0 | > >| 210 | 5 | 0 | 1 | 0 | > >| 211 | 10 | 1 | 1 | 1 | > >| 212 | 14 | 0 | 1 | 0 | > >| 213 | 8 | 6 | 1 | 1 | > >| 214 | 5 | 0 | 1 | 0 | > >| 215 | 4 | 0 | 1 | 0 | > >| 216 | 3 | 0 | 1 | 0 | > >| 217 | 3 | 0 | 1 | 0 | > >| 218 | 4 | 3 | 1 | 1 | > >| 219 | 3 | 0 | 1 | 0 | > >| 220 | 10 | 5 | 1 | 1 | > >| 221 | 6 | 6 | 1 | 1 | > >| 222 | 16 | 5 | 1 | 1 | > >| 223 | 14 | 1 | 1 | 1 | > >| 224 | 30 | 1 | 1 | 1 | > >| 225 | 3 | 0 | 1 | 0 | > >| 226 | 8 | 11 | 1 | 1 | > >| 227 | 3 | 0 | 1 | 0 | > >| 228 | 3 | 2 | 1 | 1 | > >| 229 | 4 | 3 | 1 | 1 | > >| 230 | 3 | 0 | 1 | 0 | > >| 231 | 6 | 7 | 1 | 1 | > >| 232 | 26 | 24 | 1 | 1 | > >| 233 | 4 | 2 | 1 | 1 | > >| 234 | 7 | 9 | 1 | 1 | > >| 235 | 3 | 0 | 1 | 0 | > >| 236 | 3 | 2 | 1 | 1 | > >| 237 | 4 | 0 | 1 | 0 | > >| 238 | 9 | 7 | 1 | 1 | > >| 240 | 4 | 3 | 1 | 1 | > >| 241 | 5 | 3 | 1 | 1 | > >| 242 | 3 | 3 | 1 | 1 | > >| 243 | 30 | 4 | 1 | 1 | > >| 244 | 3 | 0 | 1 | 0 | > >| 245 | 4 | 3 | 1 | 1 | > >| 246 | 4 | 3 | 1 | 1 | > >| 247 | 4 | 3 | 1 | 1 | > >| 248 | 3 | 2 | 1 | 1 | > >| 249 | 3 | 2 | 1 | 1 | > >| 250 | 3 | 3 | 1 | 1 | > >| 251 | 7 | 0 | 1 | 0 | > >| 252 | 4 | 6 | 1 | 1 | > >| 253 | 3 | 2 | 1 | 1 | > >| 254 | 13 | 6 | 1 | 1 | > >| 255 | 7 | 3 | 1 | 1 | > >| 256 | 3 | 1 | 1 | 1 | > >| 257 | 6 | 4 | 1 | 1 | > >| 258 | 1 | 6 | 1 | 1 | > >| 259 | 4 | 3 | 1 | 1 | > >| 260 | 5 | 2 | 1 | 1 | > >| 261 | 3 | 3 | 1 | 1 | > >| 262 | 5 | 0 | 1 | 0 | > >| 263 | 5 | 1 | 1 | 1 | > >| 264 | 3 | 0 | 1 | 0 | > >| 265 | 3 | 1 | 1 | 1 | > >| 266 | 4 | 1 | 1 | 1 | > >| 267 | 3 | 1 | 1 | 1 | > >| 268 | 6 | 2 | 1 | 1 | > >| 269 | 3 | 1 | 1 | 1 | > >| 270 | 4 | 0 | 1 | 0 | > >| 271 | 3 | 1 | 1 | 1 | > >| 272 | 3 | 1 | 1 | 1 | > >| 273 | 3 | 1 | 1 | 1 | > >| 274 | 10 | 2 | 1 | 1 | > >| 275 | 3 | 1 | 1 | 1 | > >+-----+-----------------+------------+----+--------------+ > > > >mysql> select * from op_networks where id=236; > >+-----+-----------------+------------+----+--------------+ > >| id | mac_address_seq | nics_count | gc | check_for_gc | > >+-----+-----------------+------------+----+--------------+ > >| 236 | 3 | 2 | 1 | 1 | > >+-----+-----------------+------------+----+--------------+ > > > >mysql> select * from vm_instance where id in (select instance_id from nics > >where removed is null and network_id=236)\G > >*************************** 1. row *************************** > > id: 348 > > name: r-348-VM > > instance_name: r-348-VM > > state: Stopped > > vm_template_id: 8 > > guest_os_id: 15 > >private_mac_address: 02:00:13:63:03:ed > > private_ip_address: 10.16.140.70 > > private_netmask: NULL > > pod_id: 1 > > data_center_id: 1 > > host_id: NULL > > last_host_id: 2 > > proxy_id: 305 > > proxy_assign_time: 2012-03-17 04:43:35 > > vnc_password: d3cc123795551907 > > ha_enabled: 1 > > limit_cpu_use: 1 > > update_count: 202 > > update_time: 2012-07-17 15:54:54 > > created: 2012-02-16 16:37:19 > > removed: NULL > > type: DomainRouter > > vm_type: DomainRouter > > account_id: 43 > > domain_id: 1 > >service_offering_id: 7 > > reservation_id: 147062dc-3596-4799-a84e-667784810067 > > hypervisor_type: VMware > >1 row in set (0.00 sec) > > > > > >2012/7/24 Alena Prokharchyk <[email protected]> > > > >> Error message: > >> > >> "Unable to remove the network id=236 as it has active Nics". > >> > >> > >> Indicates that the User vm(s) (not router) fail to expunge during the > >> account deletion, therefore the corresponding network failed to be > >> removed. The account will be marked with Removed=someDate + > >> cleanup_needed=1, and the recurring thread (runs every > >> account.clenaup.interval defined in global config) will try to cleanup > >>the > >> account again and again, until it succeeds. > >> > >> > >> > >> To find out if this is the root cause, following is needed: > >> > >> * management server log > >> * content of the op_networks table, the active nic info is stored there. > >> Nics should be counted for non-expunged user vms only (select * from > >> op_networks where id=236) > >> * information about the user vms belonging to network id=236 (select * > >> from vm_instance where id in (select instance_id from nics where removed > >> is null and network_id=236)) > >> > >> -Alena. > >> > >> On 7/24/12 11:15 AM, "Edison Su" <[email protected]> wrote: > >> > >> >Sounds like a bug, the VR is not got destroyed during deleteAccount. > >> >Could you help to fire a bug, and attach the whole mgt server log? > >> > > >> >> -----Original Message----- > >> >> From: Diego Spinola Castro [mailto:[email protected]] > >> >> Sent: Tuesday, July 24, 2012 10:03 AM > >> >> To: [email protected] > >> >> Subject: Orphaned VRouters > >> >> > >> >> Hi, i found some orphan vrouters and don't know why they still > >>running > >> >> even > >> >> there's no account anymore (i deleted account). > >> >> > >> >> Bellow management.log > >> >> > >> >> > >> >> 2012-07-19 12:39:06,437 DEBUG [cloud.async.AsyncJobManagerImpl] > >> >> (http-12449-14:null) submit async job-6598, details: AsyncJobVO > >> >> {id:6598, > >> >> userId: 32, accountId: 2, sessionKey: null, instanceType: None, > >> >> instanceId: > >> >> null, cmd: com.cloud.api.commands.DeleteAccountCmd, cmdOriginator: > >>null, > >> >> cmdInfo: > >> >> > >>{"id":"43","response":"json","sessionkey":"7uBHMfOyDdSLdtSjPB9K0FcVloQ\ > >> >> > >>u003d","ctxUserId":"32","_":"1342707903173","ctxAccountId":"2","ctxStar > >> >> tEventId":"35670"}, > >> >> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > >> >> processStatus: 0, resultCode: 0, result: null, initMsid: > >>144350996536, > >> >> completeMsid: null, lastUpdated: null, lastPolled: null, created: > >>null} > >> >> 2012-07-19 12:39:06,438 DEBUG [cloud.async.AsyncJobManagerImpl] > >> >> (Job-Executor-25:job-6598) Executing > >> >> com.cloud.api.commands.DeleteAccountCmd for job-6598 > >> >> 2012-07-19 12:39:06,443 DEBUG [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Access to Acct[43-300012_CUBOCC] granted > >>to > >> >> Acct[2-admin] by DomainChecker > >> >> 2012-07-19 12:39:06,460 DEBUG [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Remove account 43 > >> >> 2012-07-19 12:39:06,514 DEBUG [agent.transport.Request] > >> >> (Job-Executor-25:job-6598) Seq 20-573703176: Sending { Cmd , MgmtId: > >> >> 144350996536, via: 20, Ver: v1, Flags: 100011, > >> >> > >>[{"DeleteSnapshotsDirCommand":{"primaryStoragePoolNameLabel":"016b139d- > >> >> a43e-32de-a330-e57a91388695","secondaryStoragePoolURL":"nfs:// > >> >> > >>10.16.140.11/export","dcId":1,"accountId":43,"volumeId":467,"volumePath > >> >> ":"2c880c4443a44672971b964d0cfe725c","wait":0}}] > >> >> } > >> >> 2012-07-19 12:39:09,334 DEBUG [agent.transport.Request] > >> >> (Job-Executor-25:job-6598) Seq 20-573703176: Received: { Ans: , > >>MgmtId: > >> >> 144350996536, via: 20, Ver: v1, Flags: 10, { Answer } } > >> >> 2012-07-19 12:39:09,343 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details null > >> >> 2012-07-19 12:39:09,343 DEBUG [storage.snapshot.SnapshotManagerImpl] > >> >> (Job-Executor-25:job-6598) Deleted snapshotsDir for volume: 467 under > >> >> account: 43 > >> >> 2012-07-19 12:39:09,345 DEBUG [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Successfully deleted snapshots directories > >> >> for > >> >> all volumes under account 43 across all zones > >> >> 2012-07-19 12:39:09,348 DEBUG [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Destroying # of vms (accountId=43): 1 > >> >> 2012-07-19 12:39:09,348 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> >> (Job-Executor-25:job-6598) Stopped called on VM[User|i-43-347-VM] but > >> >> the > >> >> state is Destroyed > >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.capacity.CapacityManagerImpl] > >> >> (Job-Executor-25:job-6598) VM state transitted from :Destroyed to > >> >> Expunging > >> >> with event: ExpungeOperationvm's original host id: 4 new host id: > >>null > >> >> host > >> >> id before state transition: null > >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> >> (Job-Executor-25:job-6598) Destroying vm VM[User|i-43-347-VM] > >> >> 2012-07-19 12:39:09,355 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Cleaning network for vm: 347 > >> >> 2012-07-19 12:39:09,369 DEBUG [cloud.storage.StorageManagerImpl] > >> >> (Job-Executor-25:job-6598) Cleaning storage for vm: 347 > >> >> 2012-07-19 12:39:09,401 DEBUG [cloud.storage.StorageManagerImpl] > >> >> (Job-Executor-25:job-6598) Detaching Vol[467|vm=347|DATADISK] > >> >> 2012-07-19 12:39:09,402 DEBUG [cloud.storage.StorageManagerImpl] > >> >> (Job-Executor-25:job-6598) Detaching Vol[488|vm=347|DATADISK] > >> >> 2012-07-19 12:39:09,404 DEBUG [cloud.storage.StorageManagerImpl] > >> >> (Job-Executor-25:job-6598) Expunging Vol[392|vm=347|ROOT] > >> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request] > >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Sending { Cmd , MgmtId: > >> >> 144350996536, via: 3, Ver: v1, Flags: 100111, > >> >> [{"storage.DestroyCommand":{"vmName":"i-43-347- > >> >> VM","volume":{"id":392,"name":"ROOT- > >> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT- > >> >> > >>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage > >> >> PoolUuid":"30b0d978-3291-3ce3-a06f- > >> >> eea3ac2f4f51","deviceId":0},"wait":0}}] > >> >> } > >> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request] > >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Executing: { Cmd , > >>MgmtId: > >> >> 144350996536, via: 3, Ver: v1, Flags: 100111, > >> >> [{"storage.DestroyCommand":{"vmName":"i-43-347- > >> >> VM","volume":{"id":392,"name":"ROOT- > >> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT- > >> >> > >>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage > >> >> PoolUuid":"30b0d978-3291-3ce3-a06f- > >> >> eea3ac2f4f51","deviceId":0},"wait":0}}] > >> >> } > >> >> 2012-07-19 12:39:27,879 DEBUG [agent.transport.Request] > >> >> (Job-Executor-25:job-6598) Seq 3-115876736: Received: { Ans: , > >>MgmtId: > >> >> 144350996536, via: 3, Ver: v1, Flags: 110, { Answer } } > >> >> 2012-07-19 12:39:27,879 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details Success > >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.storage.StorageManagerImpl] > >> >> (Job-Executor-25:job-6598) Volume successfully expunged from 206 > >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> >> (Job-Executor-25:job-6598) Expunged VM[User|i-43-347-VM] > >> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.UserVmManagerImpl] > >> >> (Job-Executor-25:job-6598) Starting cleaning up vm > >>VM[User|i-43-347-VM] > >> >> resources... > >> >> 2012-07-19 12:39:27,947 DEBUG [network.firewall.FirewallManagerImpl] > >> >> (Job-Executor-25:job-6598) No firewall rules are found for vm id=347 > >> >> 2012-07-19 12:39:27,947 DEBUG [cloud.vm.UserVmManagerImpl] > >> >> (Job-Executor-25:job-6598) Firewall rules are removed successfully > >>as a > >> >> part of vm id=347 expunge > >> >> 2012-07-19 12:39:27,995 DEBUG [network.rules.RulesManagerImpl] > >> >> (Job-Executor-25:job-6598) Applying port forwarding rules for ip > >> >> address > >> >> id=58 as a part of vm expunge > >> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not > >> >> handled > >> >> by ExternalFirewall > >> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not > >> >> handled > >> >> by ExternalLoadBalancer > >> >> 2012-07-19 12:39:28,003 DEBUG > >> >> [network.router.VirtualNetworkApplianceManagerImpl] > >> >> (Job-Executor-25:job-6598) Router r-348-VM is in Stopped, so not > >> >> sending > >> >> apply firewall rules commands to the backend > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were > >>handled > >> >> by > >> >> DomainRouter > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not > >> >> handled > >> >> by Dhcp > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were > >>handled > >> >> by > >> >> Ovs > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were > >>handled > >> >> by > >> >> ExternalDhcp > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were > >>handled > >> >> by > >> >> BareMetal > >> >> 2012-07-19 12:39:28,003 DEBUG > >> >> [network.element.ElasticLoadBalancerElement] > >> >> (Job-Executor-25:job-6598) Not handling network with guest Type > >> >> Virtual > >> >> and traffic type Guest > >> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not > >> >> handled > >> >> by ElasticLoadBalancer > >> >> 2012-07-19 12:39:28,034 DEBUG [cloud.vm.UserVmManagerImpl] > >> >> (Job-Executor-25:job-6598) Port forwarding rules are removed > >> >> successfully > >> >> as a part of vm id=347 expunge > >> >> 2012-07-19 12:39:28,045 DEBUG [cloud.vm.UserVmManagerImpl] > >> >> (Job-Executor-25:job-6598) Removed vm id=347 from all load balancers > >>as > >> >> a > >> >> part of expunge process > >> >> 2012-07-19 12:39:28,046 DEBUG [cloud.vm.UserVmManagerImpl] > >> >> (Job-Executor-25:job-6598) Successfully cleaned up vm > >>VM[User|i-43-347- > >> >> VM] > >> >> resources as a part of expunge process > >> >> 2012-07-19 12:39:28,108 INFO [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) deleteAccount: Deleted 1 network groups > >>for > >> >> account 43 > >> >> 2012-07-19 12:39:28,108 DEBUG [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Deleting networks for account 43 > >> >> 2012-07-19 12:39:28,111 DEBUG [cloud.network.NetworkManagerImpl] > >> >> (Job-Executor-25:job-6598) Unable to remove the network id=236 as it > >> >> has > >> >> active Nics. > >> >> 2012-07-19 12:39:28,111 WARN [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Unable to destroy network > >>Ntwk[236|Guest|6] > >> >> as a > >> >> part of account id=43 cleanup. > >> >> 2012-07-19 12:39:28,114 INFO [cloud.user.AccountManagerImpl] > >> >> (Job-Executor-25:job-6598) Cleanup for account 43 is needed. > >> >> 2012-07-19 12:39:28,126 DEBUG [cloud.async.AsyncJobManagerImpl] > >> >> (Job-Executor-25:job-6598) Complete async job-6598, jobStatus: 1, > >> >> resultCode: 0, result: > >>com.cloud.api.response.SuccessResponse@49ce1227 > >> >> 2012-07-19 12:39:28,129 DEBUG [cloud.async.AsyncJobManagerImpl] > >> >> (Job-Executor-25:job-6598) Done executing > >> >> com.cloud.api.commands.DeleteAccountCmd for job-6598 > >> >> 2012-07-19 12:39:36,657 DEBUG [cloud.async.AsyncJobManagerImpl] > >> >> (http-12449-14:null) Async job-6598 completed > >> >> > >> >> > >> >> PS: all vms, snapshots and volumes was vanished after account > >>deleted. > >> >> just > >> >> vrouter wasn't. > >> > > >> > >> > >> > > > > >
