What about management log. its 10mb , how do i send to you?
mysql> select * from op_networks;
+-----+-----------------+------------+----+--------------+
| id | mac_address_seq | nics_count | gc | check_for_gc |
+-----+-----------------+------------+----+--------------+
| 200 | 2069 | 0 | 0 | 0 |
| 201 | 1 | 0 | 0 | 0 |
| 202 | 1062 | 0 | 0 | 0 |
| 203 | 1 | 0 | 0 | 0 |
| 204 | 224 | 5 | 1 | 1 |
| 205 | 13 | 0 | 1 | 0 |
| 206 | 8 | 3 | 1 | 1 |
| 207 | 12 | 0 | 1 | 0 |
| 208 | 7 | 2 | 1 | 1 |
| 209 | 10 | 0 | 1 | 0 |
| 210 | 5 | 0 | 1 | 0 |
| 211 | 10 | 1 | 1 | 1 |
| 212 | 14 | 0 | 1 | 0 |
| 213 | 8 | 6 | 1 | 1 |
| 214 | 5 | 0 | 1 | 0 |
| 215 | 4 | 0 | 1 | 0 |
| 216 | 3 | 0 | 1 | 0 |
| 217 | 3 | 0 | 1 | 0 |
| 218 | 4 | 3 | 1 | 1 |
| 219 | 3 | 0 | 1 | 0 |
| 220 | 10 | 5 | 1 | 1 |
| 221 | 6 | 6 | 1 | 1 |
| 222 | 16 | 5 | 1 | 1 |
| 223 | 14 | 1 | 1 | 1 |
| 224 | 30 | 1 | 1 | 1 |
| 225 | 3 | 0 | 1 | 0 |
| 226 | 8 | 11 | 1 | 1 |
| 227 | 3 | 0 | 1 | 0 |
| 228 | 3 | 2 | 1 | 1 |
| 229 | 4 | 3 | 1 | 1 |
| 230 | 3 | 0 | 1 | 0 |
| 231 | 6 | 7 | 1 | 1 |
| 232 | 26 | 24 | 1 | 1 |
| 233 | 4 | 2 | 1 | 1 |
| 234 | 7 | 9 | 1 | 1 |
| 235 | 3 | 0 | 1 | 0 |
| 236 | 3 | 2 | 1 | 1 |
| 237 | 4 | 0 | 1 | 0 |
| 238 | 9 | 7 | 1 | 1 |
| 240 | 4 | 3 | 1 | 1 |
| 241 | 5 | 3 | 1 | 1 |
| 242 | 3 | 3 | 1 | 1 |
| 243 | 30 | 4 | 1 | 1 |
| 244 | 3 | 0 | 1 | 0 |
| 245 | 4 | 3 | 1 | 1 |
| 246 | 4 | 3 | 1 | 1 |
| 247 | 4 | 3 | 1 | 1 |
| 248 | 3 | 2 | 1 | 1 |
| 249 | 3 | 2 | 1 | 1 |
| 250 | 3 | 3 | 1 | 1 |
| 251 | 7 | 0 | 1 | 0 |
| 252 | 4 | 6 | 1 | 1 |
| 253 | 3 | 2 | 1 | 1 |
| 254 | 13 | 6 | 1 | 1 |
| 255 | 7 | 3 | 1 | 1 |
| 256 | 3 | 1 | 1 | 1 |
| 257 | 6 | 4 | 1 | 1 |
| 258 | 1 | 6 | 1 | 1 |
| 259 | 4 | 3 | 1 | 1 |
| 260 | 5 | 2 | 1 | 1 |
| 261 | 3 | 3 | 1 | 1 |
| 262 | 5 | 0 | 1 | 0 |
| 263 | 5 | 1 | 1 | 1 |
| 264 | 3 | 0 | 1 | 0 |
| 265 | 3 | 1 | 1 | 1 |
| 266 | 4 | 1 | 1 | 1 |
| 267 | 3 | 1 | 1 | 1 |
| 268 | 6 | 2 | 1 | 1 |
| 269 | 3 | 1 | 1 | 1 |
| 270 | 4 | 0 | 1 | 0 |
| 271 | 3 | 1 | 1 | 1 |
| 272 | 3 | 1 | 1 | 1 |
| 273 | 3 | 1 | 1 | 1 |
| 274 | 10 | 2 | 1 | 1 |
| 275 | 3 | 1 | 1 | 1 |
+-----+-----------------+------------+----+--------------+
mysql> select * from op_networks where id=236;
+-----+-----------------+------------+----+--------------+
| id | mac_address_seq | nics_count | gc | check_for_gc |
+-----+-----------------+------------+----+--------------+
| 236 | 3 | 2 | 1 | 1 |
+-----+-----------------+------------+----+--------------+
mysql> select * from vm_instance where id in (select instance_id from nics
where removed is null and network_id=236)\G
*************************** 1. row ***************************
id: 348
name: r-348-VM
instance_name: r-348-VM
state: Stopped
vm_template_id: 8
guest_os_id: 15
private_mac_address: 02:00:13:63:03:ed
private_ip_address: 10.16.140.70
private_netmask: NULL
pod_id: 1
data_center_id: 1
host_id: NULL
last_host_id: 2
proxy_id: 305
proxy_assign_time: 2012-03-17 04:43:35
vnc_password: d3cc123795551907
ha_enabled: 1
limit_cpu_use: 1
update_count: 202
update_time: 2012-07-17 15:54:54
created: 2012-02-16 16:37:19
removed: NULL
type: DomainRouter
vm_type: DomainRouter
account_id: 43
domain_id: 1
service_offering_id: 7
reservation_id: 147062dc-3596-4799-a84e-667784810067
hypervisor_type: VMware
1 row in set (0.00 sec)
2012/7/24 Alena Prokharchyk <[email protected]>
> Error message:
>
> "Unable to remove the network id=236 as it has active Nics".
>
>
> Indicates that the User vm(s) (not router) fail to expunge during the
> account deletion, therefore the corresponding network failed to be
> removed. The account will be marked with Removed=someDate +
> cleanup_needed=1, and the recurring thread (runs every
> account.clenaup.interval defined in global config) will try to cleanup the
> account again and again, until it succeeds.
>
>
>
> To find out if this is the root cause, following is needed:
>
> * management server log
> * content of the op_networks table, the active nic info is stored there.
> Nics should be counted for non-expunged user vms only (select * from
> op_networks where id=236)
> * information about the user vms belonging to network id=236 (select *
> from vm_instance where id in (select instance_id from nics where removed
> is null and network_id=236))
>
> -Alena.
>
> On 7/24/12 11:15 AM, "Edison Su" <[email protected]> wrote:
>
> >Sounds like a bug, the VR is not got destroyed during deleteAccount.
> >Could you help to fire a bug, and attach the whole mgt server log?
> >
> >> -----Original Message-----
> >> From: Diego Spinola Castro [mailto:[email protected]]
> >> Sent: Tuesday, July 24, 2012 10:03 AM
> >> To: [email protected]
> >> Subject: Orphaned VRouters
> >>
> >> Hi, i found some orphan vrouters and don't know why they still running
> >> even
> >> there's no account anymore (i deleted account).
> >>
> >> Bellow management.log
> >>
> >>
> >> 2012-07-19 12:39:06,437 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> (http-12449-14:null) submit async job-6598, details: AsyncJobVO
> >> {id:6598,
> >> userId: 32, accountId: 2, sessionKey: null, instanceType: None,
> >> instanceId:
> >> null, cmd: com.cloud.api.commands.DeleteAccountCmd, cmdOriginator: null,
> >> cmdInfo:
> >> {"id":"43","response":"json","sessionkey":"7uBHMfOyDdSLdtSjPB9K0FcVloQ\
> >> u003d","ctxUserId":"32","_":"1342707903173","ctxAccountId":"2","ctxStar
> >> tEventId":"35670"},
> >> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> >> processStatus: 0, resultCode: 0, result: null, initMsid: 144350996536,
> >> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> >> 2012-07-19 12:39:06,438 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> (Job-Executor-25:job-6598) Executing
> >> com.cloud.api.commands.DeleteAccountCmd for job-6598
> >> 2012-07-19 12:39:06,443 DEBUG [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Access to Acct[43-300012_CUBOCC] granted to
> >> Acct[2-admin] by DomainChecker
> >> 2012-07-19 12:39:06,460 DEBUG [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Remove account 43
> >> 2012-07-19 12:39:06,514 DEBUG [agent.transport.Request]
> >> (Job-Executor-25:job-6598) Seq 20-573703176: Sending { Cmd , MgmtId:
> >> 144350996536, via: 20, Ver: v1, Flags: 100011,
> >> [{"DeleteSnapshotsDirCommand":{"primaryStoragePoolNameLabel":"016b139d-
> >> a43e-32de-a330-e57a91388695","secondaryStoragePoolURL":"nfs://
> >> 10.16.140.11/export","dcId":1,"accountId":43,"volumeId":467,"volumePath
> >> ":"2c880c4443a44672971b964d0cfe725c","wait":0}}]
> >> }
> >> 2012-07-19 12:39:09,334 DEBUG [agent.transport.Request]
> >> (Job-Executor-25:job-6598) Seq 20-573703176: Received: { Ans: , MgmtId:
> >> 144350996536, via: 20, Ver: v1, Flags: 10, { Answer } }
> >> 2012-07-19 12:39:09,343 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details null
> >> 2012-07-19 12:39:09,343 DEBUG [storage.snapshot.SnapshotManagerImpl]
> >> (Job-Executor-25:job-6598) Deleted snapshotsDir for volume: 467 under
> >> account: 43
> >> 2012-07-19 12:39:09,345 DEBUG [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Successfully deleted snapshots directories
> >> for
> >> all volumes under account 43 across all zones
> >> 2012-07-19 12:39:09,348 DEBUG [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Destroying # of vms (accountId=43): 1
> >> 2012-07-19 12:39:09,348 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-25:job-6598) Stopped called on VM[User|i-43-347-VM] but
> >> the
> >> state is Destroyed
> >> 2012-07-19 12:39:09,355 DEBUG [cloud.capacity.CapacityManagerImpl]
> >> (Job-Executor-25:job-6598) VM state transitted from :Destroyed to
> >> Expunging
> >> with event: ExpungeOperationvm's original host id: 4 new host id: null
> >> host
> >> id before state transition: null
> >> 2012-07-19 12:39:09,355 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-25:job-6598) Destroying vm VM[User|i-43-347-VM]
> >> 2012-07-19 12:39:09,355 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Cleaning network for vm: 347
> >> 2012-07-19 12:39:09,369 DEBUG [cloud.storage.StorageManagerImpl]
> >> (Job-Executor-25:job-6598) Cleaning storage for vm: 347
> >> 2012-07-19 12:39:09,401 DEBUG [cloud.storage.StorageManagerImpl]
> >> (Job-Executor-25:job-6598) Detaching Vol[467|vm=347|DATADISK]
> >> 2012-07-19 12:39:09,402 DEBUG [cloud.storage.StorageManagerImpl]
> >> (Job-Executor-25:job-6598) Detaching Vol[488|vm=347|DATADISK]
> >> 2012-07-19 12:39:09,404 DEBUG [cloud.storage.StorageManagerImpl]
> >> (Job-Executor-25:job-6598) Expunging Vol[392|vm=347|ROOT]
> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
> >> (Job-Executor-25:job-6598) Seq 3-115876736: Sending { Cmd , MgmtId:
> >> 144350996536, via: 3, Ver: v1, Flags: 100111,
> >> [{"storage.DestroyCommand":{"vmName":"i-43-347-
> >> VM","volume":{"id":392,"name":"ROOT-
> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
> >> 347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
> >> PoolUuid":"30b0d978-3291-3ce3-a06f-
> >> eea3ac2f4f51","deviceId":0},"wait":0}}]
> >> }
> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
> >> (Job-Executor-25:job-6598) Seq 3-115876736: Executing: { Cmd , MgmtId:
> >> 144350996536, via: 3, Ver: v1, Flags: 100111,
> >> [{"storage.DestroyCommand":{"vmName":"i-43-347-
> >> VM","volume":{"id":392,"name":"ROOT-
> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
> >> 347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
> >> PoolUuid":"30b0d978-3291-3ce3-a06f-
> >> eea3ac2f4f51","deviceId":0},"wait":0}}]
> >> }
> >> 2012-07-19 12:39:27,879 DEBUG [agent.transport.Request]
> >> (Job-Executor-25:job-6598) Seq 3-115876736: Received: { Ans: , MgmtId:
> >> 144350996536, via: 3, Ver: v1, Flags: 110, { Answer } }
> >> 2012-07-19 12:39:27,879 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details Success
> >> 2012-07-19 12:39:27,890 DEBUG [cloud.storage.StorageManagerImpl]
> >> (Job-Executor-25:job-6598) Volume successfully expunged from 206
> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-25:job-6598) Expunged VM[User|i-43-347-VM]
> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (Job-Executor-25:job-6598) Starting cleaning up vm VM[User|i-43-347-VM]
> >> resources...
> >> 2012-07-19 12:39:27,947 DEBUG [network.firewall.FirewallManagerImpl]
> >> (Job-Executor-25:job-6598) No firewall rules are found for vm id=347
> >> 2012-07-19 12:39:27,947 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (Job-Executor-25:job-6598) Firewall rules are removed successfully as a
> >> part of vm id=347 expunge
> >> 2012-07-19 12:39:27,995 DEBUG [network.rules.RulesManagerImpl]
> >> (Job-Executor-25:job-6598) Applying port forwarding rules for ip
> >> address
> >> id=58 as a part of vm expunge
> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not
> >> handled
> >> by ExternalFirewall
> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not
> >> handled
> >> by ExternalLoadBalancer
> >> 2012-07-19 12:39:28,003 DEBUG
> >> [network.router.VirtualNetworkApplianceManagerImpl]
> >> (Job-Executor-25:job-6598) Router r-348-VM is in Stopped, so not
> >> sending
> >> apply firewall rules commands to the backend
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were handled
> >> by
> >> DomainRouter
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not
> >> handled
> >> by Dhcp
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were handled
> >> by
> >> Ovs
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were handled
> >> by
> >> ExternalDhcp
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were handled
> >> by
> >> BareMetal
> >> 2012-07-19 12:39:28,003 DEBUG
> >> [network.element.ElasticLoadBalancerElement]
> >> (Job-Executor-25:job-6598) Not handling network with guest Type
> >> Virtual
> >> and traffic type Guest
> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Network Rules for network 236 were not
> >> handled
> >> by ElasticLoadBalancer
> >> 2012-07-19 12:39:28,034 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (Job-Executor-25:job-6598) Port forwarding rules are removed
> >> successfully
> >> as a part of vm id=347 expunge
> >> 2012-07-19 12:39:28,045 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (Job-Executor-25:job-6598) Removed vm id=347 from all load balancers as
> >> a
> >> part of expunge process
> >> 2012-07-19 12:39:28,046 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (Job-Executor-25:job-6598) Successfully cleaned up vm VM[User|i-43-347-
> >> VM]
> >> resources as a part of expunge process
> >> 2012-07-19 12:39:28,108 INFO [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) deleteAccount: Deleted 1 network groups for
> >> account 43
> >> 2012-07-19 12:39:28,108 DEBUG [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Deleting networks for account 43
> >> 2012-07-19 12:39:28,111 DEBUG [cloud.network.NetworkManagerImpl]
> >> (Job-Executor-25:job-6598) Unable to remove the network id=236 as it
> >> has
> >> active Nics.
> >> 2012-07-19 12:39:28,111 WARN [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Unable to destroy network Ntwk[236|Guest|6]
> >> as a
> >> part of account id=43 cleanup.
> >> 2012-07-19 12:39:28,114 INFO [cloud.user.AccountManagerImpl]
> >> (Job-Executor-25:job-6598) Cleanup for account 43 is needed.
> >> 2012-07-19 12:39:28,126 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> (Job-Executor-25:job-6598) Complete async job-6598, jobStatus: 1,
> >> resultCode: 0, result: com.cloud.api.response.SuccessResponse@49ce1227
> >> 2012-07-19 12:39:28,129 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> (Job-Executor-25:job-6598) Done executing
> >> com.cloud.api.commands.DeleteAccountCmd for job-6598
> >> 2012-07-19 12:39:36,657 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> (http-12449-14:null) Async job-6598 completed
> >>
> >>
> >> PS: all vms, snapshots and volumes was vanished after account deleted.
> >> just
> >> vrouter wasn't.
> >
>
>
>
