[
https://issues.apache.org/jira/browse/CLOUDSTACK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan updated CLOUDSTACK-2593:
--------------------------------------------
Description:
Steps to reproduce the problem:
Set up:
Advanced zone with NFS secondary storage and 1 primary storage ps1 for cluster1.
Deploy a VM.
Add 1 more primary storage ps3 for cluster1.
Stop the Vm.
Migrate volume from ps1 to ps2.
This action reports success.
But the volume did not success fully migrate to ps3.
Logs indicates java.lang.NullPointerException being encountered.
management serve logs:
2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===START=== 10.217.252.128 -- GET command=migrateVirtualMac
hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18,
userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd,
cmdOriginator: null, cmdInfo:
{"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===END=== 10.217.252.128 -- GET
command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with
event: StorageMigrationRequestedvm's original host id: 1 new host id: null host
id before state transition: null
2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-18:job-18) Storage migration failed
2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with
event: AgentReportStoppedvm's original host id: 1 new host id: null host id
before state transition: null
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after applying
overprovisioning: 9044
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after
applying overprovisioning: 16190149632
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved:
500, actual total: 9044, total with overprovisioning: 9044; new used:
1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) release mem from host: 1, old used:
1476395008,reserved: 536870912, total: 16190149632; new used:
939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0,
result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl]
(Network-Scavenger-1:null) We found network 204 to be free for the first time.
Adding it to the list: 1337001212
2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-213:null) Ping from 1
2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
===START=== 10.217.252.128 -- GET
command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-20:null) Async job-18 completed
mysql> select * from storage_pool;
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| id | name | uuid | pool_type | port |
data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes |
host_address | user_info | path |
created | removed | update_time | status | storage_provider_name |
scope |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049 |
1 | 1 | 1 | 3018555588608 | 11810778316800 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary |
2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary |
CLUSTER |
| 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049 |
1 | NULL | NULL | 0 | 0 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1 |
2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary |
ZONE |
| 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049 |
1 | 1 | 1 | 3006100733952 | 11810778316800 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2 |
2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary |
CLUSTER |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
3 rows in set (0.00 sec)
was:
management serve logs:
2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===START=== 10.217.252.128 -- GET command=migrateVirtualMac
hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18,
userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd,
cmdOriginator: null, cmdInfo:
{"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===END=== 10.217.252.128 -- GET
command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with
event: StorageMigrationRequestedvm's original host id: 1 new host id: null host
id before state transition: null
2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-18:job-18) Storage migration failed
2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with
event: AgentReportStoppedvm's original host id: 1 new host id: null host id
before state transition: null
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after applying
overprovisioning: 9044
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after
applying overprovisioning: 16190149632
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved:
500, actual total: 9044, total with overprovisioning: 9044; new used:
1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-18:job-18) release mem from host: 1, old used:
1476395008,reserved: 536870912, total: 16190149632; new used:
939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0,
result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-18:job-18) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl]
(Network-Scavenger-1:null) We found network 204 to be free for the first time.
Adding it to the list: 1337001212
2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-213:null) Ping from 1
2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
===START=== 10.217.252.128 -- GET
command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-20:null) Async job-18 completed
mysql> select * from storage_pool;
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| id | name | uuid | pool_type | port |
data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes |
host_address | user_info | path |
created | removed | update_time | status | storage_provider_name |
scope |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049 |
1 | 1 | 1 | 3018555588608 | 11810778316800 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary |
2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary |
CLUSTER |
| 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049 |
1 | NULL | NULL | 0 | 0 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1 |
2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary |
ZONE |
| 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049 |
1 | 1 | 1 | 3006100733952 | 11810778316800 |
10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2 |
2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary |
CLUSTER |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
3 rows in set (0.00 sec)
> Object_Store_Refactor - Not able to migrate volumes between primary storages
> with in the same cluster.
> ------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-2593
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2593
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Affects Versions: 4.2.0
> Environment: Build from object_store
> Reporter: Sangeetha Hariharan
> Priority: Critical
> Fix For: 4.2.0
>
>
> Steps to reproduce the problem:
> Set up:
> Advanced zone with NFS secondary storage and 1 primary storage ps1 for
> cluster1.
> Deploy a VM.
> Add 1 more primary storage ps3 for cluster1.
> Stop the Vm.
> Migrate volume from ps1 to ps2.
> This action reports success.
> But the volume did not success fully migrate to ps3.
> Logs indicates java.lang.NullPointerException being encountered.
> management serve logs:
> 2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
> ===START=== 10.217.252.128 -- GET command=migrateVirtualMac
> hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
> 2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18,
> userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
> null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd,
> cmdOriginator: null, cmdInfo:
> {"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-18:job-18) Executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
> 2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
> ===END=== 10.217.252.128 -- GET
> command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
> 2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with
> event: StorageMigrationRequestedvm's original host id: 1 new host id: null
> host id before state transition: null
> 2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
> 2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-18:job-18) Storage migration failed
> 2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with
> event: AgentReportStoppedvm's original host id: 1 new host id: null host id
> before state transition: null
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after
> applying overprovisioning: 9044
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after
> applying overprovisioning: 16190149632
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved:
> 500, actual total: 9044, total with overprovisioning: 9044; new used:
> 1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-18:job-18) release mem from host: 1, old used:
> 1476395008,reserved: 536870912, total: 16190149632; new used:
> 939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
> 2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0,
> result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
> 2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-18:job-18) Done executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
> 2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl]
> (Network-Scavenger-1:null) We found network 204 to be free for the first
> time. Adding it to the list: 1337001212
> 2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-213:null) Ping from 1
> 2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
> ===START=== 10.217.252.128 -- GET
> command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
> 2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-20:null) Async job-18 completed
> mysql> select * from storage_pool;
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> | id | name | uuid | pool_type | port
> | data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes |
> host_address | user_info | path
> | created | removed | update_time | status |
> storage_provider_name | scope |
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> | 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049
> | 1 | 1 | 1 | 3018555588608 | 11810778316800 |
> 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary
> | 2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary
> | CLUSTER |
> | 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049
> | 1 | NULL | NULL | 0 | 0 |
> 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1
> | 2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary
> | ZONE |
> | 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049
> | 1 | 1 | 1 | 3006100733952 | 11810778316800 |
> 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2
> | 2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary
> | CLUSTER |
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> 3 rows in set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
