Sanjeev N created CLOUDSTACK-5639:
-------------------------------------
Summary: [Hyper-v] cold storage does not work in hyperv
Key: CLOUDSTACK-5639
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5639
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Hypervisor Controller, Management Server
Affects Versions: 4.3.0
Environment: Latest build from 4.3 branch with
commit:4b2b6544255ee7fe12b4b4732cacd08e4ee6a1b7
Storage: SMB for both primary and secondary
Hypervisor: Hyper-v
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.3.0
[Hyper-v] cold storage does not work in hyperv
Steps to Reproduce:
================
1.Bring up CS in advanced zone with 2 hosts in a hyper-v cluster using CIFS for
both primary and secondary storage
2.Deploy one or two guest vms using default cent os template.
3.Add one more primary storage (say primary2) to the cluster
4.Stop one of the vms deployed at step2
5.Migrate the storage from source primary to new primary storage primary2
Expected Result:
=============
Storage migration should succeed when the vm is in stopped state
Actual Result:
===========
Storage migration failed with the error "Failed to migration:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:7] is
unreachable"
Observations:
============
Log snippet from the MS log:
2013-12-24 19:11:55,475 DEBUG [c.c.a.ApiServlet]
(catalina-exec-11:ctx-717ba369) ===START=== 10.146.0.134 -- GET
command=migrateVirtualMachine&storageid=fe97ca45-5cba-329a-ae79-fa02649421c1&virtualmachineid=3a1b4f23-7a43-4806-8fd5-43a9c1130872&response=json&sessionkey=uHhzduF1EppEHm%2Bibsj2%2Bqf%2Fzto%3D&_=1387892514178
2013-12-24 19:11:55,661 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-11:ctx-717ba369 ctx-d805adb4) submit async job-33, details:
AsyncJobVO {id:33, userId: 2, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
{"response":"json","sessionkey":"uHhzduF1EppEHm+ibsj2+qf/zto\u003d","virtualmachineid":"3a1b4f23-7a43-4806-8fd5-43a9c1130872","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"fe97ca45-5cba-329a-ae79-fa02649421c1","httpmethod":"GET","_":"1387892514178","ctxAccountId":"2","ctxStartEventId":"99"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-24 19:11:55,663 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-29:ctx-893d8661) Add job-33 into job monitoring
2013-12-24 19:11:55,663 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-29:ctx-893d8661) Executing AsyncJobVO {id:33, userId: 2,
accountId: 2, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
{"response":"json","sessionkey":"uHhzduF1EppEHm+ibsj2+qf/zto\u003d","virtualmachineid":"3a1b4f23-7a43-4806-8fd5-43a9c1130872","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"fe97ca45-5cba-329a-ae79-fa02649421c1","httpmethod":"GET","_":"1387892514178","ctxAccountId":"2","ctxStartEventId":"99"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2013-12-24 19:11:55,663 DEBUG [c.c.a.ApiServlet] (catalina-exec-11:ctx-717ba369
ctx-d805adb4) ===END=== 10.146.0.134 -- GET
command=migrateVirtualMachine&storageid=fe97ca45-5cba-329a-ae79-fa02649421c1&virtualmachineid=3a1b4f23-7a43-4806-8fd5-43a9c1130872&response=json&sessionkey=uHhzduF1EppEHm%2Bibsj2%2Bqf%2Fzto%3D&_=1387892514178
2013-12-24 19:11:55,794 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) VM state transitted from :Stopped
to Migrating with event: StorageMigrationRequestedvm's original host id: 4 new
host id: null host id before state transition: null
2013-12-24 19:11:56,026 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) copyAsync inspecting src type
VOLUME copyAsync inspecting dest type VOLUME
2013-12-24 19:11:56,199 DEBUG [c.c.a.t.Request] (Job-Executor-29:ctx-893d8661
ctx-d805adb4) Seq 1-1652228440: Sending { Cmd , MgmtId: 132129494109518, via:
1(10.147.40.14), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"34a817b2-e24f-4fb3-82dc-918505fcf8c8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9e132ff7-f9ef-3f17-bbc2-c44c323d96e7","id":1,"poolType":"NetworkFilesystem","host":"Host13","path":"/hyperv/primary?user=sanjeev&password=Password_123&domain=BLR","port":445,"url":"NetworkFilesystem://Host13//hyperv/primary?user=sanjeev&password=Password_123&domain=BLR/?ROLE=Primary&STOREUUID=9e132ff7-f9ef-3f17-bbc2-c44c323d96e7"}},"name":"ROOT-3","size":5368709120,"volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3,"deviceId":0,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"34a817b2-e24f-4fb3-82dc-918505fcf8c8","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR","_role":"Image"}},"name":"ROOT-3","size":5368709120,"path":"volumes/2/3","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3,"deviceId":0,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"wait":10800}}]
}
2013-12-24 19:11:56,199 DEBUG [c.c.a.t.Request] (Job-Executor-29:ctx-893d8661
ctx-d805adb4) Seq 1-1652228440: Executing: { Cmd , MgmtId: 132129494109518,
via: 1(10.147.40.14), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"34a817b2-e24f-4fb3-82dc-918505fcf8c8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9e132ff7-f9ef-3f17-bbc2-c44c323d96e7","id":1,"poolType":"NetworkFilesystem","host":"Host13","path":"/hyperv/primary?user=sanjeev&password=Password_123&domain=BLR","port":445,"url":"NetworkFilesystem://Host13//hyperv/primary?user=sanjeev&password=Password_123&domain=BLR/?ROLE=Primary&STOREUUID=9e132ff7-f9ef-3f17-bbc2-c44c323d96e7"}},"name":"ROOT-3","size":5368709120,"volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3,"deviceId":0,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"34a817b2-e24f-4fb3-82dc-918505fcf8c8","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR","_role":"Image"}},"name":"ROOT-3","size":5368709120,"path":"volumes/2/3","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3,"deviceId":0,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"wait":10800}}]
}
2013-12-24 19:11:56,208 DEBUG [c.c.a.t.Request] (Job-Executor-29:ctx-893d8661
ctx-d805adb4) Seq 1-1652228440: Received: { Ans: , MgmtId: 132129494109518,
via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-12-24 19:11:56,208 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) copy to image store failed:
org.apache.cloudstack.storage.command.CopyCommand failed on exception, Value
cannot be null.^M
Parameter name: VolumeObjectTO missing primary dataStore in spec {^M
"uuid": "34a817b2-e24f-4fb3-82dc-918505fcf8c8",^M
"volumeType": "ROOT",^M
"dataStore": {^M
"com.cloud.agent.api.to.NfsTO": {^M
"_url":
"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR",^M
"_role": "Image"^M
}^M
},^M
"name": "ROOT-3",^M
"size": 5368709120,^M
"path": "volumes/2/3",^M
"volumeId": 3,^M
"vmName": "i-2-3-VM",^M
"accountId": 2,^M
"id": 3,^M
"deviceId": 0,^M
"hypervisorType": "Hyperv"^M
}2013-12-24 19:11:56,394 DEBUG [c.c.a.t.Request] (Job-Executor-29:ctx-893d8661
ctx-d805adb4) Seq 3-177144205: Sending { Cmd , MgmtId: 132129494109518, via:
3(s-2-VM), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"34a817b2-e24f-4fb3-82dc-918505fcf8c8","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR","_role":"Image"}},"name":"ROOT-3","size":5368709120,"volumeId":3,"vmName":"i-2-3-VM","accountId":2,"id":3,"deviceId":0,"hypervisorType":"Hyperv"}},"wait":0}}]
}
2013-12-24 19:11:56,438 DEBUG [c.c.a.t.Request] (Job-Executor-29:ctx-893d8661
ctx-d805adb4) Seq 3-177144205: Received: { Ans: , MgmtId: 132129494109518,
via: 3, Ver: v1, Flags: 10, { Answer } }
2013-12-24 19:11:56,439 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) Volume 3 is not found on image
store 1, so no need to delete
2013-12-24 19:11:56,505 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@10d466d5), no need
to delete from object in store ref table
2013-12-24 19:11:56,735 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) Marking volume that was never
created as destroyed: Vol[9|vm=3|ROOT]
2013-12-24 19:11:56,788 ERROR [o.a.c.e.o.VolumeOrchestrator]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) migrate volume
failed:org.apache.cloudstack.storage.command.CopyCommand failed on exception,
Value cannot be null.
Parameter name: VolumeObjectTO missing primary dataStore in spec {^M
"uuid": "34a817b2-e24f-4fb3-82dc-918505fcf8c8",^M
"volumeType": "ROOT",^M
"dataStore": {^M
"com.cloud.agent.api.to.NfsTO": {^M
"_url":
"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR",^M
"_role": "Image"^M
}^M
},^M
"name": "ROOT-3",^M
"size": 5368709120,^M
"path": "volumes/2/3",^M
"volumeId": 3,^M
"vmName": "i-2-3-VM",^M
"accountId": 2,^M
"id": 3,^M
"deviceId": 0,^M
"hypervisorType": "Hyperv"^M
}
2013-12-24 19:11:56,788 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-29:ctx-893d8661 ctx-d805adb4) Failed to migration:
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:7] is
unreachable: migrate volume failed:
org.apache.cloudstack.storage.command.CopyCommand failed on exception, Value
cannot be null.^M
Agent log:
========
2013-12-24 11:25:55,436 [6] ERROR HypervResource.HypervResourceController
[d72f6849-8022-4183-87f1-3202083ed907] -
com.cloud.agent.api.GetStorageStatsCommand failed on exceptionIllegal
characters in path.^M
System.ArgumentException: Illegal characters in path.^M
at System.IO.Path.CheckInvalidPathChars(String path, Boolean
checkAdditional)^M
at
System.Security.Permissions.FileIOPermission.CheckIllegalCharacters(String[]
str)^M
at
System.Security.Permissions.FileIOPermission.AddPathList(FileIOPermissionAccess
access, AccessControlActions control, String[] pathListOrig, Boolean
checkForDuplicates, Boolean needFullPath, Boolean copyPathList)^M
at System.Security.Permissions.FileIOPermission..ctor(FileIOPermissionAccess
access, String[] pathList, Boolean checkForDuplicates, Boolean needFullPath)^M
at System.IO.Path.GetFullPath(String path)^M
at HypervResource.HypervResourceController.GetCapacityForLocalPath(String
localStoragePath, Int64& capacityBytes, Int64& availableBytes)^M
at HypervResource.HypervResourceController.GetStorageStatsCommand(Object
cmd)^M
2013-12-24 11:25:55,436 [6] INFO HypervResource.HypervResourceController
[d72f6849-8022-4183-87f1-3202083ed907] - {^M
"com.cloud.agent.api.GetStorageStatsAnswer": {^M
"result": false,^M
"details": "com.cloud.agent.api.GetStorageStatsCommand failed on
exceptionIllegal characters in path.",^M
"capacity": 0,^M
"used": 0,^M
"contextMap": {}^M
}^M
}^M
2013-12-24 11:26:20,795 [35] ERROR HypervResource.VolumeObjectTO
[1d5b7262-afc4-4b65-ab5a-a5b189817c5e] - VolumeObjectTO missing primary
dataStore in spec {^M
"uuid": "34a817b2-e24f-4fb3-82dc-918505fcf8c8",^M
"volumeType": "ROOT",^M
"dataStore": {^M
"com.cloud.agent.api.to.NfsTO": {^M
"_url":
"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR",^M
"_role": "Image"^M
}^M
},^M
"name": "ROOT-3",^M
"size": 5368709120,^M
"path": "volumes/2/3",^M
"volumeId": 3,^M
"vmName": "i-2-3-VM",^M
"accountId": 2,^M
"id": 3,^M
"deviceId": 0,^M
"hypervisorType": "Hyperv"^M
}^M
2013-12-24 11:26:20,795 [35] ERROR HypervResource.HypervResourceController
[1d5b7262-afc4-4b65-ab5a-a5b189817c5e] -
org.apache.cloudstack.storage.command.CopyCommand failed on exception, Value
cannot be null.^M
Parameter name: VolumeObjectTO missing primary dataStore in spec {^M
"uuid": "34a817b2-e24f-4fb3-82dc-918505fcf8c8",^M
"volumeType": "ROOT",^M
"dataStore": {^M
"com.cloud.agent.api.to.NfsTO": {^M
"_url":
"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR",^M
"_role": "Image"^M
}^M
},^M
"name": "ROOT-3",^M
"size": 5368709120,^M
"path": "volumes/2/3",^M
"volumeId": 3,^M
"vmName": "i-2-3-VM",^M
"accountId": 2,^M
"id": 3,^M
"deviceId": 0,^M
"hypervisorType": "Hyperv"^M
}^M
System.ArgumentNullException: Value cannot be null.^M
Parameter name: VolumeObjectTO missing primary dataStore in spec {^M
"uuid": "34a817b2-e24f-4fb3-82dc-918505fcf8c8",^M
"volumeType": "ROOT",^M
"dataStore": {^M
"com.cloud.agent.api.to.NfsTO": {^M
"_url":
"cifs://10.147.40.13/hyperv/secondary?user=sanjeev&password=Password_123&domain=BLR",^M
"_role": "Image"^M
}^M
},^M
"name": "ROOT-3",^M
"size": 5368709120,^M
"path": "volumes/2/3",^M
"volumeId": 3,^M
"vmName": "i-2-3-VM",^M
"accountId": 2,^M
"id": 3,^M
"deviceId": 0,^M
"hypervisorType": "Hyperv"^M
}^M
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
