Sowmya Krishnan created CLOUDSTACK-5248:
-------------------------------------------
Summary: [Hyper-V] Failure to create mount point for secondary
storage as cloud user, system VMs fail to come up
Key: CLOUDSTACK-5248
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5248
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Hypervisor Controller, Packaging, Storage Controller
Affects Versions: 4.3.0
Environment: Hyper-V, Advanced zone, packaged build
Reporter: Sowmya Krishnan
Priority: Critical
Fix For: 4.3.0
I am using an environment running on rpm packages generated with latest 4.3
branch. In this case, management server runs as cloud user. After zone is
enabled, it fails to create mount point for secondary storage.
Following error is seen while trying to mount secondary storage to copy
systemvmiso:
2013-11-22 15:22:39,805 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) Sending Connect to listener:
HypervServerDiscoverer
2013-11-22 15:22:39,807 DEBUG [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Setting up host 1
2013-11-22 15:22:39,809 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.27a6f465
2013-11-22 15:22:39,809 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.2abd16e5
2013-11-22 15:22:39,810 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.7dacc78e
2013-11-22 15:22:39,810 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.7d990da9
2013-11-22 15:22:39,810 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.722afb9e
2013-11-22 15:22:39,810 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.6c3d5bde
2013-11-22 15:22:39,811 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.6c84f798
2013-11-22 15:22:39,811 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.7d2f557b
2013-11-22 15:22:39,811 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.6dda4ac8
2013-11-22 15:22:39,811 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount:
/mnt/280320865129348.4ec447a4
2013-11-22 15:22:39,812 WARN [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create a mount point
2013-11-22 15:22:39,812 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create mount point for
cifs://10.102.192.100/cpg_v
ol/sowmya/secondary1_hyperv?user=svcacct_hyperv&password=[XXXX]&domain=CITRITE.NET
2013-11-22 15:22:39,814 ERROR [c.c.h.h.d.HypervServerDiscoverer]
(AgentTaskPool-10:ctx-46579019) Unable to create systemvm folder on secondary
storage. locat
ion: /mnt/sec/systemvm
2013-11-22 15:22:39,814 ERROR [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) Monitor HypervServerDiscoverer says there is an
error in the connect
process for 1 due to Unable to create systemvm folder on secondary storage.
location: /mnt/sec/systemvm
com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm
folder on secondary storage. location: /mnt/sec/systemvm
at
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer.prepareSecondaryStorageStore(HypervServerDiscoverer.java:400)
at
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer.processConnect(HypervServerDiscoverer.java:184)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:495)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1402)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1712)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1838)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy143.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1047)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-11-22 15:22:39,818 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) Host 1 is disconnecting with event
AgentDisconnected
2013-11-22 15:22:39,820 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) The next status of agent 1is Alert, current
status is Connecting
2013-11-22 15:22:39,820 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) Deregistering link for 1 with state Alert
2013-11-22 15:22:39,820 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-10:ctx-46579019) Remove Agent : 1
2013-11-22 15:22:39,820 DEBUG [c.c.a.m.DirectAgentAttache]
(AgentTaskPool-10:ctx-46579019) Processing disconnect 1(10.102.192.39)
Not sure why it is trying to mount on /mnt - cloud user wouldn't have
permissions in that dir.
The issue is not found in regular developer environment where most likely, the
process runs as ROOT. But it fails in an rpm set up
The same package was tried on VMware set up and we didn't find any issues there.
Workaround:
One workaround is to run the java process as ROOT user instead of cloud user.
But this is undesirable in production environment.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
