[
https://issues.apache.org/jira/browse/CLOUDSTACK-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587842#comment-13587842
]
Venkata Siva Vijayendra Bhamidipati commented on CLOUDSTACK-1228:
-----------------------------------------------------------------
Hi Kiran,
Anthony and I logged into 10.147.38.142 and took a look at the mgmt server
process and the directory structure.
[root@KiranRhel63-3 init.d]# service cloudstack-management status
cloudstack-management (pid 2941) is running...
[root@KiranRhel63-3 init.d]#
[root@KiranRhel63-3 init.d]# ps -ef | grep 2941
cloud 2941 1 3 Feb22 ? 03:59:27 /usr/lib/jvm/jre/bin/java
-Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M
-XX:MaxPermSize=800m -classpath
:::/etc/cloudstack/management:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management/bin/bootstrap.jar:/usr/share/cloudstack-management/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/usr/share/cloudstack-management
-Dcatalina.home=/usr/share/cloudstack-management -Djava.endorsed.dirs=
-Djava.io.tmpdir=/usr/share/cloudstack-management/temp
-Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
org.apache.catalina.startup.Bootstrap start
root 28753 28547 0 12:12 pts/1 00:00:00 grep 2941
[root@KiranRhel63-3 init.d]#
The mgmt server is running under the user "cloud".
Next, logged in as cloud user:
[root@KiranRhel63-3 init.d]# su - cloud
-sh-4.1$
-sh-4.1$ ls -ld /var/lib/cloud/mnt/
drwxr-xr-x. 3 cloud cloud 4096 Feb 27 12:06 /var/lib/cloud/mnt/
-sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/
drwxr-xr-x. 3 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/
-sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/7313725325399.503a5754/
drwxr-xr-x. 2 root root 4096 Feb 27 11:56
/var/lib/cloud/mnt/VM/7313725325399.503a5754/
-sh-4.1$
Things looked fine till now, but when we tried:
-sh-4.1$ mkdir -p /var/lib/cloud/mnt/VM/7313725325399.503a5767
mkdir: cannot create directory `/var/lib/cloud/mnt/VM/7313725325399.503a5767':
Permission denied
-sh-4.1$
We just changed the last two digits to create a new directory, but are hitting
a permission denied error. Checked whether /var/lib/cloud/mnt/VM or any of the
directories in the chain was already mounted, and they weren't:
-sh-4.1$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_kiranrhel633-lv_root
18G 7.8G 8.6G 48% /
tmpfs 499M 0 499M 0% /dev/shm
/dev/vda1 485M 33M 428M 8% /boot
/mnt/ISO/rhel-server-6.3-x86_64-dvd.iso
3.5G 3.5G 0 100% /media
10.147.28.7:/export/home/kiran
5.4T 2.5T 3.0T 46% /tmp/sec
10.147.28.7:/export/home/kiran/secondary
5.4T 2.5T 3.0T 46%
/var/cloudstack/mnt/VM/7614406590488.32f8324
-sh-4.1$
I am not sure what is causing this issue on your server, but it looks like a
setup issue. The mgmt server isn't able to create the mount point under
/var/lib/cloud/mnt/ and even manual creation is failing.
The workaround I can think of to immediately unblock this is for you to create
a /mnt/sec directory as the cloud user, and then mount the nfs secondary
directory to /mnt/sec/, and see how it goes. I believe the mgmt server will
proceed beyond the current point of failure then.
Also, can you please confirm what happens when you run the mgmt server as root?
Do you still see the above issue? I'm guessing that you won't. I know that you
should be able to run the mgmt server as the cloud user - I'm just asking about
root to see if it can unblock your testing user independent functionality. In
any case, you must look at why the cloud user cannot create directories in a
directory chain owned by it.
Regards,
Vijay
> Unable to Create System Vm's in the VMware Hypervisor setup.
> ------------------------------------------------------------
>
> Key: CLOUDSTACK-1228
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1228
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Install and Setup
> Affects Versions: 4.1.0
> Reporter: Kiran Koneti
> Assignee: Venkata Siva Vijayendra Bhamidipati
> Priority: Blocker
> Fix For: 4.1.0
>
> Attachments: catalina.2013-02-11.zip, catalina.zip,
> management-server-18-2-2013.zip, management-server.zip
>
>
> Created a Advanced Zone setup using the Latest 4.1 build
> (http://repo-ccp.citrix.com/releases/ASF/rhel/6.3/4.1/CloudStack-non-OSS-11-rhel6.3.tar.gz)
> While trying to create the system VM's error messages are obserevd.
> Attaching the Management server logs.
> Error message Observed:
> Executing copyTemplateFromSecondaryToPrimary. secondaryStorage:
> nfs://10.147.28.7/export/home/kiran/s, templatePathAtSecondaryStorage:
> template/tmpl/1/8/, templateName: routing-8
> 2013-02-11 23:03:41,530 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.47bfef59
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.648a617f
> 2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.2257f5b0
> 2013-02-11 23:03:41,532 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.3e22e9af
> 2013-02-11 23:03:41,533 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.1e46e9bc
> 2013-02-11 23:03:41,534 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.49fee4a0
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.c1b6a94
> 2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.5604ba42
> 2013-02-11 23:03:41,536 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.3632e0de
> 2013-02-11 23:03:41,537 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount:
> /var/lib/cloud/mnt/VM/7313725325399.22475368
> 2013-02-11 23:03:41,537 WARN [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create a mount point
> 2013-02-11 23:03:41,538 ERROR [vmware.manager.VmwareManagerImpl]
> (DirectAgent-2:10.147.40.18) Unable to create mount point for
> nfs://10.147.28.7/export/home/kiran/s
> 2013-02-11 23:03:41,538 INFO [vmware.manager.VmwareStorageManagerImpl]
> (DirectAgent-2:10.147.40.18) Secondary storage mount point: /mnt/sec
> 2013-02-11 23:03:41,538 INFO [vmware.manager.VmwareStorageManagerImpl]
> (DirectAgent-2:10.147.40.18) Executing command: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,539 DEBUG [vmware.manager.VmwareStorageManagerImpl]
> (DirectAgent-2:10.147.40.18) Executing: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-02-11 23:03:41,603 WARN [vmware.manager.VmwareStorageManagerImpl]
> (DirectAgent-2:10.147.40.18) Exception: tar --no-same-owner -xf
> /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory
> "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or
> directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
> at com.cloud.utils.script.Script.execute(Script.java:183)
> at com.cloud.utils.script.Script.execute(Script.java:161)
> at
> com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443)
> at
> com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or
> directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
> ... 15 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
