On Sun, Jul 28, 2013 at 12:17 AM, harryxiyou <[email protected]> wrote: > Hi all, > > When I add my codes of mounting gluster storage directory in > CreateDisks function of lib/cmdlib/instance_storage.py file as follows. > > > --- > lib/cmdlib/instance_storage.py | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/lib/cmdlib/instance_storage.py b/lib/cmdlib/instance_storage.py > index 4c602ae..bfb7324 100644 > --- a/lib/cmdlib/instance_storage.py > +++ b/lib/cmdlib/instance_storage.py > @@ -255,6 +255,22 @@ def CreateDisks(lu, instance, to_skip=None, > target_node_uuid=None, disks=None): > " node %s" % (file_storage_dir, > lu.cfg.GetNodeName(pnode_uuid))) > > + if instance.disk_template == constants.DT_GLUSTER_FILE: > + gluster_storage_dir = lu.cfg.GetClusterInfo().gluster_file_storage_dir > + if not os.path.ismount(gluster_storage_dir): > > + diskparams = lu.cfg.GetClusterInfo().diskparams > + glusterparams = diskparams[constants.DT_GLUSTER_FILE] > + gluster_volname = glusterparams[constants.LDP_VOLNAME] > + gluster_hostname = glusterparams[constants.LDP_HOSTNAME] > + result = lu.rpc.call_mount_gluster_storage_dir(pnode_uuid, > + gluster_hostname, > + gluster_volname, > + gluster_storage_dir) > + > + result.Raise("Failed to mount directory '%s' on" > + " node %s" % (gluster_storage_dir, > + lu.cfg.GetNodeName(pnode_uuid))) > + > disks_created = [] > for idx, device in enumerate(disks): > if to_skip and idx in to_skip: > > > > And I run "gnt-instance add -n node0.harrycluster.com -o > debootstrap+default -t gluster -s 5G --file-storage-dir mysubdir > instance0.harrycluster.com" command. It happened to me the > errors as follows. > > > root@node0:~/workshop2/ganeti# gnt-instance add -n > node0.harrycluster.com -o debootstrap+default -t gluster -s 5G > --file-storage-dir mysubdir instance0.harrycluster.com > Sat Jul 27 11:50:07 2013 * disk 0, size 5.0G > Sat Jul 27 11:50:07 2013 * creating instance disks... > Sat Jul 27 11:50:12 2013 - WARNING: Device creation failed > Failure: command execution error: > Can't create block device <Disk(type=file, logical_id=('loop', > '/srv/ganeti/gluster-file-storage/mysubdir/instance0.harrycluster.com/disk0'), > physical_id=('loop', > '/srv/ganeti/gluster-file-storage/mysubdir/instance0.harrycluster.com/disk0'), > children=None, visible as /dev/disk/0, size=5120m)> on node > node0.harrycluster.com for instance instance0.harrycluster.com: Error > while executing backend function: incomplete format > > > > > The logs in the log file is as follows: > > > root@node0:/var/log/ganeti# cat master-daemon.log | tail -40 > 2013-07-27 11:50:03,583: ganeti-masterd pid=9582/ClientReq13 INFO > Received job poll request for 3 > 2013-07-27 11:50:03,585: ganeti-masterd pid=9582/ClientReq14 INFO > Received job poll request for 3 > 2013-07-27 11:50:03,616: ganeti-masterd pid=9582/Jq3/Job3 INFO Op 1/1: > opcode INSTANCE_CREATE(instance0.harrycluster.com) waiting for locks > 2013-07-27 11:50:03,718: ganeti-masterd pid=9582/ClientReq15 INFO > Received job poll request for 3 > 2013-07-27 11:50:07,790: ganeti-masterd pid=9582/ClientReq3 INFO > Received job poll request for 3 > 2013-07-27 11:50:12,044: ganeti-masterd pid=9582/Jq3/Job3/I_CREATE > INFO Creating disk 0 for instance 'instance0.harrycluster.com' > 2013-07-27 11:50:12,148: ganeti-masterd pid=9582/Jq3/Job3/I_CREATE > WARNING Creating disk 0 for instance 'instance0.harrycluster.com' > failed > 2013-07-27 11:50:12,148: ganeti-masterd pid=9582/Jq3/Job3/I_CREATE > WARNING Device creation failed > 2013-07-27 11:50:12,201: ganeti-masterd pid=9582/Jq3/Job3 ERROR Op > 1/1: Caught exception in INSTANCE_CREATE(instance0.harrycluster.com) > Traceback (most recent call last): > File "/usr/local/lib/python2.7/dist-packages/ganeti/jqueue.py", line > 1115, in _ExecOpCodeUnlocked > timeout=timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/jqueue.py", line > 1426, in _WrapExecOpCode > return execop_fn(op, *args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 517, in ExecOpCode > calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 459, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 468, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 468, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 459, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 459, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 468, in _LockAndExecLU > result = self._LockAndExecLU(lu, level + 1, calc_timeout) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 407, in _LockAndExecLU > result = self._ExecLU(lu) > File "/usr/local/lib/python2.7/dist-packages/ganeti/mcpu.py", line > 374, in _ExecLU > result = _ProcessResult(submit_mj_fn, lu.op, lu.Exec(self.Log)) > File "/usr/local/lib/python2.7/dist-packages/ganeti/cmdlib/instance.py", > line 1282, in Exec > CreateDisks(self, iobj) > File > "/usr/local/lib/python2.7/dist-packages/ganeti/cmdlib/instance_storage.py", > line 291, in CreateDisks > raise errors.OpExecError(e.message) > OpExecError: Can't create block device <Disk(type=file, > logical_id=('loop', > '/srv/ganeti/gluster-file-storage/mysubdir/instance0.harrycluster.com/disk0'), > physical_id=('loop', > '/srv/ganeti/gluster-file-storage/mysubdir/instance0.harrycluster.com/disk0'), > children=None, visible as /dev/disk/0, size=5120m)> on node > node0.harrycluster.com for instance instance0.harrycluster.com: Error > while executing backend function: incomplete format > 2013-07-27 11:50:12,244: ganeti-masterd pid=9582/Jq3/Job3 INFO > Finished job 3, status = error > 2013-07-27 11:50:12,306: ganeti-masterd pid=9582/ClientReq1 INFO > Received job poll request for 3 > 2013-07-27 11:50:12,308: ganeti-masterd pid=9582/ClientReq2 INFO > Received job query request for 3 > > > > > root@node0:/var/log/ganeti# cat node-daemon.log | tail -20 > 2013-07-27 11:50:07,881: ganeti-noded pid=9732 INFO 999 storage dir is > /srv/ganeti/gluster-file-storage > 2013-07-27 11:50:07,881: ganeti-noded pid=9732 INFO RunCmd mount -t > glusterfs node0:/ganeti /srv/ganeti/gluster-file-storage > 2013-07-27 11:50:12,041: ganeti-noded pid=9732 INFO 999 After command executed > 2013-07-27 11:50:12,042: ganeti-noded pid=9732 INFO > 192.168.1.101:57611 POST /mount_gluster_storage_dir HTTP/1.1 200 > 2013-07-27 11:50:12,143: ganeti-noded pid=9767 ERROR Error in RPC call > Traceback (most recent call last): > File "/usr/local/lib/python2.7/dist-packages/ganeti/server/noded.py", > line 181, in HandleRequest > result = (True, method(serializer.LoadJson(req.request_body))) > File "/usr/local/lib/python2.7/dist-packages/ganeti/server/noded.py", > line 214, in perspective_blockdev_create > excl_stor) > File "/usr/local/lib/python2.7/dist-packages/ganeti/backend.py", > line 1981, in BlockdevCreate > device = bdev.Create(disk, clist, excl_stor) > File "/usr/local/lib/python2.7/dist-packages/ganeti/storage/bdev.py", > line 1804, in Create > disk.spindles, disk.params, excl_stor) > File "/usr/local/lib/python2.7/dist-packages/ganeti/storage/bdev.py", > line 866, in Create > base.ThrowError("Error in file creation: %", str(err)) > File "/usr/local/lib/python2.7/dist-packages/ganeti/storage/base.py", > line 372, in ThrowError > msg = msg % args > ValueError: incomplete format > 2013-07-27 11:50:12,146: ganeti-noded pid=9767 INFO > 192.168.1.101:57612 POST /blockdev_create HTTP/1.1 200 > > > > > When I substitute gluster_storage_dir with file_storage_dir (Eg: > If gluster_storage_dir is /srv/ganeti/gluster-file-storage and > file_storage_dir > would be /srv/ganeti/gluster-file-storage/instance0.harrycluster.com), > it works well for me. The patch would be as follows. > > --- > lib/cmdlib/instance_storage.py | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/lib/cmdlib/instance_storage.py b/lib/cmdlib/instance_storage.py > index 4c602ae..bfb7324 100644 > --- a/lib/cmdlib/instance_storage.py > +++ b/lib/cmdlib/instance_storage.py > @@ -255,6 +255,22 @@ def CreateDisks(lu, instance, to_skip=None, > target_node_uuid=None, disks=None): > " node %s" % (file_storage_dir, > lu.cfg.GetNodeName(pnode_uuid))) > > + if instance.disk_template == constants.DT_GLUSTER_FILE: > + gluster_storage_dir = file_storage_dir > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + if not os.path.ismount(gluster_storage_dir): > > + diskparams = lu.cfg.GetClusterInfo().diskparams > + glusterparams = diskparams[constants.DT_GLUSTER_FILE] > + gluster_volname = glusterparams[constants.LDP_VOLNAME] > + gluster_hostname = glusterparams[constants.LDP_HOSTNAME] > + result = lu.rpc.call_mount_gluster_storage_dir(pnode_uuid, > + gluster_hostname, > + gluster_volname, > + gluster_storage_dir) > + > + result.Raise("Failed to mount directory '%s' on" > + " node %s" % (gluster_storage_dir, > + lu.cfg.GetNodeName(pnode_uuid))) > + > disks_created = [] > for idx, device in enumerate(disks): > if to_skip and idx in to_skip: > > > > Now, it seems the error is caused by lock error or something else > but I am not clear about how to fix it. > > Would anyone please give me some suggestions? Thanks very much ;-) >
I have solved this problem ;-) The problem is that before I mount gluster storage dir ( /srv/ganeti/gluster-file-storage) a RPC has been sent to create file storage dir ( /srv/ganeti/gluster-file-storage/mysubdir/instance0.harrycluster.com ). So these two RPCs would has two jobs that may operate on the same path at the same time. After I put mount operation before the create operation, everything works well for me ;) I would update my patch. Thanks. -- Thanks Weiwei Jia (Harry Wei)
