Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
On Mar 6, 2014, at 9:56 PM, Zhangleiqiang zhangleiqi...@huawei.com wrote:
get them working. For example, in a devstack VM the only way I can get the
iSCSI target to show the new size (after an lvextend) is to delete and
recreate
the target, something jgriffiths said he doesn't want to support ;-).
I know a method can achieve it, but it maybe need the instance to pause first
(during the step2 below), but without detaching/reattaching. The steps as
follows:
1. Extend the LV
2.Refresh the size info in tgtd:
a) tgtadm --op show --mode target # get the tid and lun_id properties of
target related to the lv; the size property in output result is still the
old size before lvextend
b) tgtadm --op delete --mode logicalunit --tid={tid} --lun={lun_id} #
delete lun mapping in tgtd
c) tgtadm --op new --mode logicalunit --tid={tid} --lun={lun_id}
--backing-store=/dev/cinder-volumes/{lv-name} # re-add lun mapping
Sure, this is my current workaround, but it's what I thought we *didn't* want
to have to do.
d) tgtadm --op show --mode target #now the size property in output result
is the new size
*PS*:
a) During the procedure, the corresponding device on the compute node won't
disappear. But I am not sure the result if Instance has IO on this volume, so
maybe the instance may be paused during this procedure.
Yeah, but pausing the instance isn't an online extend. As soon as the user
can't interact with their instance, even briefly, it's an offline extend in my
view.
b) Maybe we can modify tgtadm, and make it support the operation which is
just refresh the size of backing store.
Maybe. I'd be interested in any thoughts/patches you have to accomplish this. :)
3. Rescan the lun info in compute node: iscsiadm -m node --targetname
{target_name} -R
Yeah, right now as part of this work I'm adding two extensions to Nova. One to
issue this rescan on the compute host and another to get the size of the block
device so Cinder can poll until the device is actually the new size (not an
ideal solution, but so far I don't have a better one).
I also
haven't dived into any of those other limits you mentioned (nfs_used_ratio,
etc.).
Till now, we focused on the volume which is based on *block device*. Under
this scenario, we must first try to extend the volume and notify the
hypervisor, I think one of the preconditions is to make sure the extend
operation will not affect the IO in Instance.
However, there is another scenario which maybe a little different. For
*online-extend virtual disks (qcow2, sparse, etc) whose backend storage is
file system (ext3, nfs, glusterfs, etc), the current implementation of QEMU
is as follows:
1. QEMU drain all IO
2. *QEMU* extend the virtual disk
3. QEMU resume IO
The difference is the *extend* work need be done by QEMU other than cinder
driver.
Feel free to ping me on IRC (pdmars).
I don't know your time zone, we can continue the discussion on IRC, :)
Good point. :) I'm in the US central time zone.
Paul
--
zhangleiqiang
Best Regards
-Original Message-
From: Paul Marshall [mailto:paul.marsh...@rackspace.com]
Sent: Thursday, March 06, 2014 12:56 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Luohao (brian)
Subject: Re: [openstack-dev] [Cinder] Do you think we should introduce the
online-extend feature to cinder ?
Hey,
Sorry I missed this thread a couple of days ago. I am working on a
first-pass of
this and hope to have something soon. So far I've mostly focused on getting
OpenVZ and the HP LH SAN driver working for online extend. I've had trouble
with libvirt+kvm+lvm so I'd love some help there if you have ideas about how
to
get them working. For example, in a devstack VM the only way I can get the
iSCSI target to show the new size (after an lvextend) is to delete and
recreate
the target, something jgriffiths said he doesn't want to support ;-). I also
haven't dived into any of those other limits you mentioned (nfs_used_ratio,
etc.). Feel free to ping me on IRC (pdmars).
Paul
On Mar 3, 2014, at 8:50 PM, Zhangleiqiang zhangleiqi...@huawei.com
wrote:
@john.griffith. Thanks for your information.
I have read the BP you mentioned ([1]) and have some rough thoughts about
it.
As far as I know, the corresponding online-extend command for libvirt is
blockresize, and for Qemu, the implement differs among disk formats.
For the regular qcow2/raw disk file, qemu will take charge of the
drain_all_io
and truncate_disk actions, but for raw block device, qemu will only check if
the
*Actual* size of the device is larger than current size.
I think the former need more consideration, because the extend work is done
by libvirt, Nova may need to do this first and then notify Cinder. But if we
take
allocation limit of different cinder backend drivers (such as quota,
nfs_used_ratio, nfs_oversub_ratio, etc) into account
Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
On Mar 7, 2014, at 7:55 AM, Paul Marshall paul.marsh...@rackspace.com
wrote:
On Mar 6, 2014, at 9:56 PM, Zhangleiqiang zhangleiqi...@huawei.com wrote:
get them working. For example, in a devstack VM the only way I can get the
iSCSI target to show the new size (after an lvextend) is to delete and
recreate
the target, something jgriffiths said he doesn't want to support ;-).
I know a method can achieve it, but it maybe need the instance to pause
first (during the step2 below), but without detaching/reattaching. The steps
as follows:
1. Extend the LV
2.Refresh the size info in tgtd:
a) tgtadm --op show --mode target # get the tid and lun_id properties of
target related to the lv; the size property in output result is still the
old size before lvextend
b) tgtadm --op delete --mode logicalunit --tid={tid} --lun={lun_id} #
delete lun mapping in tgtd
c) tgtadm --op new --mode logicalunit --tid={tid} --lun={lun_id}
--backing-store=/dev/cinder-volumes/{lv-name} # re-add lun mapping
Sure, this is my current workaround, but it's what I thought we *didn't* want
to have to do.
d) tgtadm --op show --mode target #now the size property in output result
is the new size
*PS*:
a) During the procedure, the corresponding device on the compute node won't
disappear. But I am not sure the result if Instance has IO on this volume,
so maybe the instance may be paused during this procedure.
Yeah, but pausing the instance isn't an online extend. As soon as the user
can't interact with their instance, even briefly, it's an offline extend in
my view.
b) Maybe we can modify tgtadm, and make it support the operation which is
just refresh the size of backing store.
Maybe. I'd be interested in any thoughts/patches you have to accomplish this.
:)
3. Rescan the lun info in compute node: iscsiadm -m node --targetname
{target_name} -R
Yeah, right now as part of this work I'm adding two extensions to Nova. One
to issue this rescan on the compute host and another to get the size of the
block device so Cinder can poll until the device is actually the new size
(not an ideal solution, but so far I don't have a better one).
Sorry, I should correct myself here: I'm adding one extension with two calls.
One to issue the rescan on the compute host and one to get the blockdev size so
Cinder can wait until it's actually the new size.
I also
haven't dived into any of those other limits you mentioned (nfs_used_ratio,
etc.).
Till now, we focused on the volume which is based on *block device*. Under
this scenario, we must first try to extend the volume and notify the
hypervisor, I think one of the preconditions is to make sure the extend
operation will not affect the IO in Instance.
However, there is another scenario which maybe a little different. For
*online-extend virtual disks (qcow2, sparse, etc) whose backend storage is
file system (ext3, nfs, glusterfs, etc), the current implementation of QEMU
is as follows:
1. QEMU drain all IO
2. *QEMU* extend the virtual disk
3. QEMU resume IO
The difference is the *extend* work need be done by QEMU other than cinder
driver.
Feel free to ping me on IRC (pdmars).
I don't know your time zone, we can continue the discussion on IRC, :)
Good point. :) I'm in the US central time zone.
Paul
--
zhangleiqiang
Best Regards
-Original Message-
From: Paul Marshall [mailto:paul.marsh...@rackspace.com]
Sent: Thursday, March 06, 2014 12:56 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Luohao (brian)
Subject: Re: [openstack-dev] [Cinder] Do you think we should introduce the
online-extend feature to cinder ?
Hey,
Sorry I missed this thread a couple of days ago. I am working on a
first-pass of
this and hope to have something soon. So far I've mostly focused on getting
OpenVZ and the HP LH SAN driver working for online extend. I've had trouble
with libvirt+kvm+lvm so I'd love some help there if you have ideas about
how to
get them working. For example, in a devstack VM the only way I can get the
iSCSI target to show the new size (after an lvextend) is to delete and
recreate
the target, something jgriffiths said he doesn't want to support ;-). I also
haven't dived into any of those other limits you mentioned (nfs_used_ratio,
etc.). Feel free to ping me on IRC (pdmars).
Paul
On Mar 3, 2014, at 8:50 PM, Zhangleiqiang zhangleiqi...@huawei.com
wrote:
@john.griffith. Thanks for your information.
I have read the BP you mentioned ([1]) and have some rough thoughts about
it.
As far as I know, the corresponding online-extend command for libvirt is
blockresize, and for Qemu, the implement differs among disk formats.
For the regular qcow2/raw disk file, qemu will take charge of the
drain_all_io
and truncate_disk actions, but for raw block device, qemu will only check
if the
*Actual* size of the device
Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
Hey, Sorry I missed this thread a couple of days ago. I am working on a first-pass of this and hope to have something soon. So far I've mostly focused on getting OpenVZ and the HP LH SAN driver working for online extend. I've had trouble with libvirt+kvm+lvm so I'd love some help there if you have ideas about how to get them working. For example, in a devstack VM the only way I can get the iSCSI target to show the new size (after an lvextend) is to delete and recreate the target, something jgriffiths said he doesn't want to support ;-). I also haven't dived into any of those other limits you mentioned (nfs_used_ratio, etc.). Feel free to ping me on IRC (pdmars). Paul On Mar 3, 2014, at 8:50 PM, Zhangleiqiang zhangleiqi...@huawei.com wrote: @john.griffith. Thanks for your information. I have read the BP you mentioned ([1]) and have some rough thoughts about it. As far as I know, the corresponding online-extend command for libvirt is “blockresize”, and for Qemu, the implement differs among disk formats. For the regular qcow2/raw disk file, qemu will take charge of the drain_all_io and truncate_disk actions, but for raw block device, qemu will only check if the *Actual* size of the device is larger than current size. I think the former need more consideration, because the extend work is done by libvirt, Nova may need to do this first and then notify Cinder. But if we take allocation limit of different cinder backend drivers (such as quota, nfs_used_ratio, nfs_oversub_ratio, etc) into account, the workflow will be more complicated. This scenario is not included by the Item 3 of BP ([1]), as it cannot be simply “just work” or notified by the compute node/libvirt after the volume is extended. This regular qcow2/raw disk files are normally stored in file system based storage, maybe the Manila project is more appropriate for this scenario? Thanks. [1]: https://blueprints.launchpad.net/cinder/+spec/inuse-extend-volume-extension -- zhangleiqiang Best Regards From: John Griffith [mailto:john.griff...@solidfire.com] Sent: Tuesday, March 04, 2014 1:05 AM To: OpenStack Development Mailing List (not for usage questions) Cc: Luohao (brian) Subject: Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ? On Mon, Mar 3, 2014 at 2:01 AM, Zhangleiqiang zhangleiqi...@huawei.com wrote: Hi, stackers: Libvirt/qemu have supported online-extend for multiple disk formats, including qcow2, sparse, etc. But Cinder only support offline-extend volumes currently. Offline-extend volume will force the instance to be shutoff or the volume to be detached. I think it will be useful if we introduce the online-extend feature to cinder, especially for the file system based driver, e.g. nfs, glusterfs, etc. Is there any other suggestions? Thanks. -- zhangleiqiang Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Zhangleiqiang, So yes, there's a rough BP for this here: [1], and some of the folks from the Trove team (pdmars on IRC) have actually started to dive into this. Last I checked with him there were some sticking points on the Nova side but we should synch up with Paul, it's been a couple weeks since I've last caught up with him. Thanks, John [1]: https://blueprints.launchpad.net/cinder/+spec/inuse-extend-volume-extension ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
Hi, stackers: Libvirt/qemu have supported online-extend for multiple disk formats, including qcow2, sparse, etc. But Cinder only support offline-extend volumes currently. Offline-extend volume will force the instance to be shutoff or the volume to be detached. I think it will be useful if we introduce the online-extend feature to cinder, especially for the file system based driver, e.g. nfs, glusterfs, etc. Is there any other suggestions? Thanks. -- zhangleiqiang Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
On Mon, Mar 3, 2014 at 2:01 AM, Zhangleiqiang zhangleiqi...@huawei.comwrote: Hi, stackers: Libvirt/qemu have supported online-extend for multiple disk formats, including qcow2, sparse, etc. But Cinder only support offline-extend volumes currently. Offline-extend volume will force the instance to be shutoff or the volume to be detached. I think it will be useful if we introduce the online-extend feature to cinder, especially for the file system based driver, e.g. nfs, glusterfs, etc. Is there any other suggestions? Thanks. -- zhangleiqiang Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Zhangleiqiang, So yes, there's a rough BP for this here: [1], and some of the folks from the Trove team (pdmars on IRC) have actually started to dive into this. Last I checked with him there were some sticking points on the Nova side but we should synch up with Paul, it's been a couple weeks since I've last caught up with him. Thanks, John [1]: https://blueprints.launchpad.net/cinder/+spec/inuse-extend-volume-extension ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ?
@john.griffith. Thanks for your information. I have read the BP you mentioned ([1]) and have some rough thoughts about it. As far as I know, the corresponding online-extend command for libvirt is blockresize, and for Qemu, the implement differs among disk formats. For the regular qcow2/raw disk file, qemu will take charge of the drain_all_io and truncate_disk actions, but for raw block device, qemu will only check if the *Actual* size of the device is larger than current size. I think the former need more consideration, because the extend work is done by libvirt, Nova may need to do this first and then notify Cinder. But if we take allocation limit of different cinder backend drivers (such as quota, nfs_used_ratio, nfs_oversub_ratio, etc) into account, the workflow will be more complicated. This scenario is not included by the Item 3 of BP ([1]), as it cannot be simply just work or notified by the compute node/libvirt after the volume is extended. This regular qcow2/raw disk files are normally stored in file system based storage, maybe the Manila project is more appropriate for this scenario? Thanks. [1]: https://blueprints.launchpad.net/cinder/+spec/inuse-extend-volume-extension -- zhangleiqiang Best Regards From: John Griffith [mailto:john.griff...@solidfire.com] Sent: Tuesday, March 04, 2014 1:05 AM To: OpenStack Development Mailing List (not for usage questions) Cc: Luohao (brian) Subject: Re: [openstack-dev] [Cinder] Do you think we should introduce the online-extend feature to cinder ? On Mon, Mar 3, 2014 at 2:01 AM, Zhangleiqiang zhangleiqi...@huawei.commailto:zhangleiqi...@huawei.com wrote: Hi, stackers: Libvirt/qemu have supported online-extend for multiple disk formats, including qcow2, sparse, etc. But Cinder only support offline-extend volumes currently. Offline-extend volume will force the instance to be shutoff or the volume to be detached. I think it will be useful if we introduce the online-extend feature to cinder, especially for the file system based driver, e.g. nfs, glusterfs, etc. Is there any other suggestions? Thanks. -- zhangleiqiang Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Zhangleiqiang, So yes, there's a rough BP for this here: [1], and some of the folks from the Trove team (pdmars on IRC) have actually started to dive into this. Last I checked with him there were some sticking points on the Nova side but we should synch up with Paul, it's been a couple weeks since I've last caught up with him. Thanks, John [1]: https://blueprints.launchpad.net/cinder/+spec/inuse-extend-volume-extension ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
