Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On 10/28/2014 02:50 PM, John Griffith wrote: On Tue, Oct 28, 2014 at 12:37 PM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: Great, thank you, Duncan. I will then proceed with the shared volume group. Dan On 10/28/2014 02:06 PM, Duncan Thomas wrote: Cinder volumes are always (unless you go change the default) in the form: volume-uuid, and since the string 'volume-' is never a valid uuid, then I think we can work around nova volumes fine when we come to write our tests. Sorry for the repeated circling on this, but I think I'm now happy. Thanks On 28 October 2014 17:53, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: On 10/28/2014 11:56 AM, Dean Troyer wrote: On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt Nova's disk names are of the form instance-uuid_disk_type. So deterministic but, unfortunately, not necessarily predictable. It sounds like Duncan is saying that Cinder needs a fixed prefix for testing its functionality. I will be honest, I am not optimistic about convincing Nova to change their disk naming scheme for the sake of LVM testing. Far more important changes have lingered for months and sometimes longer. It sounds like you are concerned about two issues with regard to the separate volume groups approach: 1) potential loop device shortage and 2) growing space demand. The second issue, it seems to me, will arise no matter which of the two solutions we choose. More space will be required for testing Nova's LVM functionality one way or another, although, using a shared volume group would permit a more efficient use of the available space. The first issue is, indeed, a direct consequence of the choice to use distinct volume groups. However, the number of available loop devices can be increased by passing the appropriate boot parameter to the kernel, which can be easy or hard depending on how the test VMs are spun up. I am not saying that we should necessarily go the way of separate volume groups but, assuming for the moment that changing Nova's disk naming scheme is not an option, we need to figure out what will bring the least amount of pain forcing Cinder tests to work around Nova volumes or create separate volume groups. Let me know what you think. Dan -- Dean Troyer dtro...@gmail.com mailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Duncan, I don't think it's possible to have multiple volume groups using the same physical volume[1]. In fact, counter-intuitively (at least to me) the nesting actually goes the other way with multiple physical volumes comprising a single volume group. The LVM naming scheme actually makes more sense with this hierarchy. So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Thank you, Dan 1. https://wiki.archlinux.org/index.php/LVM#LVM_Building_Blocks On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Hi Dan You're quite right, the nesting isn't as I thought it was, sorry to mislead you. It isn't a show stopper, it just makes testing some proposed useful functionality slightly harder. If nova were to namespace its volumes (e.g. start all the volume names with nova-*) then that would allow the problem to be easily worked around in the test, does that sound reasonable? On 28 October 2014 14:27, Dan Genin daniel.ge...@jhuapl.edu wrote: Duncan, I don't think it's possible to have multiple volume groups using the same physical volume[1]. In fact, counter-intuitively (at least to me) the nesting actually goes the other way with multiple physical volumes comprising a single volume group. The LVM naming scheme actually makes more sense with this hierarchy. So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Thank you, Dan 1. https://wiki.archlinux.org/index.php/LVM#LVM_Building_Blocks On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On 10/28/2014 11:56 AM, Dean Troyer wrote: On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt Nova's disk names are of the form instance-uuid_disk_type. So deterministic but, unfortunately, not necessarily predictable. It sounds like Duncan is saying that Cinder needs a fixed prefix for testing its functionality. I will be honest, I am not optimistic about convincing Nova to change their disk naming scheme for the sake of LVM testing. Far more important changes have lingered for months and sometimes longer. It sounds like you are concerned about two issues with regard to the separate volume groups approach: 1) potential loop device shortage and 2) growing space demand. The second issue, it seems to me, will arise no matter which of the two solutions we choose. More space will be required for testing Nova's LVM functionality one way or another, although, using a shared volume group would permit a more efficient use of the available space. The first issue is, indeed, a direct consequence of the choice to use distinct volume groups. However, the number of available loop devices can be increased by passing the appropriate boot parameter to the kernel, which can be easy or hard depending on how the test VMs are spun up. I am not saying that we should necessarily go the way of separate volume groups but, assuming for the moment that changing Nova's disk naming scheme is not an option, we need to figure out what will bring the least amount of pain forcing Cinder tests to work around Nova volumes or create separate volume groups. Let me know what you think. Dan -- Dean Troyer dtro...@gmail.com mailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On 10/28/2014 12:47 PM, Duncan Thomas wrote: Hi Dan You're quite right, the nesting isn't as I thought it was, sorry to mislead you. It isn't a show stopper, it just makes testing some proposed useful functionality slightly harder. If nova were to namespace its volumes (e.g. start all the volume names with nova-*) then that would allow the problem to be easily worked around in the test, does that sound reasonable? Changing Nova disk names is a lng shot. It's likely I will be doing something else by the time that gets merged:) So we are left with the two options of 1) using a shared volume group and, thus, complicating life for Cinder or 2) using separate volume groups potentially causing headaches for DevStack. I am trying to figure out which of these two is the lesser evil. It seems that Dean's concerns can be addressed, though, he still has to weigh in on the proposed mitigation approaches. I have little understanding of what problems a shared Cinder-Nova volume group would cause for Cinder testing. How hard would it be to make the tests work with a shared volume group? Dan On 28 October 2014 14:27, Dan Genin daniel.ge...@jhuapl.edu wrote: Duncan, I don't think it's possible to have multiple volume groups using the same physical volume[1]. In fact, counter-intuitively (at least to me) the nesting actually goes the other way with multiple physical volumes comprising a single volume group. The LVM naming scheme actually makes more sense with this hierarchy. So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Thank you, Dan 1. https://wiki.archlinux.org/index.php/LVM#LVM_Building_Blocks On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Cinder volumes are always (unless you go change the default) in the form: volume-uuid, and since the string 'volume-' is never a valid uuid, then I think we can work around nova volumes fine when we come to write our tests. Sorry for the repeated circling on this, but I think I'm now happy. Thanks On 28 October 2014 17:53, Dan Genin daniel.ge...@jhuapl.edu wrote: On 10/28/2014 11:56 AM, Dean Troyer wrote: On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt Nova's disk names are of the form instance-uuid_disk_type. So deterministic but, unfortunately, not necessarily predictable. It sounds like Duncan is saying that Cinder needs a fixed prefix for testing its functionality. I will be honest, I am not optimistic about convincing Nova to change their disk naming scheme for the sake of LVM testing. Far more important changes have lingered for months and sometimes longer. It sounds like you are concerned about two issues with regard to the separate volume groups approach: 1) potential loop device shortage and 2) growing space demand. The second issue, it seems to me, will arise no matter which of the two solutions we choose. More space will be required for testing Nova's LVM functionality one way or another, although, using a shared volume group would permit a more efficient use of the available space. The first issue is, indeed, a direct consequence of the choice to use distinct volume groups. However, the number of available loop devices can be increased by passing the appropriate boot parameter to the kernel, which can be easy or hard depending on how the test VMs are spun up. I am not saying that we should necessarily go the way of separate volume groups but, assuming for the moment that changing Nova's disk naming scheme is not an option, we need to figure out what will bring the least amount of pain forcing Cinder tests to work around Nova volumes or create separate volume groups. Let me know what you think. Dan -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On 28 October 2014 18:01, Dan Genin daniel.ge...@jhuapl.edu wrote: Changing Nova disk names is a lng shot. It's likely I will be doing something else by the time that gets merged:) So we are left with the two options of 1) using a shared volume group and, thus, complicating life for Cinder or 2) using separate volume groups potentially causing headaches for DevStack. I am trying to figure out which of these two is the lesser evil. It seems that Dean's concerns can be addressed, though, he still has to weigh in on the proposed mitigation approaches. I have little understanding of what problems a shared Cinder-Nova volume group would cause for Cinder testing. How hard would it be to make the tests work with a shared volume group? As I commented above, it looks like nova volumes always start with a UUID. If this is true then we can just make the cinder tests slightly more clever, since cinder volumes default to being called 'volume-UUID' so will never collide. If somebody can confirm that nova volumes will always start with a UUID then we can code around the shared volume group in the tests. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Great, thank you, Duncan. I will then proceed with the shared volume group. Dan On 10/28/2014 02:06 PM, Duncan Thomas wrote: Cinder volumes are always (unless you go change the default) in the form: volume-uuid, and since the string 'volume-' is never a valid uuid, then I think we can work around nova volumes fine when we come to write our tests. Sorry for the repeated circling on this, but I think I'm now happy. Thanks On 28 October 2014 17:53, Dan Genin daniel.ge...@jhuapl.edu wrote: On 10/28/2014 11:56 AM, Dean Troyer wrote: On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt Nova's disk names are of the form instance-uuid_disk_type. So deterministic but, unfortunately, not necessarily predictable. It sounds like Duncan is saying that Cinder needs a fixed prefix for testing its functionality. I will be honest, I am not optimistic about convincing Nova to change their disk naming scheme for the sake of LVM testing. Far more important changes have lingered for months and sometimes longer. It sounds like you are concerned about two issues with regard to the separate volume groups approach: 1) potential loop device shortage and 2) growing space demand. The second issue, it seems to me, will arise no matter which of the two solutions we choose. More space will be required for testing Nova's LVM functionality one way or another, although, using a shared volume group would permit a more efficient use of the available space. The first issue is, indeed, a direct consequence of the choice to use distinct volume groups. However, the number of available loop devices can be increased by passing the appropriate boot parameter to the kernel, which can be easy or hard depending on how the test VMs are spun up. I am not saying that we should necessarily go the way of separate volume groups but, assuming for the moment that changing Nova's disk naming scheme is not an option, we need to figure out what will bring the least amount of pain forcing Cinder tests to work around Nova volumes or create separate volume groups. Let me know what you think. Dan -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On Tue, Oct 28, 2014 at 12:37 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Great, thank you, Duncan. I will then proceed with the shared volume group. Dan On 10/28/2014 02:06 PM, Duncan Thomas wrote: Cinder volumes are always (unless you go change the default) in the form: volume-uuid, and since the string 'volume-' is never a valid uuid, then I think we can work around nova volumes fine when we come to write our tests. Sorry for the repeated circling on this, but I think I'm now happy. Thanks On 28 October 2014 17:53, Dan Genin daniel.ge...@jhuapl.edu wrote: On 10/28/2014 11:56 AM, Dean Troyer wrote: On Tue, Oct 28, 2014 at 9:27 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: So this brings us back to the original proposal of having separate backing files for Cinder and Nova which Dean thought might take too much space. Between Cinder, Nova and Swift (and Ceph, etc) everybody wants some loopback disk images. DevStack's Swift and Ceph configurations assume loopback devices and do no sharing. Duncan, could you please elaborate on the pain a single volume group is likely to cause for Cinder? Is it a show stopper? Back in the day, DevStack was built to configure Cinder (and Nova Volume before that) to use a specific existing volume group (VOLUME_GROUP_NAME) or create a loopback file if necessary. With the help of VOLUME_NAME_PREFIX and volume_name_template DevStack knew which logical volumes belong to Cinder and could Do The Right Thing. With three loopback files being created, all wanting larger and larger defaults, adding a fourth becomes Just One More Thing. If Nova's use of LVM is similar enough to Cinder's (uses deterministic naming for the LVs) I'm betting we could make it work. dt Nova's disk names are of the form instance-uuid_disk_type. So deterministic but, unfortunately, not necessarily predictable. It sounds like Duncan is saying that Cinder needs a fixed prefix for testing its functionality. I will be honest, I am not optimistic about convincing Nova to change their disk naming scheme for the sake of LVM testing. Far more important changes have lingered for months and sometimes longer. It sounds like you are concerned about two issues with regard to the separate volume groups approach: 1) potential loop device shortage and 2) growing space demand. The second issue, it seems to me, will arise no matter which of the two solutions we choose. More space will be required for testing Nova's LVM functionality one way or another, although, using a shared volume group would permit a more efficient use of the available space. The first issue is, indeed, a direct consequence of the choice to use distinct volume groups. However, the number of available loop devices can be increased by passing the appropriate boot parameter to the kernel, which can be easy or hard depending on how the test VMs are spun up. I am not saying that we should necessarily go the way of separate volume groups but, assuming for the moment that changing Nova's disk naming scheme is not an option, we need to figure out what will bring the least amount of pain forcing Cinder tests to work around Nova volumes or create separate volume groups. Let me know what you think. Dan -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Noticed my response never posted think there's something up with my mail client, so if you get this a few more times forgive me :) But The idea of sharing a VG between Nova and Cinder is only relevant in an all in one deployment anyway, it's a specific edge case for testing. It certainly (IMHO) does not warrant any changes in Nova and Cinder. Also keep in mind that at some point (I think we're already there) we need to consider whether our default gating and setup can continue to be done on a single node anyway. The answer to this seems relatively simple to me, Dean pointed out just add a loopback device specifically for Nova LVM testing and move on. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
On Tue, Oct 21, 2014 at 2:48 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: So then it is probably best to leave existing Cinder LVM code in lib/cinder_backends/lvm alone and create a similar set of lvm scripts for Nova, perhaps in lib/nova_backends/lvm? Dan On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev One thing to keep in mind when using AWS as an example is that the equivalent isn't Use LVM on Compute Node, the model is more use EBS (in our case Cinder) Volumes for Instances. We most certainly have this currently and there is some testing in the gate. It seems to be coming up in all sorts of tangential conversations lately. Personally I'd LOVE to see it more widely used, which in turn would likely lead to improvements. As far as the comment made about poor performance I'm not sure where that comes from. Sure, if you run iSCSI over a 1G network to a Cinder Volume on a loop back device I wouldn't expect great perf. If you run over a 10 Gig Network to a Cinder Volume backed by a descent/real disk I don't think the performance is as significant as some seem to think (full disclosure it's been a while since I've actually tested and looked at this closely). That being said, there's a good deal of work that should be done here to tweak it and improve the LVM driver in Cinder for example. LVM in general tends to be perceived as providing poor performance, but meh... some of that is based more in fud than in data (note I said some of it). All of that is kinda off topic but it did come up so I thought I'd chime in. The real question of LVM on Compute Nodes I'd mostly be interested in more feedback from folks that use that and why? I don't have any background or experience with it so I'd love some insight on why one chooses LVM on the Compute Node versus File System based instances? At some point consolidating the LVM code in Cinder and Nova should really happen as well (i.e. a shared library or having LVM code in oslo). Yet another topic and fortunately there are a few folks working on that for Kilo. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
As a side-note, the new AWS flavors seem to indicate that the Amazon infrastructure is moving to all ECS volumes (and all flash, possibly), both ephemeral and not. This makes sense, as fewer code paths and less interoperability complexity is a good thing. That the same balance of concerns should apply in OpenStack, seems likely. On Tue, Oct 21, 2014 at 7:59 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Did you mean EBS? I thought it was generally hard to get the same kind of performance from block storage that local ephemeral storage provides but perhaps Amazon has found a way. Life would certainly be much simpler with a single ephemeral backend. Storage pools (https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools) should provide some of the same benefits. On 10/21/2014 02:54 PM, Preston L. Bannister wrote: As a side-note, the new AWS flavors seem to indicate that the Amazon infrastructure is moving to all ECS volumes (and all flash, possibly), both ephemeral and not. This makes sense, as fewer code paths and less interoperability complexity is a good thing. That the same balance of concerns should apply in OpenStack, seems likely. On Tue, Oct 21, 2014 at 7:59 AM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Do you mean that Cinder will be confused by Nova's volumes in its volume group? Yeah, sure that would be similarly easy to implement. Thank you for the suggestion! Dan On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
Yes, I meant EBS not ECS. Too many similar acronyms... The thing about the Amazon folk is that they collect a lot of metrics, and pretty much do everything on a fairly empirical basis. This is a huge advantage. Starting thinking about what I could with good metrics and building on the performance characteristics of flash. Turns out ... I can see how this could work (and very, very well). But that requires a much longer write-up than I have time for at the moment. On Tue, Oct 21, 2014 at 12:11 PM, Dan Genin daniel.ge...@jhuapl.edu wrote: Did you mean EBS? I thought it was generally hard to get the same kind of performance from block storage that local ephemeral storage provides but perhaps Amazon has found a way. Life would certainly be much simpler with a single ephemeral backend. Storage pools ( https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools) should provide some of the same benefits. On 10/21/2014 02:54 PM, Preston L. Bannister wrote: As a side-note, the new AWS flavors seem to indicate that the Amazon infrastructure is moving to all ECS volumes (and all flash, possibly), both ephemeral and not. This makes sense, as fewer code paths and less interoperability complexity is a good thing. That the same balance of concerns should apply in OpenStack, seems likely. On Tue, Oct 21, 2014 at 7:59 AM, Dan Genin daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] Enable LVM ephemeral storage for Nova
So then it is probably best to leave existing Cinder LVM code in lib/cinder_backends/lvm alone and create a similar set of lvm scripts for Nova, perhaps in lib/nova_backends/lvm? Dan On 10/21/2014 03:10 PM, Duncan Thomas wrote: Sharing the vg with cinder is likely to cause some pain testing proposed features cinder reconciling backend with the cinder db. Creating a second vg sharing the same backend pv is easy and avoids all such problems. Duncan Thomas On Oct 21, 2014 4:07 PM, Dan Genin daniel.ge...@jhuapl.edu mailto:daniel.ge...@jhuapl.edu wrote: Hello, I would like to add to DevStack the ability to stand up Nova with LVM ephemeral storage. Below is a draft of the blueprint describing the proposed feature. Suggestions on architecture, implementation and the blueprint in general are very welcome. Best, Dan Enable LVM ephemeral storage for Nova Currently DevStack supports only file based ephemeral storage for Nova, e.g., raw and qcow2. This is an obstacle to Tempest testing of Nova with LVM ephemeral storage, which in the past has been inadvertantly broken (see for example, https://bugs.launchpad.net/nova/+bug/1373962), and to Tempest testing of new features based on LVM ephemeral storage, such as LVM ephemeral storage encryption. To enable Nova to come up with LVM ephemeral storage it must be provided a volume group. Based on an initial discussion with Dean Troyer, this is best achieved by creating a single volume group for all services that potentially need LVM storage; at the moment these are Nova and Cinder. Implementation of this feature will: * move code in lib/cinder/cinder_backends/lvm to lib/lvm with appropriate modifications * rename the Cinder volume group to something generic, e.g., devstack-vg * modify the Cinder initialization and cleanup code appropriately to use the new volume group * initialize the volume group in stack.sh, shortly before services are launched * cleanup the volume group in unstack.sh after the services have been shutdown The question of how large to make the common Nova-Cinder volume group in order to enable LVM ephemeral Tempest testing will have to be explored. Although, given the tiny instance disks used in Nova Tempest tests, the current Cinder volume group size may already be adequate. No new configuration options will be necessary, assuming the volume group size will not be made configurable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME Cryptographic Signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev