Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
Hi Zane, sorry for the delayed response. Comments inline. On 05/06/2014 09:09 PM, Zane Bitter wrote: On 05/05/14 13:40, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Dimitry and I have discussed this on IRC already (no-one changed their mind about anything as a result), but I just wanted to note here that I think even this idea is crazy. VMs are not allocated out of a vast global pool of resources. They're allocated on actual machines that have physical hardware costing real money in fixed ratios. Here's a (very contrived) example. Say your standard compute node can support 16 VCPUs and 64GB of RAM. You can sell a bunch of flavours: maybe 1 VCPU + 4GB, 2 VCPU + 8GB, 4 VCPU + 16GB... c. But if (as an extreme example) you sell a server with 1 VCPU and 64GB of RAM you have a big problem: 15 VCPUs that nobody has paid for and you can't sell. (Disks add a new dimension of wrongness to the problem.) You are assuming a public cloud provider use case above. As much as I tend to focus on the utility cloud model, where the incentives are around maximizing the usage of physical hardware by packing in as many paying tenants into a fixed resource, this is only one domain for OpenStack. There are, for good or bad, IT shops and telcos that frankly are willing to dump money into an inordinate amount of hardware -- and see that hardware be inefficiently used -- in order to appease the demands of their application customer tenants. The impulse of onboarding teams for these private cloud systems is to just say yes, with utter disregard to the overall cost efficiency of the proposed customer use cases. If there was a simple switching mechanism that allowed a deployer to turn on or off this ability to allow tenants to construct specialized instance type configurations, then who really loses here? Public or utility cloud providers would simply leave the switch to its default of off and folks who wanted to provide this functionality to their users could provide it. Of course, there are clear caveats around lack of portability to other clouds -- but let's face it, cross-cloud portability has other challenges beyond this particular point ;) The insight of flavours, which is fundamental to the whole concept of IaaS, is that users must pay the *opportunity cost* of their resource usage. If you allow users to opt, at their own convenience, to pay only the actual cost of the resources they use regardless of the opportunity cost to you, then your incentives are no longer aligned with your customers. Again, the above assumes a utility cloud model. Sadly, that isn't the only cloud model. You'll initially be very popular with the kind of customers who are taking advantage of you, but you'll have to hike prices across the board to make up the cost leading to a sort of dead-sea effect. A Gresham's Law of the cloud, if you will, where bad customers drive out good customers. Simply put, a cloud allowing users to define their own flavours *loses* to one with predefined flavours 10 times out of 10. In the above example, you just tell the customer: bad luck, you want 64GB of RAM, you buy 16 VCPUs whether you want them or not. It can't actually hurt to get _more_ than you wanted, even though you'd rather not pay for it (provided, of course, that everyone else *is* paying for it, and cross-subsidising you... which they won't). Now, it's not the OpenStack project's job to prevent operators from going bankrupt. But I think at the point where we are adding significant complexity to the project just to enable people to confirm the effectiveness of a very obviously infallible strategy for losing large amounts of money, it's time to draw a line. Actually, we're not proposing something more complex, IMO. What I've been discussing on IRC and other places is getting rid of the concept of flavours entirely except for in user interfaces, as an easy way of templatizing the creation of instances. Once an instance is launched, I've proposed that we don't store the instance_type_id with the instance any more. Right now, we store the memory, CPU, and root disk amounts in the instances table, so besides the instance_type extra_specs information, there is currently no need to keep the concept of an instance_type around after the instance launch sequence has been initiated. The instance_type is decomposed into its resource units and those resource units are used for scheduling decisions, not the flavour itself. In this way, an instance_type is nothing more than a UI template to make instance creation a bit easier. The problem to date is that
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On 20/05/14 12:17, Jay Pipes wrote: Hi Zane, sorry for the delayed response. Comments inline. On 05/06/2014 09:09 PM, Zane Bitter wrote: On 05/05/14 13:40, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Dimitry and I have discussed this on IRC already (no-one changed their mind about anything as a result), but I just wanted to note here that I think even this idea is crazy. VMs are not allocated out of a vast global pool of resources. They're allocated on actual machines that have physical hardware costing real money in fixed ratios. Here's a (very contrived) example. Say your standard compute node can support 16 VCPUs and 64GB of RAM. You can sell a bunch of flavours: maybe 1 VCPU + 4GB, 2 VCPU + 8GB, 4 VCPU + 16GB... c. But if (as an extreme example) you sell a server with 1 VCPU and 64GB of RAM you have a big problem: 15 VCPUs that nobody has paid for and you can't sell. (Disks add a new dimension of wrongness to the problem.) You are assuming a public cloud provider use case above. As much as I tend to focus on the utility cloud model, where the incentives are around maximizing the usage of physical hardware by packing in as many paying tenants into a fixed resource, this is only one domain for OpenStack. I was assuming the use case advanced in this thread, which sounded like a semi-public cloud model. However, I'm actually trying to argue from a higher level of abstraction here. In any situation where there are limited resources, optimal allocation of those resources will occur when the incentives of the suppliers and consumers of said resources are aligned, independently of whose definition of optimal you use. This applies equally to public clouds, private clouds, lemonade stands, and the proverbial two guys stranded on a desert island. In other words, it's an immutable property of economies, not anything specific to one use case. There are, for good or bad, IT shops and telcos that frankly are willing to dump money into an inordinate amount of hardware -- and see that hardware be inefficiently used -- in order to appease the demands of their application customer tenants. The impulse of onboarding teams for these private cloud systems is to just say yes, with utter disregard to the overall cost efficiency of the proposed customer use cases. Fine, but what I'm saying is that you can just give the customer _more_ than they really wanted (i.e. round up to the nearest flavour). You can charge them the same if you want - you can even decouple pricing from the flavour altogether if you want. But what you can't do is assume that, just because you gave the customer exactly what they needed and not one kilobyte more, you still get to use/sell the excess capacity you didn't allocate to them. Because you may not. If there was a simple switching mechanism that allowed a deployer to turn on or off this ability to allow tenants to construct specialized instance type configurations, then who really loses here? Public or utility cloud providers would simply leave the switch to its default of off and folks who wanted to provide this functionality to their users could provide it. Of course, there are clear caveats around lack of portability to other clouds -- but let's face it, cross-cloud portability has other challenges beyond this particular point ;) The insight of flavours, which is fundamental to the whole concept of IaaS, is that users must pay the *opportunity cost* of their resource usage. If you allow users to opt, at their own convenience, to pay only the actual cost of the resources they use regardless of the opportunity cost to you, then your incentives are no longer aligned with your customers. Again, the above assumes a utility cloud model. Sadly, that isn't the only cloud model. The only assumption is that resources are not (effectively) unlimited. You'll initially be very popular with the kind of customers who are taking advantage of you, but you'll have to hike prices across the board to make up the cost leading to a sort of dead-sea effect. A Gresham's Law of the cloud, if you will, where bad customers drive out good customers. Simply put, a cloud allowing users to define their own flavours *loses* to one with predefined flavours 10 times out of 10. In the above example, you just tell the customer: bad luck, you want 64GB of RAM, you buy 16 VCPUs whether you want them or not. It can't actually hurt to get _more_ than you wanted, even though you'd rather not pay for it (provided, of course, that everyone else *is* paying for it, and cross-subsidising you... which they won't). Now, it's
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
Hi! Comments inline. On 20/05/14 21:58, Zane Bitter zbit...@redhat.com wrote: On 20/05/14 12:17, Jay Pipes wrote: Hi Zane, sorry for the delayed response. Comments inline. You are assuming a public cloud provider use case above. As much as I tend to focus on the utility cloud model, where the incentives are around maximizing the usage of physical hardware by packing in as many paying tenants into a fixed resource, this is only one domain for OpenStack. I was assuming the use case advanced in this thread, which sounded like a semi-public cloud model. However, I'm actually trying to argue from a higher level of abstraction here. In any situation where there are limited resources, optimal allocation of those resources will occur when the incentives of the suppliers and consumers of said resources are aligned, independently of whose definition of optimal you use. This applies equally to public clouds, private clouds, lemonade stands, and the proverbial two guys stranded on a desert island. In other words, it's an immutable property of economies, not anything specific to one use case. This makes perfect sense. I¹d add one tiny bit though. ³Šoptimal of those resource will *eventually* occurŠ². For clouds, by rounding up to the nearest flavour you actually leave no space for optimisation. Even for the lemonade stands you¹d first observe what people prefer most before deciding on optimal allocation of water or soda bottles :) There are, for good or bad, IT shops and telcos that frankly are willing to dump money into an inordinate amount of hardware -- and see that hardware be inefficiently used -- in order to appease the demands of their application customer tenants. The impulse of onboarding teams for these private cloud systems is to just say yes, with utter disregard to the overall cost efficiency of the proposed customer use cases. +1. I¹d also add to add support of legacy applications as another reason for the utter disregard² Fine, but what I'm saying is that you can just give the customer _more_ than they really wanted (i.e. round up to the nearest flavour). You can charge them the same if you want - you can even decouple pricing from the flavour altogether if you want. But what you can't do is assume that, just because you gave the customer exactly what they needed and not one kilobyte more, you still get to use/sell the excess capacity you didn't allocate to them. Because you may not. Like I said above, if you round up you most definitely don¹t get to use the excess capacity. Also, where exactly would you place this rounding up functionality? Heat? Nova? A custom script that runs before deployment? Assume the tenant doesn¹t know what flavours are available, because template creation is done automatically outside of the cloud environment. If there was a simple switching mechanism that allowed a deployer to turn on or off this ability to allow tenants to construct specialized instance type configurations, then who really loses here? Public or utility cloud providers would simply leave the switch to its default of off and folks who wanted to provide this functionality to their users could provide it. Of course, there are clear caveats around lack of portability to other clouds -- but let's face it, cross-cloud portability has other challenges beyond this particular point ;) The insight of flavours, which is fundamental to the whole concept of IaaS, is that users must pay the *opportunity cost* of their resource usage. If you allow users to opt, at their own convenience, to pay only the actual cost of the resources they use regardless of the opportunity cost to you, then your incentives are no longer aligned with your customers. Again, the above assumes a utility cloud model. Sadly, that isn't the only cloud model. __ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - Dimitri ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
This is why there is a distinction between properties set on images vs properties set on flavours. Image properties, which a normal user can set, are restricted to aspects of the VM which don't involve consumption of compute host resources. Flavour properties, which only a user with 'flavourmanage' permission can change, control aspects of the VM config which consume finite compute resources. I think the VM property should give requirement of resource, including CPU features like AES-NI, HVM/PV vCPU type, PCI device type because the image may have some special requirement for the resource, or the minimal of RAM size required etc. But IMHO it's not easy to say that don't involve consumption of compute host resources. After all, it defines the type of resources. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On Mon, May 05, 2014 at 01:40:43PM -0400, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Splitting up flavours in that way doesn't really fly, especially for CPU RAM, because the properties you want to configure for NUMA policies cross both CPU RAM so cannot be sensibly separated. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On Mon, May 05, 2014 at 07:40:08PM +, Dimitri Mazmanov wrote: This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) NB, flavours have an important role as a means of access control against limited compute resources. eg if you have a limited number of hosts which have provide a large page sizes, you don't want any normal user to be able to define their own flavour which lets them consume large pages. This is why there is a distinction between properties set on images vs properties set on flavours. Image properties, which a normal user can set, are restricted to aspects of the VM which don't involve consumption of compute host resources. Flavour properties, which only a user with 'flavourmanage' permission can change, control aspects of the VM config which consume finite compute resources. If we were to allow a normal user to define flavours, we would need to come up with a way to deal with this access control requirement. One possible option, would be to not allow a normal user to create completely new flavours. Instead allow them to take an existing flavour to which they have access, and reduce (but not increase) its allocated resources. eg if the user wanted to create a flavour with 1.5 GB of RAM, they would have to have access to a pre-existing flavour which had more than 1.5 GB of RAM. eg they'd take a 2 GB flavour and override the RAM setting to be just 1.5 GB. This opens up a question of billing for hosting providers - perhaps they would allow the user this customization, but still bill them for the original flavour specs. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On Thu, May 08, 2014 at 11:22:38AM +0100, Daniel P. Berrange wrote: On Mon, May 05, 2014 at 07:40:08PM +, Dimitri Mazmanov wrote: This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) NB, flavours have an important role as a means of access control against limited compute resources. eg if you have a limited number of hosts which have provide a large page sizes, you don't want any normal user to be able to define their own flavour which lets them consume large pages. This is why there is a distinction between properties set on images vs properties set on flavours. Image properties, which a normal user can set, are restricted to aspects of the VM which don't involve consumption of compute host resources. Flavour properties, which only a user with 'flavourmanage' permission can change, control aspects of the VM config which consume finite compute resources. If we were to allow a normal user to define flavours, we would need to come up with a way to deal with this access control requirement. One possible option, would be to not allow a normal user to create completely new flavours. Instead allow them to take an existing flavour to which they have access, and reduce (but not increase) its allocated resources. eg if the user wanted to create a flavour with 1.5 GB of RAM, they would have to have access to a pre-existing flavour which had more than 1.5 GB of RAM. eg they'd take a 2 GB flavour and override the RAM setting to be just 1.5 GB. This opens up a question of billing for hosting providers - perhaps they would allow the user this customization, but still bill them for the original flavour specs. Of course if you take this idea to its logical conclusion, you would arguably not need to give users the ability to create flavours. If you only permit them to reduce the resources associated with a flavour they use, then you could just allow them to request a reduced set of resources at time they boot the instance. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
The way I was thinking this would work would be to allow flavor bundles if you will, which would allow 2 or more axes in one flavor (essentially preserving the existing functionality). Thus, if you needed NUMA, you could use those. - Original Message - From: Daniel P. Berrange berra...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Thursday, May 8, 2014 5:08:32 AM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) On Mon, May 05, 2014 at 01:40:43PM -0400, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Splitting up flavours in that way doesn't really fly, especially for CPU RAM, because the properties you want to configure for NUMA policies cross both CPU RAM so cannot be sensibly separated. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
P.S. I feel like this is something that should be discussed at the design summit. Perhaps there's an existing session this could be discussed in (the unsession, perhaps?) - Original Message - From: Solly Ross sr...@redhat.com To: Daniel P. Berrange berra...@redhat.com, OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Thursday, May 8, 2014 10:51:47 AM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) The way I was thinking this would work would be to allow flavor bundles if you will, which would allow 2 or more axes in one flavor (essentially preserving the existing functionality). Thus, if you needed NUMA, you could use those. - Original Message - From: Daniel P. Berrange berra...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Thursday, May 8, 2014 5:08:32 AM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) On Mon, May 05, 2014 at 01:40:43PM -0400, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Splitting up flavours in that way doesn't really fly, especially for CPU RAM, because the properties you want to configure for NUMA policies cross both CPU RAM so cannot be sensibly separated. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ |:| |: http://libvirt.org -o- http://virt-manager.org |:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ |:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc |:| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
(response inline) - Original Message - From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Sent: Tuesday, May 6, 2014 9:09:09 PM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) On 05/05/14 13:40, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Dimitry and I have discussed this on IRC already (no-one changed their mind about anything as a result), but I just wanted to note here that I think even this idea is crazy. VMs are not allocated out of a vast global pool of resources. They're allocated on actual machines that have physical hardware costing real money in fixed ratios. Here's a (very contrived) example. Say your standard compute node can support 16 VCPUs and 64GB of RAM. You can sell a bunch of flavours: maybe 1 VCPU + 4GB, 2 VCPU + 8GB, 4 VCPU + 16GB... c. But if (as an extreme example) you sell a server with 1 VCPU and 64GB of RAM you have a big problem: 15 VCPUs that nobody has paid for and you can't sell. (Disks add a new dimension of wrongness to the problem.) So the simple solution is to not allow a VM that uses all of the RAM on a given host (just don't have a RAM flavor that size), and then schedule the VM on a host that has minimal RAM usage but high CPU usage. This is especially true for disk, where you may have situations where you don't care if an otherwise large VM uses no disk (disks on network stores, etc). The insight of flavours, which is fundamental to the whole concept of IaaS, is that users must pay the *opportunity cost* of their resource usage. If you allow users to opt, at their own convenience, to pay only the actual cost of the resources they use regardless of the opportunity cost to you, then your incentives are no longer aligned with your customers. You'll initially be very popular with the kind of customers who are taking advantage of you, but you'll have to hike prices across the board to make up the cost leading to a sort of dead-sea effect. A Gresham's Law of the cloud, if you will, where bad customers drive out good customers. Simply put, a cloud allowing users to define their own flavours *loses* to one with predefined flavours 10 times out of 10. Note that this proposal wouldn't prevent flavors from being used -- you could still have flavor bundles (or something of the sort) that would act the way current flavors do. In the above example, you just tell the customer: bad luck, you want 64GB of RAM, you buy 16 VCPUs whether you want them or not. It can't actually hurt to get _more_ than you wanted, even though you'd rather not pay for it (provided, of course, that everyone else *is* paying for it, and cross-subsidising you... which they won't). Again, what if you also have a user who needs lots of CPUs, but a relatively small amount of RAM or disk? Now, it's not the OpenStack project's job to prevent operators from going bankrupt. But I think at the point where we are adding significant complexity to the project just to enable people to confirm the effectiveness of a very obviously infallible strategy for losing large amounts of money, it's time to draw a line. By the way, the whole theory behind this idea seems to be that this: nova server-create --cpu-flavor=4 --ram-flavour=16G --disk-flavor=200G minimises the cognitive load on the user, whereas this: nova server-create --flavor=4-64G-200G But, flavors aren't (and shouldn't be) named 16G. Realistically, it would look more like nova create --cpu-flavor=large --ram-flavor=medium --disk-flavor=small for many clouds. Additionally, keep in mind that not everyone uses the command line. Developers often forget that many users will want to use horizon, and selecting 4-64G-200G (or even large-medium-large) from a long list can be very annoying (suppose we have 6 RAM flavors and 6 disk flavors -- that's 36 flavors that start with 4-). will cause the user's brain to explode from its combinatorial complexity. I find this theory absurd. In other words, if you really want to lose some money, it's perfectly feasible with the existing flavour implementation. The operator is only ever 3 for-loops away from setting up every combination of flavours possible from combining the CPU, RAM and disk options, and can even apply whatever constraints they desire. All that said, Heat will expose any API that Nova implements. Choose wisely. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
For your first question, I'll probably create a BP sometime today. For your second question, allowing tenants to create flavors prevents one of the main parts of the flavor idea from working -- having flavors that nicely fit together to prevent wasted host resources. For instance suppose the normal system flavors used memory in powers of 2GB (2, 4, 8, 16, 32). Now suppose someone came in, created a private flavor that used 3GB of RAM. We now have 1GB of RAM that can never be used, unless someone decides to come along and create a 1GB flavor (actually, since RAM has even more granularity than that, you could have someone specify that they wanted 1.34GB of RAM, for instance, and then you have all sorts of weird stuff going on). Best Regards, Solly Ross - Original Message - From: Dimitri Mazmanov dimitri.mazma...@ericsson.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 3:40:08 PM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) If you actually have 64 flavors, though, and it's overwhelming your users, ... The users won¹t see all 64 flavor, only those they have defined and public. - Dimitri On 05/05/14 20:18, Chris Friesen chris.frie...@windriver.com wrote: On 05/05/2014 11:40 AM, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. I like this idea because it allows for greater flexibility, but I think we'd need to think carefully about how to expose it via horizon--maybe separate tabs within the overall flavors page? As a simplifying view you could keep the existing flavors which group all of them, while still allowing instances to specify each one separately if desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
Hi Solly, On 06/05/14 19:16, Solly Ross sr...@redhat.com wrote: For your first question, I'll probably create a BP sometime today. Great. Thanks. Happy to help with implementation. For your second question, allowing tenants to create flavors prevents one of the main parts of the flavor idea from working -- having flavors that nicely fit together to prevent wasted host resources. For instance suppose the normal system flavors used memory in powers of 2GB (2, 4, 8, 16, 32). Now suppose someone came in, created a private flavor that used 3GB of RAM. We now have 1GB of RAM that can never be used, unless someone decides to come along and create a 1GB flavor (actually, since RAM has even more granularity than that, you could have someone specify that they wanted 1.34GB of RAM, for instance, and then you have all sorts of weird stuff going on). When I said create custom flavor I never meant allowing the users such nonsense as specifying 1.34GB of RAM (this can be controlled by having constraints). Although some can be very meticulous :) Still an issue? My basic idea was to let the users think in terms of physical resources and based on that let them create the configuration they need (if they don’t find the right flavor in the global list). Best Regards, Solly Ross - Original Message - From: Dimitri Mazmanov dimitri.mazma...@ericsson.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 3:40:08 PM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) If you actually have 64 flavors, though, and it's overwhelming your users, ... The users won¹t see all 64 flavor, only those they have defined and public. __ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
-Original Message- From: Solly Ross [mailto:sr...@redhat.com] Sent: Tuesday, May 06, 2014 10:16 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) For your first question, I'll probably create a BP sometime today. For your second question, allowing tenants to create flavors prevents one of the main parts of the flavor idea from working -- having flavors that nicely fit together to prevent wasted host resources. For instance suppose the normal system flavors used memory in powers of 2GB (2, 4, 8, 16, 32). Now suppose someone came in, created a private flavor that used 3GB of RAM. We now have 1GB of RAM that can never be used, unless someone decides to come along and create a 1GB flavor (actually, since RAM has even more granularity than that, you could have someone specify that they wanted 1.34GB of RAM, for instance, and then you have all sorts of weird stuff going on). Hi, Solly, I don't think the 3G is really that important since it has no alignment requirement and will not cause fragmentation, at least not that much to against the previous suggestion. After all, 3G + 3G + 2G can sit in a 8G host quite well, maybe a multiplier of 64M is fair enough and should work in large scale system.. Of course, 1.34G is strange for an engineer :) --jyh Best Regards, Solly Ross - Original Message - From: Dimitri Mazmanov dimitri.mazma...@ericsson.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 3:40:08 PM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) If you actually have 64 flavors, though, and it's overwhelming your users, ... The users won¹t see all 64 flavor, only those they have defined and public. - Dimitri On 05/05/14 20:18, Chris Friesen chris.frie...@windriver.com wrote: On 05/05/2014 11:40 AM, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. I like this idea because it allows for greater flexibility, but I think we'd need to think carefully about how to expose it via horizon--maybe separate tabs within the overall flavors page? As a simplifying view you could keep the existing flavors which group all of them, while still allowing instances to specify each one separately if desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On Mon, May 05, 2014 at 07:40:08PM +, Dimitri Mazmanov wrote: This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) To clarify, that wasn't a proposal as such, it was merely a suggested specification of a Nova interface that could work, if we want to implement a Nova flavor resource in Heat which will actually be useful to a reasonable subset of our users. Here's the thread reference: http://lists.openstack.org/pipermail/openstack-dev/2013-November/019099.html I was responding to a question asking if trusts would somehow fix this problem, to which the answer is no, and describing challenges faced by anyone trying to implement custom flavor creation through Heat, because the Nova API creates global objects, not objects scoped to a project. Essentially I find the discussion of how to do this via Heat kinda backwards, we should first figure out how to solve this problem with Nova directly, then exposing whatever Nova interface solves the problem via Heat becomes trivial. If you actually have 64 flavors, though, and it's overwhelming your users, ... The users won¹t see all 64 flavor, only those they have defined and public. Well, that's the whole problem - private flavors aren't really private, and if a user defines a flavor, all other users *will* see it (if they have the admin role in any project) See my link above, --is-public false doesn't do what you think it does. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On 05/05/14 13:40, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Dimitry and I have discussed this on IRC already (no-one changed their mind about anything as a result), but I just wanted to note here that I think even this idea is crazy. VMs are not allocated out of a vast global pool of resources. They're allocated on actual machines that have physical hardware costing real money in fixed ratios. Here's a (very contrived) example. Say your standard compute node can support 16 VCPUs and 64GB of RAM. You can sell a bunch of flavours: maybe 1 VCPU + 4GB, 2 VCPU + 8GB, 4 VCPU + 16GB... c. But if (as an extreme example) you sell a server with 1 VCPU and 64GB of RAM you have a big problem: 15 VCPUs that nobody has paid for and you can't sell. (Disks add a new dimension of wrongness to the problem.) The insight of flavours, which is fundamental to the whole concept of IaaS, is that users must pay the *opportunity cost* of their resource usage. If you allow users to opt, at their own convenience, to pay only the actual cost of the resources they use regardless of the opportunity cost to you, then your incentives are no longer aligned with your customers. You'll initially be very popular with the kind of customers who are taking advantage of you, but you'll have to hike prices across the board to make up the cost leading to a sort of dead-sea effect. A Gresham's Law of the cloud, if you will, where bad customers drive out good customers. Simply put, a cloud allowing users to define their own flavours *loses* to one with predefined flavours 10 times out of 10. In the above example, you just tell the customer: bad luck, you want 64GB of RAM, you buy 16 VCPUs whether you want them or not. It can't actually hurt to get _more_ than you wanted, even though you'd rather not pay for it (provided, of course, that everyone else *is* paying for it, and cross-subsidising you... which they won't). Now, it's not the OpenStack project's job to prevent operators from going bankrupt. But I think at the point where we are adding significant complexity to the project just to enable people to confirm the effectiveness of a very obviously infallible strategy for losing large amounts of money, it's time to draw a line. By the way, the whole theory behind this idea seems to be that this: nova server-create --cpu-flavor=4 --ram-flavour=16G --disk-flavor=200G minimises the cognitive load on the user, whereas this: nova server-create --flavor=4-64G-200G will cause the user's brain to explode from its combinatorial complexity. I find this theory absurd. In other words, if you really want to lose some money, it's perfectly feasible with the existing flavour implementation. The operator is only ever 3 for-loops away from setting up every combination of flavours possible from combining the CPU, RAM and disk options, and can even apply whatever constraints they desire. All that said, Heat will expose any API that Nova implements. Choose wisely. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Flavors concept was introduced in clouds to solve issue with effective physical resource usage: 8Gb physical memory can be effectively splitted to two m2.my_small with 4Gb RAM or to eight m1.my_tiny with 1 GB. Let's consider example when your cloud have only 2 compute nodes with 8GB RAM: vm1 (m1.my_tiny) - node1 vm2 (m1.my_tiny) - node1 vm3 (m2.my_small) - node1 vm4 (m2.my_small) - node2 (since we could not spawn on node1) This leaves ability to spawn predictable 2 VMs with m1.my_tiny flavor on node1, and 2 VMs m1.my_tiny or 1 VM m2.my_small on node2. If user has ability to create any flavor that he likes, he can create flavor like mx.my_flavor with 3GB of RAM that could not be spawned on node1 at all, and leaves 1GB under-used on node2 when spawned there. If you will multiply number of nodes to 100 or even 1000 (like some of the OpenStack deployments) you will have very big memory under-usage. Do we really need to have ability to allocate any number of physical resources for VM? If yes, I suggest to make two ways to define number of physical resource allocation for VMs: with flavors and dynamically. Leaving to cloud admins/owners to decide which way they prefer they cloud resources to be allocated. Since small clouds may prefer flavors, and big clouds may have dynamic resource allocation (impact from under-usage will be not so big). As transition plan project-scoped flavors may do the job. On Fri, May 2, 2014 at 5:35 PM, Dimitri Mazmanov dimitri.mazma...@ericsson.com wrote: This topic has already been discussed last year and a use-case was described (see [1]). Here's a Heat blueprint for a new OS::Nova::Flavor resource: [2]. Several issues have been brought up after posting my implementation for review [3], all related to how flavors are defined/implemented in nova: - Only admin tenants can manage flavors due to the default admin rule in policy.json. - Per-stack flavor creation will pollute the global flavor list - If two stacks create a flavor with the same name, collision will occur, which will lead to the following error: ERROR (Conflict): Flavor with name dupflavor already exists. (HTTP 409) These and the ones described by Steven Hardy in [4] are related to the flavor scoping in Nova. Is there any plan/discussion to allow project scoped flavors in nova, similar to the Steven’s proposal for role-based scoping (see [4])? Currently the only purpose of the is_public flag is to hide the flavor from users without the admin role, but it’s still visible in all projects. Any plan to change this? Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Dimitri [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/018744.html [2] https://wiki.openstack.org/wiki/Heat/Blueprints/dynamic-flavors [3] https://review.openstack.org/#/c/90029 [4] http://lists.openstack.org/pipermail/openstack-dev/2013-November/019099.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com +7 (495) 640-4904, 0261 +7 (903) 156-0836 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). Best Regards, Solly Ross - Original Message - From: Serg Melikyan smelik...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 2:18:21 AM Subject: Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2) Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Flavors concept was introduced in clouds to solve issue with effective physical resource usage: 8Gb physical memory can be effectively splitted to two m2.my_small with 4Gb RAM or to eight m1.my_tiny with 1 GB. Let's consider example when your cloud have only 2 compute nodes with 8GB RAM: vm1 (m1.my_tiny) - node1 vm2 (m1.my_tiny) - node1 vm3 (m2.my_small) - node1 vm4 (m2.my_small) - node2 (since we could not spawn on node1) This leaves ability to spawn predictable 2 VMs with m1.my_tiny flavor on node1, and 2 VMs m1.my_tiny or 1 VM m2.my_small on node2. If user has ability to create any flavor that he likes, he can create flavor like mx.my_flavor with 3GB of RAM that could not be spawned on node1 at all, and leaves 1GB under-used on node2 when spawned there. If you will multiply number of nodes to 100 or even 1000 (like some of the OpenStack deployments) you will have very big memory under-usage. Do we really need to have ability to allocate any number of physical resources for VM? If yes, I suggest to make two ways to define number of physical resource allocation for VMs: with flavors and dynamically. Leaving to cloud admins/owners to decide which way they prefer they cloud resources to be allocated. Since small clouds may prefer flavors, and big clouds may have dynamic resource allocation (impact from under-usage will be not so big). As transition plan project-scoped flavors may do the job. On Fri, May 2, 2014 at 5:35 PM, Dimitri Mazmanov dimitri.mazma...@ericsson.com wrote: This topic has already been discussed last year and a use-case was described (see [1]). Here's a Heat blueprint for a new OS::Nova::Flavor resource: [2]. Several issues have been brought up after posting my implementation for review [3], all related to how flavors are defined/implemented in nova: * Only admin tenants can manage flavors due to the default admin rule in policy.json. * Per-stack flavor creation will pollute the global flavor list * If two stacks create a flavor with the same name, collision will occur, which will lead to the following error: ERROR (Conflict): Flavor with name dupflavor already exists. (HTTP 409) These and the ones described by Steven Hardy in [4] are related to the flavor scoping in Nova. Is there any plan/discussion to allow project scoped flavors in nova, similar to the Steven’s proposal for role-based scoping (see [4])? Currently the only purpose of the is_public flag is to hide the flavor from users without the admin role, but it’s still visible in all projects. Any plan to change this? Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Dimitri [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/018744.html [2] https://wiki.openstack.org/wiki/Heat/Blueprints/dynamic-flavors [3] https://review.openstack.org/#/c/90029 [4] http://lists.openstack.org/pipermail/openstack-dev/2013-November/019099.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com +7 (495) 640-4904, 0261 +7 (903) 156-0836 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
Excerpts from Solly Ross's message of 2014-05-05 08:23:41 -0700: Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). I agree with the conclusion you're arguing for, but it is a bit more complex than that. Flavor defines at least three things, and possibly 4: RAM, root disk, and vcpu, with an optional ephemeral disk. Because of that, the matrix of possibilities can get extremely large. Consider if you segment RAM as 1GB, 4GB, 8GB, 16GB, vcpu as 1, 2, 4, 8, and root disk as 10G, 50G, 100G, 1T. Your matrix is now 4³, 64 flavors. If you've never heard of the paradox of choice, consumers are slowed down by having too many choices. So less flavors will not only make your resource consumption easier to optimize, it will help new users move forward with more certainty. But the reality is, nobody needs an 8 CPU, 1T, 1GB flavor. Likewise, nobody needs a 1 CPU 16GB 10G server. Both are missing the mark with the common workloads. And a lot are filled in by higher level services like Trove, LBaaS, Swift, etc. So realistically, having 20-30 flavors, with groupings around the combinations users demand, is a known pattern that seems to work well. If a user has a workload that is poorly served by any of these, it probably makes the most sense for them to over-buy and demand a better sized flavor from the provider. Dynamically allocating flavors is just going to complicate things for everybody. That said, if Nova supports it, Heat should too. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
- Original Message - Excerpts from Solly Ross's message of 2014-05-05 08:23:41 -0700: Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). I agree with the conclusion you're arguing for, but it is a bit more complex than that. Flavor defines at least three things, and possibly 4: RAM, root disk, and vcpu, with an optional ephemeral disk. Because of that, the matrix of possibilities can get extremely large. In addition extra specifications may denote the passthrough of additional devices, adding another dimension. This seems likely to be the case in the use case outlined in the original thread [1]. Thanks, Steve [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/018744.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
If you actually have 64 flavors, though, and it's overwhelming your users, it would be pretty trivial to implement a flavor recommender where you input your requirements and it pops out the nearest flavor. That being said, I do think you're right that you can probably mitigate flavor explosion by trimming out the outlier flavors. 20-30 flavors is still a bit much, but with some clever naming, the burden of choosing a flavor can be lessened. Additionally, if this is all for the purpose of orchestration, we have a computer dealing with choosing the correct flavor, and if your computer has a problem dealing with a choice between 64 (or even 512) different flavors, perhaps it's time to upgrade :-P. Best Regards, Solly Ross - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 12:28:58 PM Subject: Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2) Excerpts from Solly Ross's message of 2014-05-05 08:23:41 -0700: Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). I agree with the conclusion you're arguing for, but it is a bit more complex than that. Flavor defines at least three things, and possibly 4: RAM, root disk, and vcpu, with an optional ephemeral disk. Because of that, the matrix of possibilities can get extremely large. Consider if you segment RAM as 1GB, 4GB, 8GB, 16GB, vcpu as 1, 2, 4, 8, and root disk as 10G, 50G, 100G, 1T. Your matrix is now 4³, 64 flavors. If you've never heard of the paradox of choice, consumers are slowed down by having too many choices. So less flavors will not only make your resource consumption easier to optimize, it will help new users move forward with more certainty. But the reality is, nobody needs an 8 CPU, 1T, 1GB flavor. Likewise, nobody needs a 1 CPU 16GB 10G server. Both are missing the mark with the common workloads. And a lot are filled in by higher level services like Trove, LBaaS, Swift, etc. So realistically, having 20-30 flavors, with groupings around the combinations users demand, is a known pattern that seems to work well. If a user has a workload that is poorly served by any of these, it probably makes the most sense for them to over-buy and demand a better sized flavor from the provider. Dynamically allocating flavors is just going to complicate things for everybody. That said, if Nova supports it, Heat should too. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
Excerpts from Solly Ross's message of 2014-05-05 10:01:12 -0700: If you actually have 64 flavors, though, and it's overwhelming your users, it would be pretty trivial to implement a flavor recommender where you input your requirements and it pops out the nearest flavor. That being said, I do think you're right that you can probably mitigate flavor explosion by trimming out the outlier flavors. 20-30 flavors is still a bit much, but with some clever naming, the burden of choosing a flavor can be lessened. Additionally, if this is all for the purpose of orchestration, we have a computer dealing with choosing the correct flavor, and if your computer has a problem dealing with a choice between 64 (or even 512) different flavors, perhaps it's time to upgrade :-P. When starting with nothing, users are pretty much going to have to guess. They'll have 64 things to choose from. The computer only gets involved when you have data. A piece of software which takes measurements of resource constraints of your application, and understands how to then choose a flavor is not exactly simple. One also needs business rules, as sometimes you may want to accept being resource constrained in exchange for reducing cost. Though it would be a pretty cool feature if one could simply run an agent on their vm and it would be able to identify the flavor that would be most efficient. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
I guess I need to describe the use-case first. An example of Telco application is IP Multimedia Subsystem (IMS) [1], which is a fairly complex beast. Each component of IMS can have very different requirements on the computing resources. If we try to capture everything in terms of flavors the list of flavors can grow very quickly and still be specific to one single application. There¹s also many more apps to deploy. Agree, one can say, just round to the best matching flavor! Will work, but not the most efficient solution (a set of 4-5 global flavors will not provide the best fitting model for every VM we need to spawn). For such applications a flavor is not the lowest level of granularity. RAM, CPU, Disk is. Hence the question. In OpenStack, tenants are bound to think in terms flavors. And if this model is the lowest level of granularity, then dynamic creation of flavors actually supports this model and allows non-trivial applications to use flavors (I guess this is why this question had been raised last year by NSN). But, there are some issues related to this :) and these issues I have written down in my first mail. Dimitri [1] http://en.wikipedia.org/wiki/IP_Multimedia_Subsystem On 05/05/14 17:23, Solly Ross sr...@redhat.com wrote: Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). Best Regards, Solly Ross - Original Message - From: Serg Melikyan smelik...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 2:18:21 AM Subject: Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2) Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Flavors concept was introduced in clouds to solve issue with effective physical resource usage: 8Gb physical memory can be effectively splitted to two m2.my_small with 4Gb RAM or to eight m1.my_tiny with 1 GB. Let's consider example when your cloud have only 2 compute nodes with 8GB RAM: vm1 (m1.my_tiny) - node1 vm2 (m1.my_tiny) - node1 vm3 (m2.my_small) - node1 vm4 (m2.my_small) - node2 (since we could not spawn on node1) This leaves ability to spawn predictable 2 VMs with m1.my_tiny flavor on node1, and 2 VMs m1.my_tiny or 1 VM m2.my_small on node2. If user has ability to create any flavor that he likes, he can create flavor like mx.my_flavor with 3GB of RAM that could not be spawned on node1 at all, and leaves 1GB under-used on node2 when spawned there. If you will multiply number of nodes to 100 or even 1000 (like some of the OpenStack deployments) you will have very big memory under-usage. Do we really need to have ability to allocate any number of physical resources for VM? If yes, I suggest to make two ways to define number of physical resource allocation for VMs: with flavors and dynamically. Leaving to cloud admins/owners to decide which way they prefer they cloud resources to be allocated. Since small clouds may prefer flavors, and big clouds may have dynamic resource allocation (impact from under-usage will be not so big). As transition plan project-scoped flavors may do the job. On Fri, May 2, 2014 at 5:35 PM, Dimitri Mazmanov dimitri.mazma...@ericsson.com wrote: This topic has already been discussed last year and a use-case was described (see [1]). Here's a Heat blueprint for a new OS::Nova::Flavor resource: [2]. Several issues have been brought up after posting my implementation for review [3], all related to how flavors are defined/implemented in nova: * Only admin tenants can manage flavors due to the default admin rule in policy.json. * Per-stack flavor creation will pollute the global flavor list * If two stacks create a flavor with the same name, collision will occur, which will lead to the following error: ERROR (Conflict): Flavor with name dupflavor already exists. (HTTP 409) These and the ones described by Steven Hardy in [4] are related to the flavor scoping in Nova. Is there any plan/discussion to allow project scoped flavors in nova, similar to the Steven¹s proposal for role-based scoping (see [4])? Currently the only purpose of the is_public flag is to hide the flavor from users without the admin role, but it¹s still visible in all projects. Any plan to change this? Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. Best Regards, Solly Ross P.S. For people who use flavor names to convey information about the workload, that probably a job better done by the VM tagging proposal (https://review.openstack.org/#/c/91444/). - Original Message - From: Dimitri Mazmanov dimitri.mazma...@ericsson.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 1:20:09 PM Subject: Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2) I guess I need to describe the use-case first. An example of Telco application is IP Multimedia Subsystem (IMS) [1], which is a fairly complex beast. Each component of IMS can have very different requirements on the computing resources. If we try to capture everything in terms of flavors the list of flavors can grow very quickly and still be specific to one single application. There¹s also many more apps to deploy. Agree, one can say, just round to the best matching flavor! Will work, but not the most efficient solution (a set of 4-5 global flavors will not provide the best fitting model for every VM we need to spawn). For such applications a flavor is not the lowest level of granularity. RAM, CPU, Disk is. Hence the question. In OpenStack, tenants are bound to think in terms flavors. And if this model is the lowest level of granularity, then dynamic creation of flavors actually supports this model and allows non-trivial applications to use flavors (I guess this is why this question had been raised last year by NSN). But, there are some issues related to this :) and these issues I have written down in my first mail. Dimitri [1] http://en.wikipedia.org/wiki/IP_Multimedia_Subsystem On 05/05/14 17:23, Solly Ross sr...@redhat.com wrote: Just to expand a bit on this, flavors are supposed to be the lowest level of granularity, and the general idea is to round to the nearest flavor (so if you have a VM that requires 3GB of RAM, go with a 4GB flavor). Hence, in my mind, it doesn't make any sense to create flavors on the fly; you should have enough flavors to suit your needs, but I can't really think of a situation where you'd need so much granularity that you'd need to create new flavors on the fly (assuming, of course, that you planned ahead and created enough flavors that you don't have VMs that are extremely over-allocated). Best Regards, Solly Ross - Original Message - From: Serg Melikyan smelik...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, May 5, 2014 2:18:21 AM Subject: Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2) Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Flavors concept was introduced in clouds to solve issue with effective physical resource usage: 8Gb physical memory can be effectively splitted to two m2.my_small with 4Gb RAM or to eight m1.my_tiny with 1 GB. Let's consider example when your cloud have only 2 compute nodes with 8GB RAM: vm1 (m1.my_tiny) - node1 vm2 (m1.my_tiny) - node1 vm3 (m2.my_small) - node1 vm4 (m2.my_small) - node2 (since we could not spawn on node1) This leaves ability to spawn predictable 2 VMs with m1.my_tiny flavor on node1, and 2 VMs m1.my_tiny or 1 VM m2.my_small on node2. If user has ability to create any flavor that he likes, he can create flavor like mx.my_flavor with 3GB of RAM that could not be spawned on node1 at all, and leaves 1GB under-used on node2 when spawned there. If you will multiply number of nodes to 100 or even 1000 (like some of the OpenStack deployments) you will have very big memory under-usage. Do we really need to have ability to allocate any number of physical resources for VM? If yes, I suggest to make two ways to define number of physical resource allocation for VMs: with flavors and dynamically. Leaving to cloud admins/owners to decide which way they prefer they cloud resources to be allocated. Since small clouds may prefer flavors, and big clouds may have dynamic resource allocation (impact from under-usage will be not so big). As transition plan project-scoped flavors may do the job. On Fri, May 2, 2014 at 5:35 PM, Dimitri Mazmanov dimitri.mazma...@ericsson.com wrote: This topic has already been discussed last year and a use-case was described (see [1]). Here's a Heat blueprint for a new OS::Nova::Flavor resource: [2]. Several issues have been brought up after posting my implementation for review [3
Re: [openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
On 05/05/2014 10:51 AM, Steve Gordon wrote: In addition extra specifications may denote the passthrough of additional devices, adding another dimension. This seems likely to be the case in the use case outlined in the original thread [1]. Thanks, Steve [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/018744.html Agreed. The ability to set arbitrary metadata on the flavor means that you could realistically have many different flavors all with identical virtual hardware but different metadata. As one example, the flavor metadata can be used to match against host aggregate metadata. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On 05/05/2014 11:40 AM, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. I like this idea because it allows for greater flexibility, but I think we'd need to think carefully about how to expose it via horizon--maybe separate tabs within the overall flavors page? As a simplifying view you could keep the existing flavors which group all of them, while still allowing instances to specify each one separately if desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
On 05/05/2014 12:18 PM, Chris Friesen wrote: As a simplifying view you could keep the existing flavors which group all of them, while still allowing instances to specify each one separately if desired. Also, if we're allowing the cpu/memory/disk to be specified independently at instance boot time, we might want to allow for arbitrary metadata to be specified as well (that would be matched as per the existing flavor extra_spec). Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] [Heat] Custom Nova Flavor creation through Heat (pt.2)
This is good! Is there a blueprint describing this idea? Or any plans describing it in a blueprint? Would happily share the work. Should we mix it with flavors in horizon though? I¹m thinking of having a separate ³Resources² page, wherein the user can ³define² resources. I¹m not a UX expert though. But let me come back to the project-scoped flavor creation issues. Why do you think it¹s such a bad idea to let tenants create flavors for their project specific needs? I¹ll refer again to the Steve Hardy¹s proposal: - Normal user : Can create a private flavor in a tenant where they have the Member role (invisible to any other users) - Tenant Admin user : Can create public flavors in the tenants where they have the admin role (visible to all users in the tenant) - Domain admin user : Can create public flavors in the domains where they have the admin role (visible to all users in all tenants in that domain) If you actually have 64 flavors, though, and it's overwhelming your users, ... The users won¹t see all 64 flavor, only those they have defined and public. - Dimitri On 05/05/14 20:18, Chris Friesen chris.frie...@windriver.com wrote: On 05/05/2014 11:40 AM, Solly Ross wrote: One thing that I was discussing with @jaypipes and @dansmith over on IRC was the possibility of breaking flavors down into separate components -- i.e have a disk flavor, a CPU flavor, and a RAM flavor. This way, you still get the control of the size of your building blocks (e.g. you could restrict RAM to only 2GB, 4GB, or 16GB), but you avoid exponential flavor explosion by separating out the axes. I like this idea because it allows for greater flexibility, but I think we'd need to think carefully about how to expose it via horizon--maybe separate tabs within the overall flavors page? As a simplifying view you could keep the existing flavors which group all of them, while still allowing instances to specify each one separately if desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova][Heat] Custom Nova Flavor creation through Heat (pt.2)
This topic has already been discussed last year and a use-case was described (see [1]). Here's a Heat blueprint for a new OS::Nova::Flavor resource: [2]. Several issues have been brought up after posting my implementation for review [3], all related to how flavors are defined/implemented in nova: * Only admin tenants can manage flavors due to the default admin rule in policy.json. * Per-stack flavor creation will pollute the global flavor list * If two stacks create a flavor with the same name, collision will occur, which will lead to the following error: ERROR (Conflict): Flavor with name dupflavor already exists. (HTTP 409) These and the ones described by Steven Hardy in [4] are related to the flavor scoping in Nova. Is there any plan/discussion to allow project scoped flavors in nova, similar to the Steven’s proposal for role-based scoping (see [4])? Currently the only purpose of the is_public flag is to hide the flavor from users without the admin role, but it’s still visible in all projects. Any plan to change this? Having project-scoped flavors will rid us of the identified issues, and will allow a more fine-grained way of managing physical resources. Dimitri [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/018744.html [2] https://wiki.openstack.org/wiki/Heat/Blueprints/dynamic-flavors [3] https://review.openstack.org/#/c/90029 [4] http://lists.openstack.org/pipermail/openstack-dev/2013-November/019099.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
