Re: [openstack-dev] [Murano] Object-oriented approach for defining Murano Applications
On Mon, Feb 24, 2014 at 4:20 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi Keith, Thank you for bringing up this question. We think that it could be done inside Heat. This is a part of our future roadmap to bring more stuff to Heat and pass all actual work to the heat engine. However it will require a collaboration between Heat and Murano teams, so that is why we want to have incubated status, to start better integration with other projects being a part of OpenStack community. I will understand Heat team when they refuse to change Heat templates to satisfy the requirements of the project which does not officially belong to OpenStack. With incubation status it will be much easier. As for the actual work, backups and snapshots are processes. It will be hard to express them in a good way in current HOT template. We see that we will use Mistral resources defined in Heat which will trig the events for backup and backup workflow associated with the application can be defined outside of Heat. I don't think that Heat team will include workflow definitions as a part of template format, while they can allow us to use resources which reference such workflows stored in a catalog. It can be an extension for HOT Software config for example, but we need to validate this approach with the heat team. For what it's worth, there's already precedent for including non-OpenStack resource plugins in Heat, in a contrib directory (which is still tested with the CI infrastructure). -- IRC: radix Christopher Armstrong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] lazy translation is breaking Heat
This change was recently merged: https://review.openstack.org/#/c/69133/ Unfortunately it didn't enable lazy translations for the unit tests, so it didn't catch the many places in Heat that won't work when lazy translations are enabled. Notably there are a lot of cases where the code adds the result of a call to _() with another string, and Message objects (which are returned from _ when lazy translations are enabled) can't be added, resulting in an exception being raised. I think the best course of action would be to revert this change, and then reintroduce it along with patches to fix all the problems, while enabling it for the unit tests so bugs won't be reintroduced in the future. Interestingly it also didn't fail any of the tempest tests, I'm not sure why. -- IRC: radix Christopher Armstrong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] lazy translation is breaking Heat
On Tue, Feb 18, 2014 at 11:14 AM, Jay S Bryant jsbry...@us.ibm.com wrote: All, Myself and Jim Carey have been working on getting the right solution for making lazy_translation work through Nova and Cinder. The patch should have also had changes to remove the use of str() in any LOG or exception messages as well as the removal of any places where strings were being '+' ed together. In the case of Cinder we are doing it as two separate patches that are dependent. I am surprised that this change got past Jenkins. In the case of Cinder and Nova unit test caught a number of problems. We will make sure to work with Liang Chen to avoid this happening again.https://review.openstack.org/#/dashboard/7135 fwiw Thomas Hervé has posted a patch to revert the introduction of lazy translation: https://review.openstack.org/#/c/74454/ -- IRC: radix Christopher Armstrong ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] [TripleO] Rolling updates spec re-written. RFC
in a scaling group). I think another use case is temporarily removing a server from a load balancer when it's being e.g. resized. -- IRC: radix http://twitter.com/radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] [TripleO] Rolling updates spec re-written. RFC
On Tue, Feb 4, 2014 at 7:34 PM, Robert Collins robe...@robertcollins.netwrote: On 5 February 2014 13:14, Zane Bitter zbit...@redhat.com wrote: That's not a great example, because one DB server depends on the other, forcing them into updating serially anyway. I have to say that even in general, this whole idea about applying update policies to non-grouped resources doesn't make a whole lot of sense to me. For non-grouped resources you control the resource definitions individually - if you don't want them to update at a particular time, you have the option of just not updating them. Well, I don't particularly like the idea of doing thousands of discrete heat stack-update calls, which would seem to be what you're proposing. On groups: autoscale groups are a problem for secure minded deployments because every server has identical resources (today) and we very much want discrete credentials per server - at least this is my understanding of the reason we're not using scaling groups in TripleO. Where you _do_ need it is for scaling groups where every server is based on the same launch config, so you need a way to control the members individually - by batching up operations (done), adding delays (done) or, even better, notifications and callbacks. So it seems like doing 'rolling' updates for any random subset of resources is effectively turning Heat into something of a poor-man's workflow service, and IMHO that is probably a mistake. I mean to reply to the other thread, but here is just as good :) - heat as a way to describe the intended state, and heat takes care of transitions, is a brilliant model. It absolutely implies a bunch of workflows - the AWS update policy is probably the key example. Being able to gracefully, *automatically* work through a transition between two defined states, allowing the nodes in question to take care of their own needs along the way seems like a pretty core function to fit inside Heat itself. Its not at all the same as 'allow users to define abitrary workflows'. -Rob Agreed. I have been assuming that the autoscaling service outside of the Heat engine would need to send several pre-calculated template changes in sequence in order to implement rolling updates for resource groups, but I think it would be much much better if Heat could take care of this as a core feature. -- Christopher Armstrong http://twitter.com/radix/ http://github.com/radix/ http://radix.twistedmatrix.com/ http://planet-if.com/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] [TripleO] Rolling updates spec re-written. RFC
Heya Clint, this BP looks really good - it should significantly simplify the implementation of scaling if this becomes a core Heat feature. Comments below. On Mon, Feb 3, 2014 at 2:46 PM, Thomas Herve thomas.he...@enovance.comwrote: So, I wrote the original rolling updates spec about a year ago, and the time has come to get serious about implementation. I went through it and basically rewrote the entire thing to reflect the knowledge I have gained from a year of working with Heat. Any and all comments are welcome. I intend to start implementation very soon, as this is an important component of the HA story for TripleO: https://wiki.openstack.org/wiki/Heat/Blueprints/RollingUpdates Hi Clint, thanks for pushing this. First, I don't think RollingUpdatePattern and CanaryUpdatePattern should be 2 different entities. The second just looks like a parametrization of the first (growth_factor=1?). Agreed. I then feel that using (abusing?) depends_on for update pattern is a bit weird. Maybe I'm influenced by the CFN design, but the separate UpdatePolicy attribute feels better (although I would probably use a property). I guess my main question is around the meaning of using the update pattern on a server instance. I think I see what you want to do for the group, where child_updating would return a number, but I have no idea what it means for a single resource. Could you detail the operation a bit more in the document? I agree that depends_on is weird and I think it should be avoided. I'm not sure a property is the right decision, though, assuming that it's the heat engine that's dealing with the rolling updates -- I think having the engine reach into a resource's properties would set a strange precedent. The CFN design does seem pretty reasonable to me, assuming an update_policy field in a HOT resource, referring to the policy that the resource should use. It also seems that the interface you're creating (child_creating/child_updating) is fairly specific to your use case. For autoscaling we have a need for more generic notification system, it would be nice to find common grounds. Maybe we can invert the relationship? Add a notified_resources attribute, which would call hooks on the parent when actions are happening. Yeah, this would be really helpful for stuff like load balancer notifications (and any of a number of different resource relationships). -- IRC: radix http://twitter.com/radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Unicode strings in Python3
On Thu, Dec 5, 2013 at 3:26 AM, Julien Danjou jul...@danjou.info wrote: On Wed, Dec 04 2013, Georgy Okrokvertskhov wrote: Quick summary: you can't use unicode() function and u' ' strings in Pyhton3. Not that it's advised, but you can use u' ' back again with Python 3.3. And this is a very useful feature for projects that want to have a single codebase that runs on both python 2 and python 3, so it's worth taking advantage of. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] Stack convergence first steps
On Thu, Dec 5, 2013 at 3:50 PM, Anderson Mesquita anderson...@thoughtworks.com wrote: Hey stackers, We've been working towards making stack convergence ( https://blueprints.launchpad.net/heat/+spec/stack-convergence) one step closer to being ready at a time. After the first patch was submitted we got positive feedback on it as well as some good suggestions as to how to move it forward. The first step (https://blueprints.launchpad.net/heat/+spec/stack-check) is to get all the statuses back from the real world resources and update our stacks accordingly so that we'll be able to move on to the next step: converge it to the desired state, fixing any errors that may have happened. We just submitted another WiP for review, and as we were doing it, a few questions were raised and we'd like to get everybody's input on them. Our main concern is around the use and purpose of the `status` of a stack/resource. `status` currently appears to represent the status of the last action taken, and it seems that we may need to repurpose it or possibly create something else to represent a stack's health (i.e. everything is up and running as expected, something smells fishy, something broke, stack's is doomed). We described this thoroughly here: https://etherpad.openstack.org/p/heat-convergence Any thoughts? Cheers, I think a lot of OpenStack projects use status fields as status of the most recent operation, and I think it's totally wrong. status should be a known state of the _object_, not an action, and if we need statuses for operations, then those operations should be addressable REST objects. Of course there are cases where object status should be updated to reflect an operating status if it's a completely exclusive operation (BUILDING and DELETING make sense, for example). -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][horizon]Heat UI related requirements roadmap
On Tue, Nov 26, 2013 at 3:24 PM, Tim Schnell tim.schn...@rackspace.comwrote: So the originally question that I attempted to pose was, Can we add a schema-less metadata section to the template that can be used for a variety of purposes?. It looks like the answer is no, we need to discuss the features that would go in the metadata section and add them to the HOT specification if they are viable. I don't necessarily agree with this answer but I accept it as viable and take responsibility for the long-winded process that it took to get to this point. I think some valid points have been made and I have re-focused my efforts into the following proposed solution. I am fine with getting rid of the concept of a schema-less metadata section. If we can arrive at a workable design for a few use cases then I think that we won't need to discuss any of the options that Zane mentioned for handling the metadata section, comments, separate file, or in the template body. Use Case #1 I see valid value in being able to group templates based on a type or keyword. This would allow any client, Horizon or a Template Catalog service, to better organize and handle display options for an end-user. I believe that Ladislav initially proposed a solution that will work here. So I will second a proposal that we add a new top-level field to the HOT specification called keywords that contains this template type. keywords: wordpress, mysql, etc My immediate inclination would be to just make keywords/tags out-of-band metadata managed by the template repository. I imagine this would be something that would be very useful to change without having to edit the template anyway. Use Case #2 The template author should also be able to explicitly define a help string that is distinct and separate from the description of an individual parameter. An example where this use case originated was with Nova Keypairs. The description of a keypair parameter might be something like, This is the name of a nova key pair that will be used to ssh to the compute instance. A help string for this same parameter would be, To learn more about nova keypairs click on this help article. I propose adding an additional field to the parameter definition: Parameters: parameter name: description: This is the name of a nova key pair that will be used to ssh to the compute instance. help: To learn more about nova key pairs click on this a href=/some/url/help article/a. This one seems a bit weirder. I don't really understand what's wrong with just adding this content to the description field. However, if there are currently any objects in HOT that don't have any mechanism for providing a description, we should definitely add them where they're missing. Do you think we need to extend the semantics of the description field to allow HTML? Use Case #3 Grouping parameters would help the client make smarter decisions about how to display the parameters for input to the end-user. This is so that all parameters related to some database resource can be intelligently grouped together. In addition to grouping these parameters together, there should be a method to ensuring that the order within the group of parameters can be explicitly stated. This way, the client can return a group of database parameters and the template author can indicate that the database instance name should be first, then the username, then the password, instead of that group being returned in a random order. Parameters: db_name: group: db order: 0 db_username: group: db order: 1 db_password: group: db order: 2 web_node_name: group: web_node order: 0 keypair: group: web_node order: 1 Have you considered just rendering them in the order that they appear in the template? I realize it's not the name (since you don't have any group names that you could use as a title for boxes around groups of parameters), but it might be a good enough compromise. If you think it's absolutely mandatory to be able to group them in named groups, then I would actually propose a prettier syntax: ParameterGroups: db: name: ... username: ... password: ... web_node: name: ... keypair: ... The ordering would be based on the ordering that they appear within the template, and you wouldn't have to repeat yourself naming the group for each parameter. Thanks very much for writing up these use cases! -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing
Re: [openstack-dev] [heat][horizon]Heat UI related requirements roadmap
On Tue, Nov 26, 2013 at 4:35 PM, Tim Schnell tim.schn...@rackspace.comwrote: From: Christopher Armstrong chris.armstr...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, November 26, 2013 4:02 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [heat][horizon]Heat UI related requirements roadmap On Tue, Nov 26, 2013 at 3:24 PM, Tim Schnell tim.schn...@rackspace.com wrote: So the originally question that I attempted to pose was, Can we add a schema-less metadata section to the template that can be used for a variety of purposes?. It looks like the answer is no, we need to discuss the features that would go in the metadata section and add them to the HOT specification if they are viable. I don't necessarily agree with this answer but I accept it as viable and take responsibility for the long-winded process that it took to get to this point. I think some valid points have been made and I have re-focused my efforts into the following proposed solution. I am fine with getting rid of the concept of a schema-less metadata section. If we can arrive at a workable design for a few use cases then I think that we won't need to discuss any of the options that Zane mentioned for handling the metadata section, comments, separate file, or in the template body. Use Case #1 I see valid value in being able to group templates based on a type or keyword. This would allow any client, Horizon or a Template Catalog service, to better organize and handle display options for an end-user. I believe that Ladislav initially proposed a solution that will work here. So I will second a proposal that we add a new top-level field to the HOT specification called keywords that contains this template type. keywords: wordpress, mysql, etc My immediate inclination would be to just make keywords/tags out-of-band metadata managed by the template repository. I imagine this would be something that would be very useful to change without having to edit the template anyway. *I'm not exactly sure what you are suggesting here, but I think that adding these keywords to the template will be less error prone than attempting to derive them some other way.* Basically, I'm just suggesting putting the tags outside of template. Not deriving them -- I still think they should be explicitly specified, but just putting them in e.g. the database instead of directly in the template. Basically, in a public repository of templates, I can imagine tags being based on third-party or moderator input, instead of just based on what the template author says. Keeping them outside of the template would allow content moderators to do that without posting a new version of the template. Anyway, I don't feel that strongly about this - if there's a strong enough desire to see tags in the template, then I won't argue against it. Use Case #2 The template author should also be able to explicitly define a help string that is distinct and separate from the description of an individual parameter. An example where this use case originated was with Nova Keypairs. The description of a keypair parameter might be something like, This is the name of a nova key pair that will be used to ssh to the compute instance. A help string for this same parameter would be, To learn more about nova keypairs click on this help article. I propose adding an additional field to the parameter definition: Parameters: parameter name: description: This is the name of a nova key pair that will be used to ssh to the compute instance. help: To learn more about nova key pairs click on this a href=/some/url/help article/a. This one seems a bit weirder. I don't really understand what's wrong with just adding this content to the description field. However, if there are currently any objects in HOT that don't have any mechanism for providing a description, we should definitely add them where they're missing. Do you think we need to extend the semantics of the description field to allow HTML? *Description and help are separate things from a UI perspective. A description might be displayed as a label in a form or in a paragraph somewhere around the input. A help string is typically displayed as hover text when focusing on the input or hovering/clicking on a question mark icon next to the field. We could technically separate these things in the code but because they serve separate purposes I would prefer to have them be defined explicitly.* Okay, fair enough. Use Case #3 Grouping parameters would help the client make smarter decisions about how to display the parameters for input to the end-user. This is so that all parameters related to some database resource can
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 21, 2013 at 5:18 AM, Zane Bitter zbit...@redhat.com wrote: On 20/11/13 23:49, Christopher Armstrong wrote: On Wed, Nov 20, 2013 at 2:07 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 20/11/13 16:07, Christopher Armstrong wrote: On Tue, Nov 19, 2013 at 4:27 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com mailto:zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 19/11/13 19:14, Christopher Armstrong wrote: thought we had a workable solution with the LoadBalancerMember idea, which you would use in a way somewhat similar to CinderVolumeAttachment in the above example, to hook servers up to load balancers. I haven't seen this proposal at all. Do you have a link? How does it handle the problem of wanting to notify an arbitrary service (i.e. not necessarily a load balancer)? It's been described in the autoscaling wiki page for a while, and I thought the LBMember idea was discussed at the summit, but I wasn't there to verify that :) https://wiki.openstack.org/wiki/Heat/AutoScaling#LBMember.3F Basically, the LoadBalancerMember resource (which is very similar to the CinderVolumeAttachment) would be responsible for removing and adding IPs from/to the load balancer (which is actually a direct mapping to the way the various LB APIs work). Since this resource lives with the server resource inside the scaling unit, we don't really need to get anything _out_ of that stack, only pass _in_ the load balancer ID. I see a couple of problems with this approach: 1) It makes the default case hard. There's no way to just specify a server and hook it up to a load balancer like you can at the moment. Instead, you _have_ to create a template (or template snippet - not really any better) to add this extra resource in, even for what should be the most basic, default case (scale servers behind a load balancer). We can provide a standard resource/template for this, LoadBalancedServer, to make the common case trivial and only require the user to pass parameters, not a whole template. 2) It relies on a plugin being present for any type of thing you might want to notify. I don't understand this point. What do you mean by a plugin? I was assuming OS::Neutron::PoolMember (not LoadBalancerMember -- I went and looked up the actual name) would become a standard Heat resource, not a third-party thing (though third parties could provide their own through the usual heat extension mechanisms). (fwiw the rackspace load balancer API works identically, so it seems a pretty standard design). At summit and - to the best of my recollection - before, we talked about scaling a generic group of resources and passing notifications to a generic controller, with the types of both defined by the user. I was expecting you to propose something based on webhooks, which is why I was surprised not to see anything about it in the API. (I'm not prejudging that that is the way to go... I'm actually wondering if Marconi has a role to play here.) I think the main benefit of PoolMember is: 1) it matches with the Neutron LBaaS API perfectly, just like all the rest of our resources, which represent individual REST objects. 2) it's already understandable. I don't understand the idea behind notifications or how they would work to solve our problems. You can keep saying that the notifications idea will solve our problems, but I can't figure out how it would solve our problem unless someone actually explains it :) -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 21, 2013 at 12:31 PM, Zane Bitter zbit...@redhat.com wrote: On 21/11/13 18:44, Christopher Armstrong wrote: 2) It relies on a plugin being present for any type of thing you might want to notify. I don't understand this point. What do you mean by a plugin? I was assuming OS::Neutron::PoolMember (not LoadBalancerMember -- I went and looked up the actual name) would become a standard Heat resource, not a third-party thing (though third parties could provide their own through the usual heat extension mechanisms). I mean it requires a resource type plugin written in Python. So cloud operators could provide their own implementations, but ordinary users could not. Okay, but that sounds like a general problem to solve (custom third-party plugins supplied by the user instead of cloud operators, which is an idea I really love btw), and I don't see why it should be a point against the idea of simply using a Neutron::PoolMember in a scaling unit. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Tue, Nov 19, 2013 at 4:27 PM, Zane Bitter zbit...@redhat.com wrote:
On 19/11/13 19:14, Christopher Armstrong wrote:
[snip]
It'd be interesting to see some examples, I think. I'll provide some
examples of my proposals, with the following caveats:
Excellent idea, thanks :)
- I'm assuming a separation of launch configuration from scaling group,
as you proposed -- I don't really have a problem with this.
- I'm also writing these examples with the plural resources parameter,
which there has been some bikeshedding around - I believe the structure
can be the same whether we go with singular, plural, or even
whole-template-as-a-string.
# trivial example: scaling a single server
POST /launch_configs
{
name: my-launch-config,
resources: {
my-server: {
type: OS::Nova::Server,
properties: {
image: my-image,
flavor: my-flavor, # etc...
}
}
}
}
This case would be simpler with my proposal, assuming we allow a default:
POST /launch_configs
{
name: my-launch-config,
parameters: {
image: my-image,
flavor: my-flavor, # etc...
}
}
If we don't allow a default it might be something more like:
POST /launch_configs
{
name: my-launch-config,
parameters: {
image: my-image,
flavor: my-flavor, # etc...
},
provider_template_uri: http://heat.example.com/
tenant_id/resources_types/OS::Nova::Server/template
}
POST /groups
{
name: group-name,
launch_config: my-launch-config,
min_size: 0,
max_size: 0,
}
This would be the same.
(and then, the user would continue on to create a policy that scales the
group, etc)
# complex example: scaling a server with an attached volume
POST /launch_configs
{
name: my-launch-config,
resources: {
my-volume: {
type: OS::Cinder::Volume,
properties: {
# volume properties...
}
},
my-server: {
type: OS::Nova::Server,
properties: {
image: my-image,
flavor: my-flavor, # etc...
}
},
my-volume-attachment: {
type: OS::Cinder::VolumeAttachment,
properties: {
volume_id: {get_resource: my-volume},
instance_uuid: {get_resource: my-server},
mountpoint: /mnt/volume
}
}
}
}
This appears slightly more complex on the surface; I'll explain why in a
second.
POST /launch_configs
{
name: my-launch-config,
parameters: {
image: my-image,
flavor: my-flavor, # etc...
}
provider_template: {
hot_format_version: some random date,
parameters {
image_name: {
type: string
},
flavor: {
type: string
} # c. ...
},
resources {
my-volume: {
type: OS::Cinder::Volume,
properties: {
# volume properties...
}
},
my-server: {
type: OS::Nova::Server,
properties: {
image: {get_param: image_name},
flavor: {get_param: flavor}, # etc...
}
},
my-volume-attachment: {
type: OS::Cinder::VolumeAttachment,
properties: {
volume_id: {get_resource: my-volume},
instance_uuid: {get_resource: my-server},
mountpoint: /mnt/volume
}
}
},
outputs {
public_ip_address: {
Value: {get_attr: [my-server,
public_ip_address]} # c. ...
}
}
}
(BTW the template could just as easily be included in the group rather
than the launch config. If we put it here we can validate the parameters
though.)
There are a number of advantages to including the whole template, rather
than a resource snippet:
- Templates are versioned!
- Templates accept parameters
- Templates can provide outputs - we'll need these when we go to do
notifications (e.g. to load balancers).
The obvious downside is there's a lot of fiddly stuff to include in the
template (hooking up the parameters and outputs), but this is almost
entirely mitigated by the fact that the user can get a template, ready
built with the server hooked up, from the API by hitting
/resource_types/OS::Nova::Server/template and just edit in the Volume and
VolumeAttachment. (For a different example, they could of course begin with
a different
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Wed, Nov 20, 2013 at 2:07 PM, Zane Bitter zbit...@redhat.com wrote: On 20/11/13 16:07, Christopher Armstrong wrote: On Tue, Nov 19, 2013 at 4:27 PM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 19/11/13 19:14, Christopher Armstrong wrote: [snip] There are a number of advantages to including the whole template, rather than a resource snippet: - Templates are versioned! - Templates accept parameters - Templates can provide outputs - we'll need these when we go to do notifications (e.g. to load balancers). The obvious downside is there's a lot of fiddly stuff to include in the template (hooking up the parameters and outputs), but this is almost entirely mitigated by the fact that the user can get a template, ready built with the server hooked up, from the API by hitting /resource_types/OS::Nova::__Server/template and just edit in the Volume and VolumeAttachment. (For a different example, they could of course begin with a different resource type - the launch config accepts any keys for parameters.) To the extent that this encourages people to write templates where the outputs are actually supplied, it will help reduce the number of people complaining their load balancers aren't forwarding any traffic because they didn't surface the IP addresses. My immediate reaction is to counter-propose just specifying an entire template instead of parameters and template separately, but I think the As an API, I think that would be fine, though inconsistent between the default (no template provided) and non-default cases. When it comes to implementing Heat resources to represent those, however, it would make the templates much less composable. If you wanted to reference anything from the surrounding template (including parameters), you would have to define the template inline and resolve references there. Whereas if you can pass parameters, then you only need to include the template from a separate file, or to reference a URL. Yeah, that's a good point, but I could also imagine if you're *not* actually trying to dynamically parameterize the flavor and image in the above example, you wouldn't need to use parameters at all, so the example could get a bit shorter. (to diverge from the topic momentarily) I've been getting a little bit concerned about how we'll deal with templates-within-templates... It seems a *bit* unfortunate that users will be forced to use separate files for their scaled and outer templates, instead of having the option to specify them inline, but I can't think of a very satisfying way to solve that problem. Maybe an escape function that prevents heat from evaluating any of the function calls inside? crux will be this point you mentioned: - Templates can provide outputs - we'll need these when we go to do notifications (e.g. to load balancers). Can you explain this in a bit more depth? It seems like whatever it is may be the real deciding factor that means that your proposal can do something that a resources or a template parameter can't do. I What I'm proposing _is_ a template parameter... I don't see any difference. A resources parameter couldn't do this though, because the resources section obviously doesn't contain outputs. In any event, when we notify a Load Balancer, or _any_ type of thing that needs a notification, we need to pass it some data. At the moment, for load balancers, we pass the IDs of the servers (I originally thought we passed IP addresses directly, hence possibly misleading comments earlier). But our scaling unit is a template which may contain multiple servers, or no servers. And the thing that gets notified may not even be a load balancer. So there is no way to infer what the right data to send is, we will need the user to tell us. The outputs section of the template seems like a good mechanism to do it. Hmm, okay. I still don't think I understand entirely how you expect outputs to be used, especially in context of the AS API. Can you give an example of how they would actually be used? I guess I don't yet understand all the implications of notification -- is that a new idea for icehouse? For what it's worth, I'm coming around to the idea of specifying the whole template in the API (or as a URI), but I'd still like to make sure I have a really good idea of the benefits it grants to justify the extra verbosity. thought we had a workable solution with the LoadBalancerMember idea, which you would use in a way somewhat similar to CinderVolumeAttachment in the above example, to hook servers up to load balancers. I haven't seen this proposal at all. Do you have a link? How does it handle the problem of wanting to notify an arbitrary service (i.e. not necessarily a load balancer)? It's been described in the autoscaling wiki page for a while, and I thought the LBMember idea was discussed
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Mon, Nov 18, 2013 at 5:57 AM, Zane Bitter zbit...@redhat.com wrote: On 16/11/13 11:15, Angus Salkeld wrote: On 15/11/13 08:46 -0600, Christopher Armstrong wrote: On Fri, Nov 15, 2013 at 3:57 AM, Zane Bitter zbit...@redhat.com wrote: On 15/11/13 02:48, Christopher Armstrong wrote: On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld asalk...@redhat.com mailto:asalk...@redhat.com wrote: On 14/11/13 10:19 -0600, Christopher Armstrong wrote: http://docs.heatautoscale.__apiary.io/ http://docs.heatautoscale.apiary.io/ I've thrown together a rough sketch of the proposed API for autoscaling. It's written in API-Blueprint format (which is a simple subset of Markdown) and provides schemas for inputs and outputs using JSON-Schema. The source document is currently at https://github.com/radix/heat/__raw/as-api-spike/ autoscaling.__apibp https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp Things we still need to figure out: - how to scope projects/domains. put them in the URL? get them from the token? - how webhooks are done (though this shouldn't affect the API too much; they're basically just opaque) Please read and comment :) Hi Chistopher In the group create object you have 'resources'. Can you explain what you expect in there? I thought we talked at summit about have a unit of scaling as a nested stack. The thinking here was: - this makes the new config stuff easier to scale (config get applied  per scaling stack) - you can potentially place notification resources in the scaling  stack (think marconi message resource - on-create it sends a  message) - no need for a launchconfig - you can place a LoadbalancerMember resource in the scaling stack  that triggers the loadbalancer to add/remove it from the lb. I guess what I am saying is I'd expect an api to a nested stack. Well, what I'm thinking now is that instead of resources (a mapping of resources), just have resource, which can be the template definition for a single resource. This would then allow the user to specify a Stack resource if they want to provide multiple resources. How does that sound? My thought was this (digging into the implementation here a bit): - Basically, the autoscaling code works as it does now: creates a template containing OS::Nova::Server resources (changed from AWS::EC2::Instance), with the properties obtained from the LaunchConfig, and creates a stack in Heat. - LaunchConfig can now contain any properties you like (I'm not 100% sure about this one*). - The user optionally supplies a template. If the template is supplied, it is passed to Heat and set in the environment as the provider for the OS::Nova::Server resource. I don't like the idea of binding to OS::Nova::Server specifically for autoscaling. I'd rather have the ability to scale *any* resource, including nested stacks or custom resources. It seems like jumping through hoops to big +1 here, autoscaling should not even know what it is scaling, just some resource. solum might want to scale all sorts of non-server resources (and other users). I'm surprised by the negative reaction to what I suggested, which is a completely standard use of provider templates. Allowing a user-defined stack of resources to stand in for an unrelated resource type is the entire point of providers. Everyone says that it's a great feature, but if you try to use it for something they call it a hack. Strange. To clarify this position (which I already did in IRC), replacing one concrete resource with another that means something in a completely different domain is a hack -- say, replacing server with group of related resources. However, replacing OS::Nova::Server with something which still does something very much like creating a server is reasonable -- e.g., using a different API like one for creating containers or using a different cloud provider's API. So, allow me to make a slight modification to my proposal: - The autoscaling service manages a template containing OS::Heat::ScaledResource resources. This is an imaginary resource type that is not backed by a plugin in Heat. - If no template is supplied by the user, the environment declares another resource plugin as the provider for OS::Heat::ScaledResource (by default it would be OS::Nova::Server, but this should probably be configurable by the deployer... so if you had a region full of Docker containers and no Nova servers, you could set it to OS::Docker::Container or something). - If a provider template is supplied by the user, it would be specified as the provider in the environment file. This, I hope, demonstrates that autoscaling needs no knowledge whatsoever about what it is scaling to use
Re: [openstack-dev] [Solum] Some initial code copying for db/migration
On Mon, Nov 18, 2013 at 3:00 PM, Dan Smith d...@danplanet.com wrote: Sorry for the delay in responding to this... * Moved the _obj_classes registry magic out of ObjectMetaClass and into its own method for easier use. Since this is a subclass based implementation, having a separate method feels more appropriate for a factory/registry pattern. This is actually how I had it in my initial design because I like explicit registration. We went off on this MetaClass tangent, which buys us certain things, but which also makes certain things quite difficult. Pros for metaclass approach: - Avoids having to decorate things (meh) - Automatic to the point of not being able to create an object type without registering it even if you wanted to Cons for metaclass approach: - Maybe a bit too magical - Can make testing hard (see where we save/restore the registry between each test) - I think it might make subclass implementations harder - Definitely more complicated to understand Chris much preferred the metaclass approach, so I'm including him here. He had some reasoning that won out in the original discussion, although I don't really remember what that was. It's almost always possible to go without metaclasses without losing much relevant brevity, and improving clarity. I strongly recommend against their use. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Sun, Nov 17, 2013 at 2:57 PM, Steve Baker sba...@redhat.com wrote: On 11/15/2013 05:19 AM, Christopher Armstrong wrote: http://docs.heatautoscale.apiary.io/ I've thrown together a rough sketch of the proposed API for autoscaling. It's written in API-Blueprint format (which is a simple subset of Markdown) and provides schemas for inputs and outputs using JSON-Schema. The source document is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp Apologies if I'm about to re-litigate an old argument, but... At summit we discussed creating a new endpoint (and new pythonclient) for autoscaling. Instead I think the autoscaling API could just be added to the existing heat-api endpoint. Arguments for just making auto scaling part of heat api include: * Significantly less development, packaging and deployment configuration of not creating a heat-autoscaling-api and python-autoscalingclient * Autoscaling is orchestration (for some definition of orchestration) so belongs in the orchestration service endpoint * The autoscaling API includes heat template snippets, so a heat service is a required dependency for deployers anyway * End-users are still free to use the autoscaling portion of the heat API without necessarily being aware of (or directly using) heat templates and stacks * It seems acceptable for single endpoints to manage many resources (eg, the increasingly disparate list of resources available via the neutron API) Arguments for making a new auto scaling api include: * Autoscaling is not orchestration (for some narrower definition of orchestration) * Autoscaling implementation will be handled by something other than heat engine (I have assumed the opposite) (no doubt this list will be added to in this thread) What do you think? I would be fine with this. Putting the API at the same endpoint as Heat's API can be done whether we decide to document the API as a separate thing or not. Would you prefer to see it as literally just more features added to the Heat API, or an autoscaling API that just happens to live at the same endpoint? -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Fri, Nov 15, 2013 at 4:16 AM, Zane Bitter zbit...@redhat.com wrote: On 14/11/13 19:58, Christopher Armstrong wrote: On Thu, Nov 14, 2013 at 10:44 AM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 14/11/13 18:51, Randall Burt wrote: Perhaps, but I also miss important information as a legitimate caller as to whether or not my scaling action actually happened or I've been a little too aggressive with my curl commands. The fact that I get anything other than 404 (which the spec returns if its not a legit hook) means I've found *something* and can simply call it endlessly in a loop causing havoc. Perhaps the web hooks *should* be authenticated? This seems like a pretty large hole to me, especially if I can max someone's resources by guessing the right url. Web hooks MUST be authenticated. Do you mean they should have an X-Auth-Token passed? Or an X-Trust-ID? Maybe an X-Auth-Token, though in many cases I imagine it would be derived from a Trust. In any event, it should be something provided by Keystone because that is where authentication implementations belong in OpenStack. The idea was that webhooks are secret (and should generally only be passed around through automated systems, not with human interaction). This is usually how webhooks work, and it's actually how they work now in Heat -- even though there's a lot of posturing about signed requests and so forth, in the end they are literally just secret URLs that give you the capability to perform some operation (if you have the URL, you don't need anything else to execute them). I think we should simplify this to to just be a random revokable blob. This is the weakest possible form of security - the whole secret gets passed on the wire for every request and logged in innumerable places. There's no protection at all against replay attacks (other than, hopefully, SSL). A signature, a timestamp and a nonce all seem like prudent precautions to add. I can get behind the idea of adding timestamp and nonce + signature for the webhooks, as long as they're handled better than they are now :) i.e., the webhook handler should assert that the timestamp is recent and non-repeated. This probably means storing stuff in MySQL (or a centralized in-memory DB). My understanding is that even though we have signed URLs for webhooks in the current Heat autoscaling system, they're effectively just static blobs. My original proposal for simple webhooks was based entirely around the idea that the current stuff is too complex, and offers no additional security over a random string jammed into a URL. (signing a static random string doesn't make it more guessable than the original random string...) -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][mistral] EventScheduler vs Mistral scheduling
On Thu, Nov 14, 2013 at 4:29 AM, Renat Akhmerov rakhme...@mirantis.comwrote: As for EventScheduler proposal, I think it actually fits Mistral model very well. What described in EvenScheduler is basically the ability to configure webhooks to be called periodically or at a certain time. First of all, from the very beginning the concept of scheduling has been considered a very important capability of Mistral. And from Mistral perspective calling a webhook is just a workflow consisting of one task. In order to simplify consumption of the service we can implement API methods to work specifically with webhooks in a convenient way (without providing any workflow definitions using DSL etc.). I have already suggested before that we can provide API shortcuts for scheduling individual tasks rather than complex workflows so it has an adjacent meaning. I other words, I now tend to think it doesn’t make sense to have EventScheduler a standalone service. What do you think? I agree that I don't think it makes sense to have a whole new project just for EventScheduler. Mistral seems like a pretty good fit. Convenience APIs similar to the EventScheduler API for just saying run this webhook on this schedule would be nice, too, but I wouldn't raise a fuss if they didn't exist and I had to actually define a trivial workflow. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] rough draft of Heat autoscaling API
http://docs.heatautoscale.apiary.io/ I've thrown together a rough sketch of the proposed API for autoscaling. It's written in API-Blueprint format (which is a simple subset of Markdown) and provides schemas for inputs and outputs using JSON-Schema. The source document is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp Things we still need to figure out: - how to scope projects/domains. put them in the URL? get them from the token? - how webhooks are done (though this shouldn't affect the API too much; they're basically just opaque) Please read and comment :) -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt randall.b...@rackspace.comwrote: Good stuff! Some questions/comments: If web hooks are associated with policies and policies are independent entities, how does a web hook specify the scaling group to act on? Does calling the web hook activate the policy on every associated scaling group? Not sure what you mean by policies are independent entities. You may have missed that the policy resource lives hierarchically under the group resource. Policies are strictly associated with one scaling group, so when a policy is executed (via a webhook), it's acting on the scaling group that the policy is associated with. Regarding web hook execution and cool down, I think the response should be something like 307 if the hook is on cool down with an appropriate retry-after header. Indicating whether a webhook was found or whether it actually executed anything may be an information leak, since webhook URLs require no additional authentication other than knowledge of the URL itself. Responding with only 202 means that people won't be able to guess at random URLs and know when they've found one. On Nov 14, 2013, at 10:57 AM, Randall Burt randall.b...@rackspace.com wrote: On Nov 14, 2013, at 10:19 AM, Christopher Armstrong chris.armstr...@rackspace.com wrote: http://docs.heatautoscale.apiary.io/ I've thrown together a rough sketch of the proposed API for autoscaling. It's written in API-Blueprint format (which is a simple subset of Markdown) and provides schemas for inputs and outputs using JSON-Schema. The source document is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp Things we still need to figure out: - how to scope projects/domains. put them in the URL? get them from the token? This may be moot considering the latest from the keystone devs regarding token scoping to domains/projects. Basically, a token is scoped to a single domain/project from what I understood, so domain/project is implicit. I'm still of the mind that the tenant doesn't belong so early in the URI, since we can already surmise the actual tenant from the authentication context, but that's something for Openstack at large to agree on. - how webhooks are done (though this shouldn't affect the API too much; they're basically just opaque) Please read and comment :) -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
Thanks for the comments, Zane.
On Thu, Nov 14, 2013 at 9:56 AM, Zane Bitter zbit...@redhat.com wrote:
On 14/11/13 17:19, Christopher Armstrong wrote:
http://docs.heatautoscale.apiary.io/
I've thrown together a rough sketch of the proposed API for autoscaling.
It's written in API-Blueprint format (which is a simple subset of
Markdown) and provides schemas for inputs and outputs using JSON-Schema.
The source document is currently atÂ
https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp
Things we still need to figure out:
- how to scope projects/domains. put them in the URL? get them from the
token?
- how webhooks are done (though this shouldn't affect the API too much;
they're basically just opaque)
My 2c: the way I designed the Heat API was such that extant stacks can be
addressed uniquely by name. Humans are pretty good with names, not so much
with 128 bit numbers. The consequences of this for the design were:
- names must be unique per-tenant
- the tenant-id appears in the endpoint URL
However, the rest of OpenStack seems to have gone in a direction where the
name is really just a comment field, everything is addressed only by
UUID. A consequence of this is that it renders the tenant-id in the URL
pointless, so many projects are removing it.
Unfortunately, one result is that if you create a resource and e.g. miss
the Created response for any reason and thus do not have the UUID, there is
now no safe, general automated way to delete it again. (There are obviously
heuristics you could try.) To solve this problem, there is a proposal
floating about for clients to provide another unique ID when making the
request, which would render a retry of the request idempotent. That's
insufficient, though, because if you decide to roll back instead of retry
you still need a way to delete using only this ID.
So basically, that design sucks for both humans (who have to remember
UUIDs instead of names) and machines (Heat). However, it appears that I am
in a minority of one on this point, so take it with a grain of salt.
Please read and comment :)
A few comments...
#1 thing is that the launch configuration needs to be somehow represented.
In general we want the launch configuration to be a provider template, but
we'll want to create a shortcut for the obvious case of just scaling
servers. Maybe we pass a provider template (or URL) as well as parameters,
and the former is optional.
I'm a little unclear as to what point you're making here. Right now, the
launch configuration is specified in the scaling group by the resources
property of the request json body. It's not a full template, but just a
snippet of a set of resources you want scaled.
As an aside, maybe we should replace this with a singlular resource and
allow people to use a Stack resource if they want to represent multiple
resources.
I guess we can have a simpler API for using an old-style, server-specific
launch configuration, but I am skeptical of the benefit, since specifying
a single Instance resource is pretty simple.
Successful creates should return 201 Created, not 200 OK.
Okay, I'll update that. I think I also forgot to specify some success
responses for things that need them.
Responses from creates should include the UUID as well as the URI.
(Getting into minor details here.)
Okay.
Policies are scoped within groups, so do they need a unique id or would a
name do?
I guess we could get rid of the ID and only have a name, what do other
people think?
I'm not sure I understand the webhooks part... webhook-exec is the thing
that e.g. Ceilometer will use to signal an alarm, right? Why is it not
called something like /groups/{group_id}/policies/{policy_id}/alarm ?
(Maybe because it requires different auth middleware? Or does it?)
Mostly because it's unnecessary. The idea was to generate a secret, opaque,
revokable ID that maps to the specific policy.
And the other ones are setting up the notification actions? Can we call
them notifications instead of webhooks? (After all, in the future we will
probably want to add Marconi support, and maybe even Mistral support.) And
why are these attached to the policy? Isn't the notification connected to
changes in the group, rather than anything specific to the policy? Am I
misunderstanding how this works? What is the difference between 'uri' and
'capability_uri'?
Policies represent ways to change a group (add +5% to this group).
Webhooks execute policies.
A capability URI is a URI which represents a capability to do something
all by itself. capability_uri would be the webhook-exec thing. The regular
URI would be the thing under
/groups/{group_id}/policies/{policy_id}/webhooks. That URI needs to exist
so you can perform the DELETE operation on it. (but you can't DELETE the
capability_uri, only POST to it to execute the policy).
I'll think more about webhooks vs notifications.
You need to define PUT/PATCH
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 10:46 AM, Georgy Okrokvertskhov
gokrokvertsk...@mirantis.com wrote:
Hi,
It would be great if API specs contain a list of attributes\parameters one
can pass during group creation. I believe Zane already asked about
LaunchConfig, but I think new autoscaling API creation was specifically
designed to move from limited AWS ElasticLB to something with more broad
features. There is a BP I submitted while ago
https://blueprints.launchpad.net/heat/+spec/autoscaling-instancse-typization.
We discussed it in IRC chat with heat team and we got to the conclusion
that this will be supported in new autoscaling API. Probably it is already
supported, but it is quite hard to figure this out from the existing API
specs without examples.
The API spec does contain a list of attributes/parameters that you can pass
during group creation (and all the other operations) -- see the Schema
sections under each. In case you didn't notice, you can click on each
action to expand details under it.
Thanks
Georgy
On Thu, Nov 14, 2013 at 9:56 AM, Zane Bitter zbit...@redhat.com wrote:
On 14/11/13 17:19, Christopher Armstrong wrote:
http://docs.heatautoscale.apiary.io/
I've thrown together a rough sketch of the proposed API for autoscaling.
It's written in API-Blueprint format (which is a simple subset of
Markdown) and provides schemas for inputs and outputs using JSON-Schema.
The source document is currently atÂ
https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp
Things we still need to figure out:
- how to scope projects/domains. put them in the URL? get them from the
token?
- how webhooks are done (though this shouldn't affect the API too much;
they're basically just opaque)
My 2c: the way I designed the Heat API was such that extant stacks can be
addressed uniquely by name. Humans are pretty good with names, not so much
with 128 bit numbers. The consequences of this for the design were:
- names must be unique per-tenant
- the tenant-id appears in the endpoint URL
However, the rest of OpenStack seems to have gone in a direction where
the name is really just a comment field, everything is addressed only by
UUID. A consequence of this is that it renders the tenant-id in the URL
pointless, so many projects are removing it.
Unfortunately, one result is that if you create a resource and e.g. miss
the Created response for any reason and thus do not have the UUID, there is
now no safe, general automated way to delete it again. (There are obviously
heuristics you could try.) To solve this problem, there is a proposal
floating about for clients to provide another unique ID when making the
request, which would render a retry of the request idempotent. That's
insufficient, though, because if you decide to roll back instead of retry
you still need a way to delete using only this ID.
So basically, that design sucks for both humans (who have to remember
UUIDs instead of names) and machines (Heat). However, it appears that I am
in a minority of one on this point, so take it with a grain of salt.
Please read and comment :)
A few comments...
#1 thing is that the launch configuration needs to be somehow
represented. In general we want the launch configuration to be a provider
template, but we'll want to create a shortcut for the obvious case of just
scaling servers. Maybe we pass a provider template (or URL) as well as
parameters, and the former is optional.
Successful creates should return 201 Created, not 200 OK.
Responses from creates should include the UUID as well as the URI.
(Getting into minor details here.)
Policies are scoped within groups, so do they need a unique id or would a
name do?
I'm not sure I understand the webhooks part... webhook-exec is the thing
that e.g. Ceilometer will use to signal an alarm, right? Why is it not
called something like /groups/{group_id}/policies/{policy_id}/alarm ?
(Maybe because it requires different auth middleware? Or does it?)
And the other ones are setting up the notification actions? Can we call
them notifications instead of webhooks? (After all, in the future we will
probably want to add Marconi support, and maybe even Mistral support.) And
why are these attached to the policy? Isn't the notification connected to
changes in the group, rather than anything specific to the policy? Am I
misunderstanding how this works? What is the difference between 'uri' and
'capability_uri'?
You need to define PUT/PATCH methods for most of these also, obviously (I
assume you just want to get this part nailed down first).
cheers,
Zane.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Georgy Okrokvertskhov
Technical Program Manager,
Cloud and Infrastructure Services,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 10:44 AM, Zane Bitter zbit...@redhat.com wrote: On 14/11/13 18:51, Randall Burt wrote: On Nov 14, 2013, at 11:30 AM, Christopher Armstrong chris.armstr...@rackspace.com mailto:chris.armstr...@rackspace.com wrote: On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt randall.b...@rackspace.com mailto:randall.b...@rackspace.com wrote: Regarding web hook execution and cool down, I think the response should be something like 307 if the hook is on cool down with an appropriate retry-after header. I strongly disagree with this even ignoring the security issue mentioned below. Being in the cooldown period is NOT an error, and the caller should absolutely NOT try again later - the request has been received and correctly acted upon (by doing nothing). Yeah, I think it's fine to just let it always be 202. Also, they don't actually return 404 when they don't exist -- I had that in an earlier version of the spec but I thought I deleted it before posting it to this list. Indicating whether a webhook was found or whether it actually executed anything may be an information leak, since webhook URLs require no additional authentication other than knowledge of the URL itself. Responding with only 202 means that people won't be able to guess at random URLs and know when they've found one. Perhaps, but I also miss important information as a legitimate caller as to whether or not my scaling action actually happened or I've been a little too aggressive with my curl commands. The fact that I get anything other than 404 (which the spec returns if its not a legit hook) means I've found *something* and can simply call it endlessly in a loop causing havoc. Perhaps the web hooks *should* be authenticated? This seems like a pretty large hole to me, especially if I can max someone's resources by guessing the right url. Web hooks MUST be authenticated. Do you mean they should have an X-Auth-Token passed? Or an X-Trust-ID? The idea was that webhooks are secret (and should generally only be passed around through automated systems, not with human interaction). This is usually how webhooks work, and it's actually how they work now in Heat -- even though there's a lot of posturing about signed requests and so forth, in the end they are literally just secret URLs that give you the capability to perform some operation (if you have the URL, you don't need anything else to execute them). I think we should simplify this to to just be a random revokable blob. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum/Heat] Is Solum really necessary?
On Thu, Nov 14, 2013 at 11:04 AM, Sam Alba sam.a...@gmail.com wrote: Hi Jay, I think Heat is an ingredient for Solum. When you build a PaaS, you need to control the app at different levels: #1 describing your app (basically your stack) #2 Pushing your code #3 Deploying it #4 Controlling the runtime (restart, get logs, scale, changing resources allocation, etc...) I think Heat is a major component for step 3. But I think Heat's job ends at the end of the deployment (the status of the stack is COMPLETED in Heat after processing the template correctly). It's nice though to rely on Heat's template generation for describing the stack, it's one more thing to delegate to Heat. In other words, I see Heat as an engine for deployment (at least in the context of Solum) and have something on top to manage the other steps. I'd say that Heat does (or should do) more than just the initial deployment -- especially with recent discussion around healing / convergence. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 11:00 AM, Randall Burt randall.b...@rackspace.comwrote: On Nov 14, 2013, at 12:44 PM, Zane Bitter zbit...@redhat.com wrote: On 14/11/13 18:51, Randall Burt wrote: On Nov 14, 2013, at 11:30 AM, Christopher Armstrong chris.armstr...@rackspace.com mailto:chris.armstr...@rackspace.com wrote: On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt randall.b...@rackspace.com mailto:randall.b...@rackspace.com wrote: Regarding web hook execution and cool down, I think the response should be something like 307 if the hook is on cool down with an appropriate retry-after header. I strongly disagree with this even ignoring the security issue mentioned below. Being in the cooldown period is NOT an error, and the caller should absolutely NOT try again later - the request has been received and correctly acted upon (by doing nothing). But how do I know nothing was done? I may have very good reasons to re-scale outside of ceilometer or other mechanisms and absolutely SHOULD try again later. As it stands, I have no way of knowing that my scaling action didn't happen without examining my physical resources. 307 is a legitimate response in these cases, but I'm certainly open to other suggestions. I agree there should be a way to find out what happened, but in a way that requires a more strongly authenticated request. My preference would be to use an audit log system (I haven't been keeping up with the current thoughts on the design for Heat's event/log API) that can be inspected via API. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 12:52 PM, Randall Burt randall.b...@rackspace.comwrote: On Nov 14, 2013, at 1:05 PM, Christopher Armstrong chris.armstr...@rackspace.com wrote: On Thu, Nov 14, 2013 at 11:00 AM, Randall Burt randall.b...@rackspace.com wrote: On Nov 14, 2013, at 12:44 PM, Zane Bitter zbit...@redhat.com wrote: On 14/11/13 18:51, Randall Burt wrote: On Nov 14, 2013, at 11:30 AM, Christopher Armstrong chris.armstr...@rackspace.com mailto:chris.armstr...@rackspace.com wrote: On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt randall.b...@rackspace.com mailto:randall.b...@rackspace.com wrote: Regarding web hook execution and cool down, I think the response should be something like 307 if the hook is on cool down with an appropriate retry-after header. I strongly disagree with this even ignoring the security issue mentioned below. Being in the cooldown period is NOT an error, and the caller should absolutely NOT try again later - the request has been received and correctly acted upon (by doing nothing). But how do I know nothing was done? I may have very good reasons to re-scale outside of ceilometer or other mechanisms and absolutely SHOULD try again later. As it stands, I have no way of knowing that my scaling action didn't happen without examining my physical resources. 307 is a legitimate response in these cases, but I'm certainly open to other suggestions. I agree there should be a way to find out what happened, but in a way that requires a more strongly authenticated request. My preference would be to use an audit log system (I haven't been keeping up with the current thoughts on the design for Heat's event/log API) that can be inspected via API. Fair enough. I'm just thinking of folks who want to set this up but use external tools/monitoring solutions for the actual eventing. Having those tools grep through event logs seems a tad cumbersome, but I do understand the desire to make these un-authenticated secrets makes that terribly difficult. Calling it unauthenticated might be a bit misleading; it's authenticated by the knowledge of the URL (which implies a trust and policy to execute). -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API
On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld asalk...@redhat.com wrote: On 14/11/13 10:19 -0600, Christopher Armstrong wrote: http://docs.heatautoscale.apiary.io/ I've thrown together a rough sketch of the proposed API for autoscaling. It's written in API-Blueprint format (which is a simple subset of Markdown) and provides schemas for inputs and outputs using JSON-Schema. The source document is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp Things we still need to figure out: - how to scope projects/domains. put them in the URL? get them from the token? - how webhooks are done (though this shouldn't affect the API too much; they're basically just opaque) Please read and comment :) Hi Chistopher In the group create object you have 'resources'. Can you explain what you expect in there? I thought we talked at summit about have a unit of scaling as a nested stack. The thinking here was: - this makes the new config stuff easier to scale (config get applied per scaling stack) - you can potentially place notification resources in the scaling stack (think marconi message resource - on-create it sends a message) - no need for a launchconfig - you can place a LoadbalancerMember resource in the scaling stack that triggers the loadbalancer to add/remove it from the lb. I guess what I am saying is I'd expect an api to a nested stack. Well, what I'm thinking now is that instead of resources (a mapping of resources), just have resource, which can be the template definition for a single resource. This would then allow the user to specify a Stack resource if they want to provide multiple resources. How does that sound? -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Do we need to clean up resource_id after deletion?
On Tue, Nov 12, 2013 at 7:47 AM, Zane Bitter zbit...@redhat.com wrote: On 02/11/13 05:30, Clint Byrum wrote: Excerpts from Christopher Armstrong's message of 2013-11-01 11:34:56 -0700: Vijendar and I are trying to figure out if we need to set the resource_id of a resource to None when it's being deleted. This is done in a few resources, but not everywhere. To me it seems either a) redundant, since the resource is going to be deleted anyway (thus deleting the row in the DB that has the resource_id column) b) actively harmful to useful debuggability, since if the resource is soft-deleted, you'll not be able to find out what physical resource it represented before it's cleaned up. Is there some specific reason we should be calling resource_id_set(None) in a check_delete_complete method? I've often wondered why some do it, and some don't. Seems to me that it should be done not inside each resource plugin but in the generic resource handling code. However, I have not given this much thought. Perhaps others can provide insight into why it has been done that way. There was a time in the very early days of Heat development when deleting something that had already disappeared usually resulted in an error (i.e. we mostly weren't catching NotFound exceptions). I expect this habit dates from that era. I can't think of any reason we still need this, and I agree that it seems unhelpful for debugging. cheers, Zane. Thanks Zane and others who have responded. My recent patch (now already merged) won't delete the resource_id. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Policy on spelling and grammar
On Mon, Nov 11, 2013 at 12:19 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from David Kranz's message of 2013-11-11 09:58:59 -0800: I have seen a wide variety of -1'ing (and in many cases approving) patches for minor spelling or grammatical errors and think we need a policy about this. Given the large number of contributors for whom English is not their native language, I would be in favor of rejecting spelling errors in variable or method names but being more lenient in comments, commit messages, READMEs, etc. What do you all think? The point of code review is to find defects. Misspelled words are defects in the English language encoded in the change. In fact, commit messages in particular are critical to get right as they cannot ever be fixed, and they are generally the most useful when under a stressful situation trying to determine the nature of a regression. Many of our contributors are also newbies to python, and we do not let them get away with obvious mistakes in python code. English is just a language with a different interpreter (a more forgiving one, for sure, but also one with many versions at various stages of implementation). In fact, our large percentage of non-native english speakers is a reason to be extremely careful about grammar and spelling so as not to confuse non-native speakers with incorrect grammar and spelling. I believe that if a -1 for a spelling mistake is causing more than an extremely short turn around time then either the submitter is not engaged with the project and thus not responsive to the -1, or the reviewers are over-taxed and the given project needs more reviewers. It would be so much nicer if there were some easy way for the reviewer himself to fix the typos directly (in a way that can trivially be accepted by the submitter of the patch into his own patch -- with a click of a button). -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Bad review patterns
On Wed, Nov 6, 2013 at 2:34 AM, Radomir Dopieralski openst...@sheep.art.plwrote: Hello, I'm quite new in the OpenStack project, but I love it already. What is especially nifty is the automated review system -- I'm really impressed. I'm coming from a project in which we also did reviews of every change -- although it was mostly manual, and just one review was enough to merge -- and at some point in that project I noticed that it is very easy to give reviews that are unhelpful, frustrating and just get in the way of the actual work. I started paying attention to how I am reviewing code, and I managed to come up with several patterns that are bad. Once I know the patterns, it's easier to recognize when I'm doing something wrong and rethink the review. I would like to share the patterns that I noticed. Agreed on all points. I think Gerrit is nice in that it automates a lot of stuff, but unfortunately the workflow has not encouraged the best behavior for reviewers. This is a good list to follow -- but how can we be sure people will? This mailing list thread will only be seen by a small number of reviewers over the life of the project, I'm sure. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Create a source repo for the API specification?
On Sat, Nov 2, 2013 at 4:39 PM, Jay Pipes jaypi...@gmail.com wrote: Hi all, One of the most important aspects in the early stages of Solum development will be the consensus building and stabilization of the Solum API specification. A solid API spec aid in the speeding up the pace of innovation in the Solum contributor community. One of the aspects of the Keystone development process that I think is a big benefit is the separate source repository that stores the OpenStack Identity API specifications: https://github.com/openstack/identity-api When new versions of the API specification are debated or new extensions are proposed, patches are made to the specification markdown documents and reviewed in the exact same manner that regular code is on the https://review.openstack.org Gerrit site. Contributors are free to annotate the proposed changes to the API specification in the same way that they would make inline code comments on a regular code review. Here's an example for a proposed change that I recently made: https://review.openstack.org/#/c/54215/10 I'd like to propose that Solum do the same: have a separate source repository for the API specification. Thoughts? -jay I like this idea. I'd also propose that the format of the specification be something machine-readable, such as API-Blueprint (a simple subset of markdown, apiblueprint.org, also what apiary uses, if you've ever seen that) or RAML (a more structured YAML-based syntax, raml.org). API-Blueprint is closer to what the keystone document uses. Making the documentation machine-readable means that it's much easier to verify that, in practice, the implementation of an API matches its specification and documentation, which is a problem that plagues many OpenStack projects right now. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] Do we need to clean up resource_id after deletion?
Vijendar and I are trying to figure out if we need to set the resource_id of a resource to None when it's being deleted. This is done in a few resources, but not everywhere. To me it seems either a) redundant, since the resource is going to be deleted anyway (thus deleting the row in the DB that has the resource_id column) b) actively harmful to useful debuggability, since if the resource is soft-deleted, you'll not be able to find out what physical resource it represented before it's cleaned up. Is there some specific reason we should be calling resource_id_set(None) in a check_delete_complete method? -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Curvature and Donabe repos are now public!
On Thu, Oct 3, 2013 at 1:43 PM, Debojyoti Dutta ddu...@gmail.com wrote: Hi! We @Cisco just made the following repos public https://github.com/CiscoSystems/donabe https://github.com/CiscoSystems/curvature Donabe was pitched as a recursive container before Heat days. Curvature is an alternative interactive GUI front end to openstack that can handle virtual resources, templates and can instantiate Donabe workloads. The D3 + JS stuff was incorporated into Horizon. A short demo was shown last summit and can be found at http://www.openstack.org/summit/portland-2013/session-videos/presentation/interactive-visual-orchestration-with-curvature-and-donabe Congrats to the primary developers: @CaffeinatedBrad @John_R_Davidge @Tehsmash_ @JackPeterFletch ... Special thanks to @lewtucker for supporting this. Hope this leads to more cool stuff for the Openstack community! Congrats! I'm glad you guys finally released the code :) Does Cisco (or anyone else) plan to continue to put development resources into these projects, or should we basically view them as reference code for solving particular problems? Thanks, -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] question about stack updates, instance groups and wait conditions
Hello Simon! I've put responses below. On Tue, Sep 17, 2013 at 7:57 AM, Simon Pasquier simon.pasqu...@bull.net wrote: Hello, I'm testing stack updates with instance group and wait conditions and I'd like to get feedback from the Heat community. My template declares an instance group resource with size = N and a wait condition resource with count = N (N being passed as a parameter of the template). Each group's instance is calling cfn-signal (with a different id!) at the end of the user data script and my stack creates with no error. Now when I update my stack to run N+X instances, the instance group gets updated with size=N+X but since the wait condition is deleted and recreated, the count value should either be updated to X or my existing instances should re-execute cfn-signal. This is a pretty interesting scenario; I don't think we have a very good solution for it yet. To cope with this situation, I've found 2 options: 1/ declare 2 parameters in my template: nb of instances (N for creation, N+X for update) and count of wait conditions (N for creation, X for update). See [1] for the details. 2/ declare only one parameter in my template (the size of the group) and leverage cfn-hup on the existing instances to re-execute cfn-signal. See [2] for the details. The solution 1 is not really user-friendly and I found that solution 2 is a bit complicated. Does anybody know a simpler way to achieve the same result? I definitely think #1 is better than #2, but you're right, it's also not very nice. I'm kind of confused about your examples though, because you don't show anything that depends on ComputeReady in your template. I guess I can imagine some scenarios, but it's not very clear to me how this works. It'd be nice to make sure the new autoscaling solution that we're working on will support your case in a nice way, but I think we need some more information about what you're doing. The only time this would have an effect is if there's another resource depending on the ComputeReady *that's also being updated at the same time*, because the only effect that a dependency has is to wait until it is met before performing create, update, or delete operations on other resources. So I think it would be nice to understand your use case a little bit more before continuing discussion. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse
Hi Mike, I have a *slightly* better idea of the kind of stuff you're talking about, but I think it would really help if you could include some concrete real-world use cases and describe why a holistic scheduler inside of Heat is necessary for solving them. On Fri, Sep 20, 2013 at 2:13 AM, Mike Spreitzer mspre...@us.ibm.com wrote: I have written a new outline of my thoughts, you can find it at https://docs.google.com/document/d/1RV_kN2Io4dotxZREGEks9DM0Ih_trFZ-PipVDdzxq_E It is intended to stand up better to independent study. However, it is still just an outline. I am still learning about stuff going on in OpenStack, and am learning and thinking faster than I can write. Trying to figure out how to cope. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat
Hi Michael! Thanks for this summary. There were some minor
inaccuracies, but I appreciate you at least trying when I should have
summarized it earlier. I'll give some feedback inline.
First, though, I have recently worked a lot on the wiki page for the
blueprint. It's available here:
https://wiki.openstack.org/wiki/Heat/AutoScaling
It still might need a little bit more cleaning up and probably a more
holistic example, but it should be pretty close now. I will say that I
changed it to specify the Heat resources for using autoscale instead
of the APIs of the AS API mostly for convenience because they're
easily specifiable. The AS API should be derived pretty obviously from
the resources.
On Thu, Sep 19, 2013 at 6:35 AM, Mike Spreitzer mspre...@us.ibm.com wrote:
I'd like to try to summarize this discussion, if nothing else than to see
whether I have correctly understood it. There is a lot of consensus, but I
haven't heard from Adrian Otto since he wrote some objections. I'll focus
on trying to describe the consensus; Adrian's concerns are already collected
in a single message. Or maybe this is already written in some one place?
Yeah. Sorry I didn't link that wiki page earlier; it was in a pretty
raw and chaotic form.
The consensus is that there should be an autoscaling (AS) service that is
accessible via its own API. This autoscaling service can scale anything
describable by a snippet of Heat template (it's not clear to me exactly what
sort of syntax this is; is it written up anywhere?).
Yes. See the wiki page above; it's basically just a mapping exactly
like the Resources section in a typical Heat template. e.g.
{..., Resources: {mywebserver: {Type: OS::Nova::Server}, ...}}
The autoscaling
service is stimulated into action by a webhook call. The user has the
freedom to arrange calls on that webhook in any way she wants. It is
anticipated that a common case will be alarms raised by Ceilometer. For
more specialized or complicated logic, the user is free to wire up anything
she wants to call the webhook.
This is accurate.
An instance of the autoscaling service maintains an integer variable, which
is the current number of copies of the thing being autoscaled. Does the
webhook call provide a new number, or +1/-1 signal, or ...?
The webhook provides no parameters. The amount of change is encoded
into the policy that the webhook is associated with. Policies can
change it the same way they can in current AWS-based autoscaling: +/-
fixed number, or +/- percent, or setting it to a specific number
directly.
There was some discussion of a way to indicate which individuals to remove,
in the case of decreasing the multiplier. I suppose that would be an option
in the webhook, and one that will not be exercised by Ceilometer alarms.
I don't think the webhook is the right place to do that. That should
probably be a specific thing in the AS API.
(It seems to me that there is not much auto in this autoscaling service
--- it is really a scaling service driven by an external controller. This
is not a criticism, I think this is a good factoring --- but maybe not the
best naming.)
I think the policies are what qualify it for the auto term. You can
have webhook policies or schedule-based policies (and maybe more
policies in the future). The policies determine how to change the
group.
The autoscaling service does its job by multiplying the heat template
snippet (the thing to be autoscaled) by the current number of copies and
passing this derived template to Heat to make it so. As the desired
number of copies changes, the AS service changes the derived template that
it hands to Heat. Most commentators argue that the consistency and
non-redundancy of making the AS service use Heat outweigh the extra
path-length compared to a more direct solution.
Agreed.
Heat will have a resource type, analogous to
AWS::AutoScaling::AutoScalingGroup, through which the template author can
request usage of the AS service.
Yes.
OpenStack in general, and Heat in particular, need to be much better at
traceability and debuggability; the AS service should be good at these too.
Agreed.
Have I got this right?
Pretty much! Thanks for the summary :-)
--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat
I apologize that this mail will appear at the incorrect position in the thread, but I somehow got unsubscribed from openstack-dev due to bounces and didn't receive the original email. On 9/11/13 03:15 UTC, Adrian Otto adrian.o...@rackspace.com wrote: So, I don't intend to argue the technical minutia of each design point, but I challenge you to make sure that we (1) arrive at a simple system that any OpenStack user can comprehend, I think there is tension between simplicity of the stack and simplicity of the components in that stack. We're making sure that the components will be simple, self-contained, and easy to understand, and the stack will need to plug them together in an interesting way. (2) responds quickly to alarm stimulus, Like Zane, I don't really buy the argument that the API calls to Heat will make any significant impact on the speed of autoscaling. There are MUCH bigger wins in e.g. improving the ability for people to use cached, pre-configured images vs a couple of API calls. Even once you've optimized all of that, booting an instance still takes much, much longer than running the control code. (3) is unlikely to fail, I know this isn't exactly what you mentioned, but I have some things to say not about resilience but instead about reaction to failures. The traceability and debuggability of errors is something that unfortunately plagues all of OpenStack, both for developers and end-users. It is fortunate that OpenStack compononts make good use of each other, but unfortunate that 1. there's no central, filterable logging facility (without investing significant ops effort to deploy a third-party one yourself); 2. not enough consistent tagging of requests throughout the system that allows operators looking at logs to understand how a user's original request led to some ultimate error; 3. no ubiquitous mechanisms for propagating errors between service APIs in a way that ultimately lead back to the consumer of the service; 4. many services don't even report detailed information with errors that happen internally. I believe we'll have to do what we can, especially in #3 and #4, to make sure that the users of autoscaling and Heat have good visibility into the system when errors occur. (4) can be easily customized with user-supplied logic that controls how the scaling happens, and under what conditions. I think this is a good argument for using Heat for the scaling resources instead of doing it separately. One of the biggest new features that the new AS design provides is the ability to scale *any* resource, not just AWS::EC2::Instance. This means you can write your own custom resource with custom logic and scale it trivially. Doing it in terms of resources instead of launch configurations provides a lot of flexibility, and a Resource implementation is a nice way to wrap up that custom logic. If we implemented this in the AS service without using Heat, we'd either be constrained to nova instances again, or have to come up with our own API for customization. As far as customizing the conditions under which scaling happens, that's provided at the lowest common denominator by providing a webhook trigger for scaling policies (on top of which will be implemented convenient Ceilometer integration support). Users will be able to provide their own logic and hit the webhook whenever they want to execute the policy. It would be better if we could explain Autoscale like this: Heat - Autoscale - Nova, etc. -or- User - Autoscale - Nova, etc. This approach allows use cases where (for whatever reason) the end user does not want to use Heat at all, but still wants something simple to be auto-scaled for them. Nobody would be scratching their heads wondering why things are going in circles. The Heat behind Autoscale isn't something that the *consumer* of the service knows about, only the administrator. Granted, the API design that we're running with *does* currently require the user to provide snippets of heat resource templates -- just specifying the individual resources that should be scaled -- but I think it would be trivial to support an alternative type of launch configuration that does the translation to heat templates in the background, if we really want to hide all the Heatiness from a user who just wants the simplicity of knowing only about Nova and autoscaling. To conclude, I'd like to just say I basically agree with what Clint, Keith, and Steven have said in other messages in this thread. I doesn't appear that the design of Heat autoscaling (informed by Zane, Clint, Angus and others) fails to meet the criteria you've brought up. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat
On Fri, Aug 16, 2013 at 1:35 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Zane Bitter's message of 2013-08-16 09:36:23 -0700: On 16/08/13 00:50, Christopher Armstrong wrote: *Introduction and Requirements* So there's kind of a perfect storm happening around autoscaling in Heat right now. It's making it really hard to figure out how I should compose this email. There are a lot of different requirements, a lot of different cool ideas, and a lot of projects that want to take advantage of autoscaling in one way or another: Trove, OpenShift, TripleO, just to name a few... I'll try to list the requirements from various people/projects that may be relevant to autoscaling or scaling in general. 1. Some users want a service like Amazon's Auto Scaling or Rackspace's Otter -- a simple API that doesn't really involve orchestration. 2. If such a API exists, it makes sense for Heat to take advantage of its functionality instead of reimplementing it. +1, obviously. But the other half of the story is that the API is likely be implemented using Heat on the back end, amongst other reasons because that implementation already exists. (As you know, since you wrote it ;) So, just as we will have an RDS resource in Heat that calls Trove, and Trove will use Heat for orchestration: user = [Heat =] Trove = Heat = Nova there will be a similar workflow for Autoscaling: user = [Heat =] Autoscaling - Heat = Nova After a lot of consideration and an interesting IRC discussion, I think the point above makes it clear for me. Autoscaling will have a simpler implementation by making use of Heat's orchestration capabilities, but the fact that Heat will also use autoscaling is orthogonal to that. That does beg the question of why this belongs in Heat. Originally we had taken the stance that there must be only one control system, lest they have a policy-based battle royale. If we only ever let autoscaled resources be controlled via Heat (via nested stack produced by autoscaling), then there can be only one.. control service (Heat). By enforcing that autoscaling always talks to the world via Heat though, I think that reaffirms for me that autoscaling, while not really the same project (seems like it could happily live in its own code tree), will be best served by staying inside the OpenStack Orchestration program. The question of private RPC or driving it via the API is not all that interesting to me. I do prefer the SOA method and having things talk via their respective public APIs as it keeps things loosely coupled and thus easier to fit into one's brain and debug/change. I agree with using only public APIs. I have managed to fit this model of autoscaling managing a completely independent Heat stack into my brain, and I am willing to take it and run with it. Thanks to Zane and Clint for hashing this out with me in a 2-hour IRC design discussion, it was incredibly helpful :-) -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat
On Thu, Aug 15, 2013 at 6:39 PM, Randall Burt randall.b...@rackspace.comwrote: On Aug 15, 2013, at 6:20 PM, Angus Salkeld asalk...@redhat.com wrote: On 15/08/13 17:50 -0500, Christopher Armstrong wrote: 2. There should be a new custom-built API for doing exactly what the autoscaling service needs on an InstanceGroup, named something unashamedly specific -- like instance-group-adjust. Pros: It'll do exactly what it needs to do for this use case; very little state management in autoscale API; it lets Heat do all the orchestration and only give very specific delegation to the external autoscale API. Cons: The API grows an additional method for a specific use case. I like this one above: adjust(new_size, victim_list=['i1','i7']) So if you are reducing the new_size we look in the victim_list to choose those first. This should cover Clint's use case as well. -Angus We could just support victim_list=[1, 7], since these groups are collections of identical resources. Simple indexing should be sufficient, I would think. Perhaps separating the stimulus from the actions to take would let us design/build toward different policy implementations. Initially, we could have a HeatScalingPolicy that works with the signals that a scaling group can handle. When/if AS becomes an API outside of Heat, we can implement a fairly simple NovaScalingPolicy that includes the args to pass to nova boot. I don't agree with using indices. I'd rather use the actual resource IDs. For one, indices can change out from under you. Also, figuring out the index of the instance you want to kill is probably an additional step most of the time you actually care about destroying specific instances. 3. the autoscaling API should update the Size Property of the InstanceGroup resource in the stack that it is placed in. This would require the ability to PATCH a specific piece of a template (an operation isomorphic to update-stack). I think a PATCH semantic for updates would be generally useful in terms of quality of life for API users. Not having to pass the complete state and param values for trivial updates would be quite nice regardless of its implications to AS. Agreed. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] How the autoscale API should control scaling in Heat
*Introduction and Requirements* So there's kind of a perfect storm happening around autoscaling in Heat right now. It's making it really hard to figure out how I should compose this email. There are a lot of different requirements, a lot of different cool ideas, and a lot of projects that want to take advantage of autoscaling in one way or another: Trove, OpenShift, TripleO, just to name a few... I'll try to list the requirements from various people/projects that may be relevant to autoscaling or scaling in general. 1. Some users want a service like Amazon's Auto Scaling or Rackspace's Otter -- a simple API that doesn't really involve orchestration. 2. If such a API exists, it makes sense for Heat to take advantage of its functionality instead of reimplementing it. 3. If Heat integrates with that separate API, however, that API will need two ways to do its work: 1. native instance-launching functionality, for the simple use 2. a way to talk back to Heat to perform orchestration-aware scaling operations. 4. There may be things that are different than AWS::EC2::Instance that we would want to scale (I have personally been playing around with the concept of a ResourceGroup, which would maintain a nested stack of resources based on an arbitrary template snippet). 5. Some people would like to be able to perform manual operations on an instance group -- such as Clint Byrum's recent example of remove instance 4 from resource group A. Please chime in with your additional requirements if you have any! Trove and TripleO people, I'm looking at you :-) *TL;DR* Point 3.2. above is the main point of this email: exactly how should the autoscaling API talk back to Heat to tell it to add more instances? I included the other points so that we keep them in mind while considering a solution. *Possible Solutions* I have heard at least three possibilities so far: 1. the autoscaling API should maintain a full template of all the nodes in the autoscaled nested stack, manipulate it locally when it wants to add or remove instances, and post an update-stack to the nested-stack associated with the InstanceGroup. Pros: It doesn't require any changes to Heat. Cons: It puts a lot of burden of state management on the autoscale API, and it arguably spreads out the responsibility of orchestration to the autoscale API. Also arguable is that automated agents outside of Heat shouldn't be managing an internal template, which are typically developed by devops people and kept in version control. 2. There should be a new custom-built API for doing exactly what the autoscaling service needs on an InstanceGroup, named something unashamedly specific -- like instance-group-adjust. Pros: It'll do exactly what it needs to do for this use case; very little state management in autoscale API; it lets Heat do all the orchestration and only give very specific delegation to the external autoscale API. Cons: The API grows an additional method for a specific use case. 3. the autoscaling API should update the Size Property of the InstanceGroup resource in the stack that it is placed in. This would require the ability to PATCH a specific piece of a template (an operation isomorphic to update-stack). Pros: The API modification is generic, simply a more optimized version of update-stack; very little state management required in autoscale API. Cons: This would essentially require manipulating the user-provided template. (unless we have a concept of private properties, which perhaps wouldn't appear in the template as provided by the user, but can be manipulated with such an update stack operation?) *Addenda* Keep in mind that there are use cases which require other types of manipulation of the InstanceGroup -- not just the autoscaling API. For example, see Clint's #5 above. Also, about implementation: Andrew Plunk and I have begun work on Heat resources for Rackspace's Otter, which I think will be a really good proof of concept for how this stuff should work in the Heat-native autoscale API. I am trying to gradually work the design into the native Heat autoscaling design, and we will need to solve the autoscale-controlling-InstanceGroup issue soon. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Tue, Jul 2, 2013 at 11:38 PM, Robert Collins robe...@robertcollins.net wrote: Radix points out I missed the naunce that you're targeting the users of python-novaclient, for instance, rather than python-novaclient's own tests. On 3 July 2013 16:29, Robert Collins robe...@robertcollins.net wrote: What I'd like is for each client library, in addition to the actual implementation, is that they ship a fake, in-memory, version of the API. The fake implementations should take the same arguments, have the same return values, raise the same exceptions, and otherwise be identical, besides the fact that they are entirely in memory and never make network requests. So, +1 on shipping a fake reference copy of the API. -1 on shipping it in the client. The server that defines the API should have two implementations - the production one, and a testing fake. The server tests should exercise *both* code paths [e.g. using testscenarios] to ensure there is no skew between them. Then the client tests can be fast and efficient but not subject to implementation skew between fake and prod implementations. Back on Launchpad I designed a similar thing, but with language neutrality as a goal : https://dev.launchpad.net/ArchitectureGuide/ServicesRequirements#Test_fake And in fact, I think that that design would work well here, because we have multiple language bindings - Python, Ruby, PHP, Java, Go etc, and all of them will benefit from a low(ms or less)-latency test fake. So taking the aspect I missed into account I'm much happier with the idea of shipping a fake in the client, but... AFAICT many of our client behaviours are only well defined in the presence of a server anyhow. So it seems to me that a fast server fake can be used in tests of python-novaclient, *and* in tests of code using python-novaclient (including for instance, heat itself), and we get to write it just once per server, rather than once per server per language binding. -Rob I want to make sure I understond you. Let's say I have a program named cool-cloud-tool, and it uses python-novaclient, python-keystoneclient, and three other clients for OpenStack services. You're suggesting that its test suite should start up instances of all those OpenStack services with in-memory or otherwise localized backends, and communicate with them using standard python-*client functionality? I can imagine that being a useful thing, if it's very easy to do, and won't increase my test execution time too much. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Tue, Jul 2, 2013 at 11:10 AM, Kurt Griffiths kurt.griffi...@rackspace.com wrote: The idea has merit; my main concern is that we would be duplicating significant chunks of code/logic between the fakes and the real services. How can we do this in a DRY way? I've done it a few different ways for libraries I've worked on. Usually, the fakes don't actually duplicate much code from the real implementation. But in the cases they do, I've had situations like this: class RealImplementation(object): def do_network_stuff(self, stuff): ... def low_level_operation(self): return self.do_network_stuff(GET /integer) def high_level_operation(self): return self.low_level_operation() + 5 I'd just create a subclass like this: class FakeImplementation(RealImplementation): def do_network_stuff(self, stuff): raise NotImplementedError(This should never be called!) def low_level_operation(self): return self.integer # or however you implement your fake This has two interesting properties: 1. I don't have to reimplement the high_level_operation 2. If I forget to implement a fake version of some method that invokes do_network_stuff, then it will blow up with a NotImplementedError so my test doesn't accidentally do real network stuff. This is just an example from some recent work I did on a simple RPC client with an HTTP API (unrelated to OpenStack client libraries), but that just so happens to be the case that Alex is discussing, so I think it can work well. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Does it make sense to have a resource-create API?
On Tue, Jun 18, 2013 at 11:01 PM, Adrian Otto adrian.o...@rackspace.comwrote: On Jun 18, 2013, at 6:09 PM, Angus Salkeld asalk...@redhat.com wrote: To me one of the most powerful and apealing things of Heat is the ability to reproducibly re-create a stack from a template. This new public API is going to make this difficult. Adding an API only makes it difficult if you decide to bypass templates and use the API. You can still be disciplined and keep your templates updated to achieve the reproducible goal. Yes, and API of this sort is a sharp instrument, but it can be useful if applied properly. It seems we could trivialize the task of keeping your template up to date by providing an API for fetching a template that reflects the current stack. Does that sound sensible given the current direction of the design of Heat? -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
