On 11/26/13 4:16 PM, "Steve Baker" <sba...@redhat.com> wrote:
>On 11/27/2013 10:24 AM, Tim Schnell wrote: >> So the originally question that I attempted to pose was, "Can we add a >> schema-less metadata section to the template that can be used for a >> variety of purposes?". It looks like the answer is no, we need to >>discuss >> the features that would go in the metadata section and add them to the >>HOT >> specification if they are viable. I don't necessarily agree with this >> answer but I accept it as viable and take responsibility for the >> long-winded process that it took to get to this point. >> >> I think some valid points have been made and I have re-focused my >>efforts >> into the following proposed solution. >> >> I am fine with getting rid of the concept of a schema-less metadata >> section. If we can arrive at a workable design for a few use cases then >>I >> think that we won't need to discuss any of the options that Zane >>mentioned >> for handling the metadata section, comments, separate file, or in the >> template body. >> >> Use Case #1 >> I see valid value in being able to group templates based on a type or >> keyword. This would allow any client, Horizon or a Template Catalog >> service, to better organize and handle display options for an end-user. >> >> I believe that Ladislav initially proposed a solution that will work >>here. >> So I will second a proposal that we add a new top-level field to the HOT >> specification called "keywords" that contains this template type. >> >> keywords: wordpress, mysql, etcŠ >+1, but lets make the most of yaml and give it structure: >keywords: [wordpress, mysql, lamp] >keywords: >- wordpress >- mysql >- lamp > >> >> Use Case #2 >> The template author should also be able to explicitly define a help >>string >> that is distinct and separate from the description of an individual >> parameter. An example where this use case originated was with Nova >> Keypairs. The description of a keypair parameter might be something >>like, >> "This is the name of a nova key pair that will be used to ssh to the >> compute instance." A help string for this same parameter would be, "To >> learn more about nova keypairs click on this help article." >> >> I propose adding an additional field to the parameter definition: >> >> Parameters: >> <parameter name>: >> description: This is the name of a nova key pair that >> will be used to >> ssh to the compute instance. >> help: To learn more about nova key pairs click on this >> <a >> href="/some/url/">help article</a>. >+1, this could be used today in the stack create screens of horizon. > >> Use Case #3 >> Grouping parameters would help the client make smarter decisions about >>how >> to display the parameters for input to the end-user. This is so that all >> parameters related to some database resource can be intelligently >>grouped >> together. In addition to grouping these parameters together, there >>should >> be a method to ensuring that the order within the group of parameters >>can >> be explicitly stated. This way, the client can return a group of >>database >> parameters and the template author can indicate that the database >>instance >> name should be first, then the username, then the password, instead of >> that group being returned in a random order. >> >> Parameters: >> db_name: >> group: db >> order: 0 >> db_username: >> group: db >> order: 1 >> db_password: >> group: db >> order: 2 >> web_node_name: >> group: web_node >> order: 0 >> keypair: >> group: web_node >> order: 1 >> >> These are the use cases that have been clearly defined. The original >> purpose of the metadata section was to "future-proof" (I say >>future-proof, >> you say pre-optimize ;) ) rapid iterations to potential client design. >>The >> intent of the strategy was so that we did not overburden the Heat Core >> team with requirements that may fluctuate or change as we attempt to >> improve the user experience of Heat for the community. In hindsight I >>can >> see how that intent was misconstrued and may have come off as >> condescending and I apologize for that. >We *need* a solution for grouping and ordering in horizon for generating >the parameter fields. All that is being done currently (I think) is >sorting by mandatory, then parameter name, which often pushes important >parameters to the bottom of the screen. I'd prefer explicit attributes >rather than inferring from a special parser that preserves the order of >parameters in the template. The parameter fields are built from the >results of a template-validate call. > >Also, a group should really have its own order and description to >display, so we may need a new top-level section, eg >parameter-groups: > db: > description: Database configuration options > order: 1 > web_node: > description: Web server configuration > order: 0 >parameters: > # as above > >A template-validate call would need to return all the group and ordering >information, but otherwise heat can ignore this extra data. I agree with all of your modifications, although bringing up the template-validate call reminded me that the implementation of this use case should also imply a new REST endpoint specifically for returning parameters. It seems like the current implementation in Horizon is a bit hack-y by calling template-validate instead of something like "get-parameters". >> I think that these proposed use cases will add real value to Heat and >> would require only changes to the HOT specification. If these are >> acceptable please let me know and I will drop the existing blueprint and >> patches in favor of something that implements this design. If there are >> issues to discuss here please suggest alternative solutions so that we >>can >> continue to move forward with these improvements. >I think there are 3 potential blueprints here which all have merit. I >struck use cases #2 and #3 when I was working on heat horizon, so these >would have immediate benefit to a real open-source heat UI. >> I have no doubt that this is just the beginning of plenty of >>requirements >> being driven from an attempt to improve the overall end user experience >>of >> Heat and I will continue to vet the use cases and proposed >>implementations >> with the community before suggesting a solution. I know that this >>proposal >> leaves out a few use cases that have been discussed like template >> author/version and that is because I believe that some of the arguments >> made are valid and should be looked into before proposing changes to >>Heat. >> Most of those other use cases probably can be served from some other >> solution like git or the future template catalog. >> >> Thanks, >> Tim >> >> >> >> >> >> On 11/26/13 12:31 PM, "Zane Bitter" <zbit...@redhat.com> wrote: >> >>> On 26/11/13 03:26, Keith Bray wrote: >>>> >>>> On 11/25/13 5:46 PM, "Clint Byrum" <cl...@fewbar.com> wrote: >>>> >>>>> Excerpts from Tim Schnell's message of 2013-11-25 14:51:39 -0800: >>>>>> Hi Steve, >>>>>> >>>>>> As one of the UI developers driving the requirements behind these >>>>>>new >>>>>> blueprints I wanted to take a moment to assure you and the rest of >>>>>>the >>>>>> Openstack community that the primary purpose of pushing these >>>>>> requirements >>>>>> out to the community is to help improve the User Experience for Heat >>>>>> for >>>>>> everyone. Every major UI feature that I have implemented for Heat >>>>>>has >>>>>> been >>>>>> included in Horizon, see the Heat Topology, and these requirements >>>>>> should >>>>>> improve the value of Heat, regardless of the UI. >>>>>> >>>>>> >>>>>> Stack/template metadata >>>>>> We have a fundamental need to have the ability to reference some >>>>>> additional metadata about a template that Heat does not care about. >>>>>> There >>>>>> are many possible use cases for this need but the primary point is >>>>>> that >>>>>> we >>>>>> need a place in the template where we can iterate on the schema of >>>>>>the >>>>>> metadata without going through a lengthy design review. As far as I >>>>>> know, >>>>>> we are the only team attempting to actually productize Heat at the >>>>>> moment >>>>>> and this means that we are encountering requirements and requests >>>>>>that >>>>>> do >>>>>> not affect Heat directly but simply require Heat to allow a little >>>>>> wiggle >>>>>> room to flesh out a great user experience. >>>>>> >>>>> Wiggle room is indeed provided. But reviewers need to understand your >>>>> motivations, which is usually what blueprints are used for. If you're >>>>> getting push back, it is likely because your blueprints to not make >>>>>the >>>>> use cases and long term vision obvious. >>>> Clint, can you be more specific on what is not clear about the use >>>>case? >>> The part where it's used for something. What I'm hearing in this thread >>> is "We need this for a template catalog, but we're not going to talk >>> about that until later. Trust us, though, we really need it." >>> >>> That's a problem, because none of this stuff makes sense except in the >>> context of the template catalog. What does the template catalog do? Who >>> writes the templates in the catalog? Why is the template catalog not >>> backed by git? Who can say which parts of the solution belong in the >>> template catalog and which in Heat when we don't even know what it's a >>> solution to? >>> >>>> What I am seeing is that the use case of meta data is not what is >>>>being >>> I'm contesting it: >>> >>> >>>https://review.openstack.org/#/c/56703/7/doc/source/template_guide/hot_s >>>pe >>> c.rst >>> >>> The problem here isn't that the process wasn't followed, it's that all >>> of these things - one-shot outputs, operator-specific structured >>> metadata sections in templates, APIs to report versions of randomly >>> selected daemons - are fundamentally bad ideas. And they're all bad in >>> exactly the same way: they demand that Heat implement something that it >>> can only do in a half-baked manner, because it's slightly easier (or >>> seems slightly easier) than implementing something in some external >>>tool >>> that can work. >>> >>> It's difficult to escape the conclusion that these features have been >>> decided upon behind closed doors with the community excluded, by people >>> who are not especially familiar with Heat, with a view to minimising >>>the >>> effort required to implement some other (proprietary?) thing and not >>> necessarily to coming up with the best implementation. If this is your >>> development process then you are Doing Open Source Wrong. At a very >>> minimum you need to work about 10x harder to explain all of these >>>design >>> decisions - in particular the reasons why the obvious alternatives that >>> don't appear to have been considered are insufficient - to the >>> community. It's much, much easier, though, to just do your design in >>>the >>> open with community involvement. >>> >>> Take this metadata section idea. (Please!) Let's step back and examine >>> the very last decision in the chain - assume for a moment that we >>>accept >>> that this metadata needs to be included in the template. There are >>>three >>> obvious ways to do that: >>> >>> 1) Include them as YAML comments >>> 2) Include them in the body of the template >>> 3) Include them as a separate YAML document in the same file >>> >>> How many of those were actually considered? My bet is only #2. It >>> doesn't really matter though, because there is not one word of design >>> documentation that would help me understand the reasons for rejecting >>>#1 >>> & #3 (neither of which require any changes to Heat or the template >>> format). How can I have confidence that #2 is the correct solution to >>> the problem? Especially if I don't agree (I don't)? >>> >>> The number of options not considered grows (and my confidence in the >>> solution shrinks) exponentially as you step all the way back through >>>the >>> chain of undocumented decisions to the actual thing that drove this >>>idea. >>> >>>> contested, but that the Blueprint of where meta data should go is >>>>being >>>> contested by only a few (but not all) of the core devs. The Blueprint >>>> for >>> I contested that first because without knowing what the feature is even >>> for, there's nothing else to contest. >>> >>>> in-template metadata was already approved for Icehouse, but now that >>>> work >>>> has been delivered on the implementation of that blueprint, the >>>> blueprint >>>> itself is being contested: >>>> >>>>https://blueprints.launchpad.net/heat/+spec/namespace-stack-metadata >>> It seems clear that we have to communicate better to new contributors >>> what the expectations are around getting features in. But I would be >>> really sad if we have to jump straight in to the sort of legalistic >>> processes that some of the larger OpenStack programs follow. This is >>> still a relatively small project; we should be able to handle these >>> questions by talking to each other. >>> >>>> I'd like to propose that the blueprint that has been accepted go forth >>>> with the code that exactly implements it, and if there are alternative >>>> proposals and appropriate reasons for the community to come to >>>>consensus >>>> on a different approach, that we then iterate and move the data >>>> (deprecate >>>> the older feature if necessary, e.g. If that decision comes after >>>> Icehouse, else of a different/better implementation comes before >>>> Icehouse, >>>> then no harm done). >>> -1 >>> >>>>>> There is precedence for an optional metadata section that can >>>>>>contain >>>>>> any >>> As Clint said, you're comparing APIs to templates. Those are very >>> different. >>> >>>>>> end-user data in other Openstack projects and it is necessary in >>>>>>order >>>>>> to >>>>>> iterate quickly and provide value to Heat. >>> Non sequitur. >>> >>>>> Nobody has said you can't have meta-data on stacks, which is what >>>>>other >>>>> projects use. >>>>> >>>>>> There are many use cases that can be discussed here, but I wanted to >>>>>> reiterate an initial discussion point that, by definition, >>>>>> "stack/template_metadata" does not have any hard requirements in >>>>>>terms >>>>>> of >>>>>> schema or what does or does not belong in it. >>>>>> >>>>>> One of the initial use cases is to allow template authors to >>>>>> categorize >>>>>> the template as a specific "type". >>>>>> >>>>>> template_metadata: >>>>>> short_description: Wordpress >>> First of all, "short_description" is a dreadful name for something that >>> is basically tags. >>> >>> Secondly, these are tags in the template catalog. So why are they >>> incorporated in the template, and not stored in the template catalog? >>> >>> Finally, if you really want this it can be trivially implemented using >>> comments. >>> >>>>> Interesting. Would you support adding a "category" keyword to python >>>>>so >>>>> we don't have to put it in setup.cfg and so that the egg format >>>>>doesn't >>>>> need that section? Pypi can just parse the python to categorize the >>>>> apps >>>>> when they're uploaded. We could also have a file on disk for qcow2 >>>>> images >>>>> that we upload to glance that will define the meta-data. >>>>> >>>>> To be more direct, I don't think the templates themselves are where >>>>> this >>>>> meta-data belongs. A template is self-aware by definition, it doesn't >>>>> need the global metadata section to tell it that it is WordPress. For >>>>> anything else that needs to be globally referenced there are >>>>> parameters. >>>>> Having less defined inside the template means that you get _more_ >>>>> wiggle >>>>> room for your template repository. >>>> Clint, you are correct that the Template does not need to know what it >>>> is. >>>> It's every other service (and users of those services) that a >>>>Template >>>> passes through or to that would care to know what it is. We are >>>> suggesting >>>> we put that meta data in the template file and expressly ignore it for >>>> purposes of parsing the template language in the Heat engine, so we >>>> agree >>>> it not a necessary part of the template. Sure, we could encode the >>>> metadata info in a separate catalog... but, take the template out of >>>> the >>>> catalog and now all that useful associated data is lost or would need >>>>to >>>> be recreated by someone or some service. That does not make the >>>> template >>>> portable, and that is a key aspect of what we are trying to achieve >>>>(all >>>> user-facing clients, like Horizon, or humans reading the file, can >>>>take >>>> advantage). We don't entirely know yet what is most useful in >>>> portability >>> What's most useful in portability is placing the minimum requirements >>> (especially undocumented and/or non-standard requirements) on the >>> template author, since you can't control the template author. >>> >>>> and what isn't, so meta data in-template provides the "wiggle room" >>>> innovation space to suss that out. We already know of some specific >>>>use >>>> cases of data that we feel are important, which Tim identified one >>>> specific example.. As specific metadata items become popular or prove >>>>to >>>> be useful to rely on by the larger community or service operators >>>> (public >>>> and private) of Heat, we as a community can drive that information >>>>back >>>> into the schema for the template or some portable format mechanism. >>>> >>>>> I 100% support having a template catalog. IMO it should be glance, >>>>> which is our catalog service in OpenStack. Who cares if nova or heat >>>>> are >>>>> consuming images or templates. It is just sharable blobs of data and >>>>> meta-data in a highly scalable service. It already has the concept of >>>>> global and tenant-scope. It just needs an image type of 'hot' and >>>>>then >>>>> heat can start consuming templates from glance. And the template >>>>> authors >>>>> should maintain some packaging meta-data in glance to communicate to >>>>> users that this is "Wordpress" and "Single-Node". If Glance's >>>>>meta-data >>>>> is too limiting, expand it! I'm sure image authors and consumers >>>>>would >>>>> appreciate that. >>>> This is definitely interesting... And takes the long view IMO. Let me >>>> explain: I don't anticipate Heat catalog'ing in Glance is something >>>> that >>>> has a high chance of getting implemented in the Icehouse timeframe (at >>>> least, not more so than in-template metadata), do you? From a SOA >>>> service >>>> deployer perspective, I'm sure you can appreciate that rolling out new >>>> functionality in Glance to support an Orchestration project use case >>>>is >>>> not simple, and requires strong business justification and >>>>coordination >>>> as >>>> an operator of a cloud.. One worth exploring for sure, but not the >>>>go-to >>>> default strategy. I view the metadata change as very minor with little >>>> to >>>> no disruption to any service, including Heat (Heat just ignores the >>> I view it as an open invitation to fragment the template format and the >>> ecosystem. >>> >>>> metadata, completely).. This fits very well in an iterative >>>>development >>>> model. New blueprints could be raised, as you suggested, to move >>>> metadata >>>> and catalog features into Glance. My concern is that if we go the >>>> Glance >>>> route now, we are encouraging a precedent that we aren't iterative >>>>(due >>>> to >>>> abandoning an already accepted blueprint in favor of a more complex >>>>and >>>> time intensive solution) and that we won't get this implemented within >>>> the >>>> current release cycle. >>>> >>>>>> This would let the client of the Heat API group the templates by >>>>>>type >>>>>> which would create a better user experience when selecting or >>>>>>managing >>>>>> templates. The the end-user could select "Wordpress" and drill down >>>>>> further to select templates with different options, "single node", >>>>>>"2 >>>>>> web >>>>>> nodes", etc... >>>>>> >>>>> That is all api stuff, not language stuff. >>>> If this were done solely at the API, it would have to be maintained >>>> 1-to-1 >>>> with a template (in which case there is an implicit and explicit >>>> association), and exported with the template in order to port the >>>> template. >>> I don't understand all this stuff about 'portability'. From your >>> previous descriptions of the template catalog, it sounded like it's for >>> templates hand-picked by the operator. Why would one operator want to >>> import the tags defined by another operator anyway? And Why wouldn't >>> that just be a git clone of the backing repository? And how does any of >>> this make the template more "portable"??? >>> >>>>>> Once a feature has consistently proven that it adds value to Heat or >>>>>> Horizon, then I would suggest that we can discuss the schema for >>>>>>that >>>>>> feature and codify it then. >>>>>> >>>>>> In order to keep the discussion simple, I am only responding to the >>>>>> need >>>>>> for stack/template metadata at the moment but I'm sure discussions >>>>>>on >>>>>> the >>>>>> management api and template catalog will follow. >>>>>> >>>>> Your example puts the template catalog in front of this feature, and >>>>>I >>>>> think that exposes this feature as misguided. >>>> I'd like template author and creation date portable so that it goes >>>> wherever the template goes in a portable format that consumers (any >>>> client) of the Heat service can understand. I'd also like this >>> Put it in a comment. >>> >>>> information to be available for a deployed stack in Heat, so it is >>>>need >>>> not be catalog specific. If author/creation-date information were >>>>only >>> If it needs to be available for a deployed stack (you haven't made an >>> argument for this yet) then add it into the template-proper. If Heat >>> needs to read it then some sort of wild-west schemaless metadata area >>>is >>> definitely not the place to put it. >>> >>> Alternately, maybe we need to look at storing templates in such a way >>> that the comments are preserved. I would definitely look favourably on >>>a >>> blueprint to that effect. >>> >>>> available in the catalog, then we would have to entirely wrap the Heat >>>> API >>>> in another service or risk drift in the data between Heat and the >>>> separate >>>> non-wrapping catalog service. Instead of raising a blueprint for every >>>> piece of data that may be useful to deploying this service (some of >>>> which >>> Please don't raise a blueprint for every patch. e.g. a feature and its >>> documentation >>> >>>(https://blueprints.launchpad.net/heat/+spec/update-stack-metadata-spec) >>> are not two blueprints. >>> >>>> certain folks may not care about [e.g. Many of the folks arguing >>>>against >>>> metadata all together], the in-template metadata is, IMO, a suitable >>>> approach to experiment and then we can drive back experiential >>>>use-cases >>>> into future improvements (e.g. in-language schema changes or catalog >>>> changes). At Rackspace we have experience running a nearly identical >>>> service, so we know this data, in this particular place, is a valid >>>>use >>>> case for consumers of an Orchestration service. >>>> Example: >>>> (1) Cloud customer of service operator deploys a template (may or may >>>> not >>>> have come from the catalog) >>> If it didn't come from the catalog provided by the operator, what are >>> the chances that it contains this information anyway? How is the >>>average >>> template author going to know about the schema for the metadata read by >>> your proprietary tool? >>> >>>> (2) Customer encounters problem with the stack and calls support >>>> (3) Support specialist [not the service operator/developer] is trying >>>>to >>>> figure out who to contact for additional help given failures seen with >>>> the >>>> stack. The dev team of the service is not the expert in application >>>>foo >>>> and did not develop the template. It's not a Heat issue, it's a >>>> template/application foo issue. >>> This sounds like a good argument for Heat preserving comments. >>> >>>> Having this detail come from Glance, as you pointed out, would be >>>>great. >>>> But, my hope is that many folks will develop and share templates, and >>>> the >>>> folks developing templates won't necessarily be running glance or have >>>> access to upload their own images into glance (that is a service >>>> operator >>>> choice). If we force this basic info to glance, we are limiting our >>>> portability (and therefore adoption story) of Heat. >>> What's really going to limit portability is having multiple overlapping >>> undocumented proprietary schemas for parts of the template. I actually >>> think this has the potential to damage the Heat ecosystem, and that's >>> one of the reasons I'm pushing back on it. >>> >>> regards, >>> Zane. >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev