Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

On 11/10/2017 03:20 AM, Giulio Fidente wrote:

On 11/26/2015 03:17 PM, Jiří Stránský wrote:

On 26.11.2015 14:12, Jiří Stránský wrote:


[...]


It seems TripleO is hitting similar composability and sanity limits with
the top-down approach, and the number of parameters which can only be
fed via parameter_defaults is increasing. (The disadvantage of
parameter_defaults is that, unlike hiera, we currently have no clear
namespacing rules, which means a higher chance of conflict. Perhaps the
unit tests suggested in another subthread would be a good start, maybe
we could even think about how to do proper namespacing.)


Does what i described seem somewhat accurate? Should we maybe buy into
the concept of "composable templates, externally fed
hierarchy-transcending parameters" for the long term?


I now realized i might have used too generic or Puppetish terms in the
explanation, perhaps drowning the gist of the message a bit :) What i'm
suggesting is: let's consider going with parameter_defaults wherever we
can, for the sake of composability, and figure out what is the best way
to prevent parameter name collisions.

+1 I like very much the idea of parameter_defaults + strictier
namespacing rules

Specifically regarding namespaces, puppet was great but ansible doesn't
seem to be as good (at least to me), in fact I think we have chances for
conflicts in both THT and the ansible playbooks



I will point out that we now have a pep8 check that all definitions of a 
parameter have the exact same type, description, and default.  That 
makes it highly unlikely that someone would duplicate a parameter name 
but want to use it for something different from the original.


There are a few exclusions remaining so it's possible those could still 
get duplicated, but they're pretty specific (it only ignores certain 
fields on the listed parameters) so it's not a high likelihood.


It would be good if somebody would clean up the remaining 
inconsistencies in tht so we can validate as many parameters as 
possible.  There are some I'm not sure we can change that this point 
because it would affect users if we did, but there are quite a few that 
I think we can still address: 
https://github.com/openstack/tripleo-heat-templates/blob/master/tools/yaml-validate.py#L62 
 Everything below the TODO still needs to be looked at.  In a lot of 
cases it's just a question of making the descriptions consistent across 
all of the definitions.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Bogdan Dobrelya]

On 11/10/17 10:20 AM, Giulio Fidente wrote:

On 11/26/2015 03:17 PM, Jiří Stránský wrote:

On 26.11.2015 14:12, Jiří Stránský wrote:


[...]


It seems TripleO is hitting similar composability and sanity limits with
the top-down approach, and the number of parameters which can only be
fed via parameter_defaults is increasing. (The disadvantage of
parameter_defaults is that, unlike hiera, we currently have no clear
namespacing rules, which means a higher chance of conflict. Perhaps the
unit tests suggested in another subthread would be a good start, maybe
we could even think about how to do proper namespacing.)


Does what i described seem somewhat accurate? Should we maybe buy into
the concept of "composable templates, externally fed
hierarchy-transcending parameters" for the long term?


I now realized i might have used too generic or Puppetish terms in the
explanation, perhaps drowning the gist of the message a bit :) What i'm
suggesting is: let's consider going with parameter_defaults wherever we
can, for the sake of composability, and figure out what is the best way
to prevent parameter name collisions.

+1 I like very much the idea of parameter_defaults + strictier
namespacing rules

Specifically regarding namespaces, puppet was great but ansible doesn't
seem to be as good (at least to me), in fact I think we have chances for
conflicts in both THT and the ansible playbooks



Tripleo docs should have this explained, like in the ansible docs [1]

[1] 
http://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Giulio Fidente]

On 11/26/2015 03:17 PM, Jiří Stránský wrote:
> On 26.11.2015 14:12, Jiří Stránský wrote:

[...]

>> It seems TripleO is hitting similar composability and sanity limits with
>> the top-down approach, and the number of parameters which can only be
>> fed via parameter_defaults is increasing. (The disadvantage of
>> parameter_defaults is that, unlike hiera, we currently have no clear
>> namespacing rules, which means a higher chance of conflict. Perhaps the
>> unit tests suggested in another subthread would be a good start, maybe
>> we could even think about how to do proper namespacing.)
>>
>>
>> Does what i described seem somewhat accurate? Should we maybe buy into
>> the concept of "composable templates, externally fed
>> hierarchy-transcending parameters" for the long term?
> 
> I now realized i might have used too generic or Puppetish terms in the
> explanation, perhaps drowning the gist of the message a bit :) What i'm
> suggesting is: let's consider going with parameter_defaults wherever we
> can, for the sake of composability, and figure out what is the best way
> to prevent parameter name collisions.
+1 I like very much the idea of parameter_defaults + strictier
namespacing rules

Specifically regarding namespaces, puppet was great but ansible doesn't
seem to be as good (at least to me), in fact I think we have chances for
conflicts in both THT and the ansible playbooks
-- 
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jiri Tomasek]

On 11/25/2015 03:17 PM, Jay Dobies wrote:

I think at the same time we add a mechanism to distinguish between
internal and external parameters, we need to add something to indicate
required v. optional.

With a nested stack, anything that's not part of the top-level 
parameter

contract is defaulted. The problem is that it loses information on what
is a valid default v. what's simply defaulted to pass validation.


I thought the nested validation spec was supposed to handle that though?
  To me, required vs. optional should be as simple as "Does the 
parameter

definition have a 'default' key?  If yes, then it's optional, if no,
then it's required for the user to pass a value via a parameter or
parameter_default".  I realize we may not have been following that up to
now for various reasons, but it seems like Heat is already providing a
pretty explicit mechanism for marking params as required, so we ought to
use it.


Ya, I was mistaken here. Taking a look at the cinder-netapp.yaml, it 
looks like we're using this correctly:


...
  CinderNetappBackendName:
type: string
default: 'tripleo_netapp'
  CinderNetappLogin:
type: string
  CinderNetappPassword:
type: string
hidden: true
...


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


I need to read the thread once again, but I'd like to add a few 
observations from the GUI implementation:


The nested validation as it works right now, requires that all root 
template parameters need to have 'default' or 'value' set, otherwise the 
heat validation fails and no parameters are returned. This is a sort of 
a blocker because we need to use this to retrieve the parameters and let 
user set the values for them. This means, that to be able to list the 
parameters, we need to make sure that all root template parameters have 
'default' set, which is not optimal.


Other observation (maybe a bit outside of the topic) is that the list of 
parameters defined in root template is huge, It would be nice if root 
template and more possibly root environment included resource registry 
only for the roles/templates that are explicitly required for the 
minimal deployment (controller, compute) and split other roles into 
separate  optional environments.
In current situation the user is required to set flavors, node counts 
etc. for all roles defined in root template even though he is not going 
to use them (sets the node_count to 0)



Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

On 11/20/2015 06:05 PM, Ben Nemec wrote:
> Thinking about this some more makes me wonder if we need a sample config
> generator like oslo.config.  It would work off something similar to the
> capabilities map, where you would say
> 
> SSL:
>   templates:
> -puppet/extraconfig/tls/tls-cert-inject.yaml
>   output:
> -environments/enable-ssl.yaml
> 
> And the tool would look at that, read all the params from
> tls-cert-inject.yaml and generate the sample env file.  We'd have to be
> able to do a few new things with the params in order for this to work:
> 
> -Need to specify whether a param is intended to be set as a top-level
> param, parameter_defaults (which we informally do today with the Can be
> overridden by parameter_defaults comment), or internal, to define params
> that shouldn't be exposed in the sample config and are only intended as
> an interface between templates.  There wouldn't be any enforcement of
> the internal type, but Python relies on convention for its private
> members so there's precedent. :-)
> -There would have to be some way to pick out only certain params from a
> template, since I think there are almost certainly features that are
> configured using a subset of say puppet/controller.yaml which obviously
> can't just take the params from an entire file.  Although maybe this is
> an indication that we could/should refactor the templates to move some
> of these optional params into their own separate files (at this point I
> think I should take a moment to mention that this is somewhat of a brain
> dump, so I haven't thought through all of the implications yet and I'm
> not sure it all makes sense).
> 
> The nice thing about generating these programmatically is we would
> formalize the interface of the templates somewhat, and it would be
> easier to keep sample envs in sync with the actual implementation.
> You'd never have to worry about someone adding a param to a file but
> forgetting to update the env (or at least it would be easy to catch and
> fix when they did, just run "tox -e genconfig").
> 
> I'm not saying this is a simple or short-term solution, but I'm curious
> what people think about setting this as a longer-term goal, because as I
> think our discussion in Tokyo exposed, we're probably going to have a
> bit of an explosion of sample envs soon and we're going to need some way
> to keep them sane.

So I went ahead and started on this:
https://review.openstack.org/#/c/253638/

It's still got some issues that I'm aware of, and surely some I'm not,
but I think I like where it's going.  Before I spend a ton of time
polishing it and writing up all the environments, I wanted to post
something to get feedback on how people feel about the way it works.

I know it needs to indicate which params don't have defaults and are
thus required (right now I think everything just gets a '' value), and
it would be nice if we could move the 'private' definitions into the
templates themselves somehow.  I left some other thoughts in the commit
message too.

Anyway, please take a look and let me know if this is something I should
pursue.

> 
> Some more comments inline.
> 
> On 11/19/2015 10:16 AM, Steven Hardy wrote:
>> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
>>> On 11/16/2015 04:25 PM, Steven Hardy wrote:
 Hi all,

 I wanted to start some discussion re $subject, because it's been apparrent
 that we have a lack of clarity on this issue (and have done ever since we
 started using parameter_defaults).
>>>
>>> [...]
>>>
 How do people feel about this example, and others like it, where we're
 enabling common, but not mandatory functionality?
>>>
>>> At first I was thinking about something as simple as: "don't use top-level
>>> params for resources which the registry doesn't enable by default".
>>>
>>> It seems to be somewhat what we tried to do with the existing pluggable
>>> resources.
>>>
>>> Also, not to hijack the thread but I wanted to add another question related
>>> to a similar issue:
>>>
>>>   Is there a reason to prefer use of parameters: instead of
>>> parameter_defaults: in the environment files?
>>>
>>> It looks to me that by defaulting to parameter_defaults: users won't need to
>>> update their environment files in case the parameter is moved from top-level
>>> into a specific nested stack so I'm inclined to prefer this. Are there
>>> reasons not to?
>>
>> The main reason is scope - if you use "parameters", you know the data flow
>> happens via the parent template (e.g overcloud-without-mergepy) and you
>> never have to worry about naming collisions outside of that template.
>>
>> But if you use parameter_defaults, all parameters values defined that way
>> are effectively global, and you then have to be much more careful that you
>> never shadow a parameter name and get an unexpected value passed in to it.
>>
>> Here's another example of why we need to decide this btw:
>>
>> https://review.openstack.org/#/c/229471/
>>
>> 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jiří Stránský]

My personal preference is to say:

1. Any templates which are included in the default environment (e.g
overcloud-resource-registry-puppet.yaml), must expose their parameters
via overcloud-without-mergepy.yaml

2. Any templates which are included in the default environment, but via a
"noop" implementation *may* expose their parameters provided they are
common and not implementation/vendor specific.

3. Any templates exposing vendor specific interfaces (e.g at least anything
related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
parameters via the top level template.

How does this sound?


Pardon the longer e-mail please, but i think this topic is very far 
reaching and impactful on the future of TripleO, perhaps even strategic, 
and i'd like to present some food for thought.



I think as we progress towards more composable/customizable overcloud, 
using parameter_defaults will become a necessity in more and more places 
in the templates.


Nowadays we can get away with hierarchical passing of some parameters 
from the top-level template downwards because we can make very strong 
assumptions about how the overcloud is structured, and what each piece 
of the overcloud takes as its parameters. Even though we support 
customization via the resource registry, it's still mostly just 
switching between alternate implementations of the same thing, not 
strong composability.



I would imagine that going forward, TripleO would receive feature 
requests to add custom node types into the deployment, be it e.g. 
separating neutron network node functionality out of controller node 
onto its own hardware, or adding custom 3rd party node types into the 
overcloud, which need to integrate with the rest of the overcloud tightly.


When such scenario is considered, even the most code-static parameters 
like node-type-specific ExtraConfig, or a nova flavor to use for a node 
type, suddenly become dynamic on the code level (think 
parameter_defaults), simply because we can't predict upfront what node 
types we'll have.



I think a parallel with how Puppet evolved can be observed here. It used 
to be that Puppet classes included in deployments formed a sort-of 
hierarchy and got their parameters fed in a top-down cascade. This 
carried limitations on composability of machine configuration manifests 
(collisions when using the same class from multiple places, huge number 
of parameters in the higher-level manifests). Hiera was introduced to 
solve the problem, and nowadays top-level Puppet manifests contain a lot 
of include statements, and the parameter values are mostly read from 
external hiera data files, and hiera values transcend through the class 
hierarchy freely. This hinders easy discoverability of "what settings 
can i tune within this machine's configuration", but judging by the 
adoption of the approach, the benefits probably outweigh the drawbacks. 
In Puppet's case, at least :)


It seems TripleO is hitting similar composability and sanity limits with 
the top-down approach, and the number of parameters which can only be 
fed via parameter_defaults is increasing. (The disadvantage of 
parameter_defaults is that, unlike hiera, we currently have no clear 
namespacing rules, which means a higher chance of conflict. Perhaps the 
unit tests suggested in another subthread would be a good start, maybe 
we could even think about how to do proper namespacing.)



Does what i described seem somewhat accurate? Should we maybe buy into 
the concept of "composable templates, externally fed 
hierarchy-transcending parameters" for the long term?


Thanks for reading this far :)


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jiří Stránský]

On 26.11.2015 14:12, Jiří Stránský wrote:




My personal preference is to say:

1. Any templates which are included in the default environment (e.g
overcloud-resource-registry-puppet.yaml), must expose their parameters
via overcloud-without-mergepy.yaml

2. Any templates which are included in the default environment, but via a
"noop" implementation *may* expose their parameters provided they are
common and not implementation/vendor specific.

3. Any templates exposing vendor specific interfaces (e.g at least anything
related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
parameters via the top level template.

How does this sound?


Pardon the longer e-mail please, but i think this topic is very far
reaching and impactful on the future of TripleO, perhaps even strategic,
and i'd like to present some food for thought.


I think as we progress towards more composable/customizable overcloud,
using parameter_defaults will become a necessity in more and more places
in the templates.

Nowadays we can get away with hierarchical passing of some parameters
from the top-level template downwards because we can make very strong
assumptions about how the overcloud is structured, and what each piece
of the overcloud takes as its parameters. Even though we support
customization via the resource registry, it's still mostly just
switching between alternate implementations of the same thing, not
strong composability.


I would imagine that going forward, TripleO would receive feature
requests to add custom node types into the deployment, be it e.g.
separating neutron network node functionality out of controller node
onto its own hardware, or adding custom 3rd party node types into the
overcloud, which need to integrate with the rest of the overcloud tightly.

When such scenario is considered, even the most code-static parameters
like node-type-specific ExtraConfig, or a nova flavor to use for a node
type, suddenly become dynamic on the code level (think
parameter_defaults), simply because we can't predict upfront what node
types we'll have.


I think a parallel with how Puppet evolved can be observed here. It used
to be that Puppet classes included in deployments formed a sort-of
hierarchy and got their parameters fed in a top-down cascade. This
carried limitations on composability of machine configuration manifests
(collisions when using the same class from multiple places, huge number
of parameters in the higher-level manifests). Hiera was introduced to
solve the problem, and nowadays top-level Puppet manifests contain a lot
of include statements, and the parameter values are mostly read from
external hiera data files, and hiera values transcend through the class
hierarchy freely. This hinders easy discoverability of "what settings
can i tune within this machine's configuration", but judging by the
adoption of the approach, the benefits probably outweigh the drawbacks.
In Puppet's case, at least :)

It seems TripleO is hitting similar composability and sanity limits with
the top-down approach, and the number of parameters which can only be
fed via parameter_defaults is increasing. (The disadvantage of
parameter_defaults is that, unlike hiera, we currently have no clear
namespacing rules, which means a higher chance of conflict. Perhaps the
unit tests suggested in another subthread would be a good start, maybe
we could even think about how to do proper namespacing.)


Does what i described seem somewhat accurate? Should we maybe buy into
the concept of "composable templates, externally fed
hierarchy-transcending parameters" for the long term?


I now realized i might have used too generic or Puppetish terms in the 
explanation, perhaps drowning the gist of the message a bit :) What i'm 
suggesting is: let's consider going with parameter_defaults wherever we 
can, for the sake of composability, and figure out what is the best way 
to prevent parameter name collisions.




Thanks for reading this far :)


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Dan Prince]

On Thu, 2015-11-19 at 16:16 +, Steven Hardy wrote:
> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
> > On 11/16/2015 04:25 PM, Steven Hardy wrote:
> > > Hi all,
> > > 
> > > I wanted to start some discussion re $subject, because it's been
> > > apparrent
> > > that we have a lack of clarity on this issue (and have done ever
> > > since we
> > > started using parameter_defaults).
> > 
> > [...]
> > 
> > > How do people feel about this example, and others like it, where
> > > we're
> > > enabling common, but not mandatory functionality?
> > 
> > At first I was thinking about something as simple as: "don't use
> > top-level
> > params for resources which the registry doesn't enable by default".
> > 
> > It seems to be somewhat what we tried to do with the existing
> > pluggable
> > resources.
> > 
> > Also, not to hijack the thread but I wanted to add another question
> > related
> > to a similar issue:
> > 
> >   Is there a reason to prefer use of parameters: instead of
> > parameter_defaults: in the environment files?
> > 
> > It looks to me that by defaulting to parameter_defaults: users
> > won't need to
> > update their environment files in case the parameter is moved from
> > top-level
> > into a specific nested stack so I'm inclined to prefer this. Are
> > there
> > reasons not to?
> 
> The main reason is scope - if you use "parameters", you know the data
> flow
> happens via the parent template (e.g overcloud-without-mergepy) and
> you
> never have to worry about naming collisions outside of that template.
> 
> But if you use parameter_defaults, all parameters values defined that
> way
> are effectively global, and you then have to be much more careful
> that you
> never shadow a parameter name and get an unexpected value passed in
> to it.
> 
> Here's another example of why we need to decide this btw:
> 
> https://review.openstack.org/#/c/229471/
> 
> Here, we have some workers parameters, going only into
> controller.yaml -
> this is fine, but the new options are completely invisible to users
> who
> look at the overcloud_without_mergepy parameters schema as their
> interface
> (in particular I'm thinking of any UI here).
> 
> My personal preference is to say:
> 
> 1. Any templates which are included in the default environment (e.g
> overcloud-resource-registry-puppet.yaml), must expose their
> parameters
> via overcloud-without-mergepy.yaml
> 
> 2. Any templates which are included in the default environment, but
> via a
> "noop" implementation *may* expose their parameters provided they are
> common and not implementation/vendor specific.
> 
> 3. Any templates exposing vendor specific interfaces (e.g at least
> anything
> related to the OS::TripleO::*ExtraConfig* interfaces) must not expose
> any
> parameters via the top level template.
> 
> How does this sound?

I'd be fine adding these rules into a HACKING document in tripleo-heat-
templates. Perhaps some of these rules fall into "recommendations"
though as there may be exceptions where we want to break things out.

With the new composable roles stuff I was actually going to try to add
all parameters for each available role to the top level template. See
how that might look here:

https://review.openstack.org/#/c/237370/7/overcloud-without-mergepy.yam
l,cm

This would mean a lot more top level parameters... however I think they
will be grouped nicely so they make sense. The compute.yaml and
controller.yaml templates would be much leaner and when it is all
finished just contain parameters related to creation of the baremetal
server and network settings.

Dan


> 
> This does mean we suffer some template bloat from (1) and (2), but it
> makes
> the job of any UI or other tool requiring user input much easier, I
> think?
> 
> Steve
> 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Dan Prince]

On Fri, 2015-11-20 at 16:53 +0100, Jiri Tomasek wrote:
> On 11/16/2015 04:25 PM, Steven Hardy wrote:
> > Hi all,
> > 
> > I wanted to start some discussion re $subject, because it's been
> > apparrent
> > that we have a lack of clarity on this issue (and have done ever
> > since we
> > started using parameter_defaults).
> > 
> > Some context:
> > 
> > - Historically TripleO has provided a fairly comprehensive "top
> > level"
> >    parameters interface, where many per-role and common options are
> >    specified, then passed in to the respective ResourceGroups on
> > deployment
> > 
> > https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tre
> > e/overcloud-without-mergepy.yaml#n14
> > 
> > The nice thing about this approach is it gives a consistent API to
> > the
> > operator, e.g the parameters schema for the main overcloud template
> > defines
> > most of the expected inputs to the deployment.
> > 
> > The main disadvantage is a degree of template bloat, where we wire
> > dozens
> > of parameters into each ResourceGroup, and from there into whatever
> > nested
> > templates consume them.
> > 
> > - When we started adding interfaces (such as all the
> > OS::TripleO::*ExtraConfig*
> >    interfaces, there was a need to enable passing arbitrary
> > additional
> >    values to nested templates, with no way of knowing what they are
> > (e.g to
> >    enable wiring in third-party pieces we have no knowledge of or
> > which
> >    require implementation-specific arguments which don't make sense
> > for all
> >    deployments.
> > 
> > To do this, we made use of the heat parameter_defaults interface,
> > which
> > (unlike normal parameters) have global scope (visible to all nested
> > stacks,
> > without explicitly wiring in the values from the parent):
> > 
> > http://docs.openstack.org/developer/heat/template_guide/environment
> > .html#define-defaults-to-parameters
> > 
> > The nice thing about this approach is its flexibility, any
> > arbitrary
> > values can be provided without affecting the parent templates, and
> > it can
> > allow for a terser implementation because you only specify the
> > parameter
> > definition where it's actually used.
> > 
> > The main disadvantage of this approach is it becomes very much
> > harder to
> > discover an API surface for the operator, e.g the parameters that
> > must be
> > provided on deployment by any CLI/UI tools etc.  This has been
> > partially
> > addressed by the new-for-liberty nested validation heat feature,
> > but
> > there's still a bunch of unsolved complexity around how to actually
> > consume
> > that data and build a coherent consolidated API for user
> > interaction:
> > 
> > https://github.com/openstack/heat-specs/blob/master/specs/liberty/n
> > ested-validation.rst
> > 
> > My question is, where do we draw the line on when to use each
> > interface?
> > 
> > My position has always been that we should only use
> > parameter_defaults for
> > the ExtraConfig interfaces, where we cannot know what reasonable
> > parameters
> > are.  And for all other "core" functionality, we should accept the
> > increased
> > template verbosity and wire arguments in from overcloud-without-
> > mergepy.
> > 
> > However we've got some patches which fall into a grey area, e.g
> > this SSL
> > enablement patch:
> > 
> > https://review.openstack.org/#/c/231930/46/overcloud-without-mergep
> > y.yaml
> > 
> > Here we're actually removing some existing (non functional) top-
> > level
> > parameters, and moving them to parameter_defaults.
> > 
> > I can see the logic behind it, it does make the templates a bit
> > cleaner,
> > but at the expense of discoverablility of those (probably not
> > implementation dependent) parameters.
> > 
> > How do people feel about this example, and others like it, where
> > we're
> > enabling common, but not mandatory functionality?
> > 
> > In particular I'm keen to hear from Mainn and others interested in
> > building
> > UIs on top of TripleO as to which is best from that perspective,
> > and how
> > such arguments may be handled relative to the capabilities mapping
> > proposed
> > here:
> > 
> > https://review.openstack.org/#/c/242439/
> > 
> > Thanks!
> > 
> > Steve
> > 
> > ___
> > ___
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsu
> > bscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> I think I'll try to do a bit of a recap to make sure I understand 
> things. It may shift slightly off the topic of this thread but I
> think 
> it is worth it and it will describe what the GUI is able/expecting to
> work with.
> 
> Template defines parameters and passes them to child templates via 
> resource properties.
> Root template parameter values are set by (in order of precedence):
> 1. 'parameters' param in 'stack create' api call or 'parameters'
> 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

On 11/25/2015 09:13 AM, Dan Prince wrote:
> On Fri, 2015-11-20 at 16:53 +0100, Jiri Tomasek wrote:
>> On 11/16/2015 04:25 PM, Steven Hardy wrote:
>>> Hi all,
>>>
>>> I wanted to start some discussion re $subject, because it's been
>>> apparrent
>>> that we have a lack of clarity on this issue (and have done ever
>>> since we
>>> started using parameter_defaults).
>>>
>>> Some context:
>>>
>>> - Historically TripleO has provided a fairly comprehensive "top
>>> level"
>>>parameters interface, where many per-role and common options are
>>>specified, then passed in to the respective ResourceGroups on
>>> deployment
>>>
>>> https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tre
>>> e/overcloud-without-mergepy.yaml#n14
>>>
>>> The nice thing about this approach is it gives a consistent API to
>>> the
>>> operator, e.g the parameters schema for the main overcloud template
>>> defines
>>> most of the expected inputs to the deployment.
>>>
>>> The main disadvantage is a degree of template bloat, where we wire
>>> dozens
>>> of parameters into each ResourceGroup, and from there into whatever
>>> nested
>>> templates consume them.
>>>
>>> - When we started adding interfaces (such as all the
>>> OS::TripleO::*ExtraConfig*
>>>interfaces, there was a need to enable passing arbitrary
>>> additional
>>>values to nested templates, with no way of knowing what they are
>>> (e.g to
>>>enable wiring in third-party pieces we have no knowledge of or
>>> which
>>>require implementation-specific arguments which don't make sense
>>> for all
>>>deployments.
>>>
>>> To do this, we made use of the heat parameter_defaults interface,
>>> which
>>> (unlike normal parameters) have global scope (visible to all nested
>>> stacks,
>>> without explicitly wiring in the values from the parent):
>>>
>>> http://docs.openstack.org/developer/heat/template_guide/environment
>>> .html#define-defaults-to-parameters
>>>
>>> The nice thing about this approach is its flexibility, any
>>> arbitrary
>>> values can be provided without affecting the parent templates, and
>>> it can
>>> allow for a terser implementation because you only specify the
>>> parameter
>>> definition where it's actually used.
>>>
>>> The main disadvantage of this approach is it becomes very much
>>> harder to
>>> discover an API surface for the operator, e.g the parameters that
>>> must be
>>> provided on deployment by any CLI/UI tools etc.  This has been
>>> partially
>>> addressed by the new-for-liberty nested validation heat feature,
>>> but
>>> there's still a bunch of unsolved complexity around how to actually
>>> consume
>>> that data and build a coherent consolidated API for user
>>> interaction:
>>>
>>> https://github.com/openstack/heat-specs/blob/master/specs/liberty/n
>>> ested-validation.rst
>>>
>>> My question is, where do we draw the line on when to use each
>>> interface?
>>>
>>> My position has always been that we should only use
>>> parameter_defaults for
>>> the ExtraConfig interfaces, where we cannot know what reasonable
>>> parameters
>>> are.  And for all other "core" functionality, we should accept the
>>> increased
>>> template verbosity and wire arguments in from overcloud-without-
>>> mergepy.
>>>
>>> However we've got some patches which fall into a grey area, e.g
>>> this SSL
>>> enablement patch:
>>>
>>> https://review.openstack.org/#/c/231930/46/overcloud-without-mergep
>>> y.yaml
>>>
>>> Here we're actually removing some existing (non functional) top-
>>> level
>>> parameters, and moving them to parameter_defaults.
>>>
>>> I can see the logic behind it, it does make the templates a bit
>>> cleaner,
>>> but at the expense of discoverablility of those (probably not
>>> implementation dependent) parameters.
>>>
>>> How do people feel about this example, and others like it, where
>>> we're
>>> enabling common, but not mandatory functionality?
>>>
>>> In particular I'm keen to hear from Mainn and others interested in
>>> building
>>> UIs on top of TripleO as to which is best from that perspective,
>>> and how
>>> such arguments may be handled relative to the capabilities mapping
>>> proposed
>>> here:
>>>
>>> https://review.openstack.org/#/c/242439/
>>>
>>> Thanks!
>>>
>>> Steve
>>>
>>> ___
>>> ___
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsu
>>> bscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> I think I'll try to do a bit of a recap to make sure I understand 
>> things. It may shift slightly off the topic of this thread but I
>> think 
>> it is worth it and it will describe what the GUI is able/expecting to
>> work with.
>>
>> Template defines parameters and passes them to child templates via 
>> resource properties.
>> Root template parameter values are set by (in order of precedence):
>> 1. 'parameters' param in 'stack 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

I think at the same time we add a mechanism to distinguish between
internal and external parameters, we need to add something to indicate
required v. optional.

With a nested stack, anything that's not part of the top-level parameter
contract is defaulted. The problem is that it loses information on what
is a valid default v. what's simply defaulted to pass validation.


I thought the nested validation spec was supposed to handle that though?
  To me, required vs. optional should be as simple as "Does the parameter
definition have a 'default' key?  If yes, then it's optional, if no,
then it's required for the user to pass a value via a parameter or
parameter_default".  I realize we may not have been following that up to
now for various reasons, but it seems like Heat is already providing a
pretty explicit mechanism for marking params as required, so we ought to
use it.


Ya, I was mistaken here. Taking a look at the cinder-netapp.yaml, it 
looks like we're using this correctly:


...
  CinderNetappBackendName:
type: string
default: 'tripleo_netapp'
  CinderNetappLogin:
type: string
  CinderNetappPassword:
type: string
hidden: true
...


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

On 11/23/2015 01:20 AM, Juan Antonio Osorio wrote:
> 
> 
> On Sat, Nov 21, 2015 at 2:05 AM, Ben Nemec  > wrote:
> 
> Thinking about this some more makes me wonder if we need a sample config
> generator like oslo.config.  It would work off something similar to the
> capabilities map, where you would say
> 
> SSL:
>   templates:
> -puppet/extraconfig/tls/tls-cert-inject.yaml
>   output:
> -environments/enable-ssl.yaml
> 
> And the tool would look at that, read all the params from
> tls-cert-inject.yaml and generate the sample env file.  We'd have to be
> able to do a few new things with the params in order for this to work:
> 
> -Need to specify whether a param is intended to be set as a top-level
> param, parameter_defaults (which we informally do today with the Can be
> overridden by parameter_defaults comment), or internal, to define params
> that shouldn't be exposed in the sample config and are only intended as
> an interface between templates.  There wouldn't be any enforcement of
> the internal type, but Python relies on convention for its private
> members so there's precedent. :-)
> 
> Perhaps a convention could be done in a similar fashion to how things
> are done in
> python. Where parameters passed from top-level could be defined as they
> are defined
> now (with camel-case type of definition) and non-top-parameters could be
> defined with
> lowercase and underscores (or with an underscore prefix). That could
> make things
> clearer and allow us to have a more programmatic approach in the future.
> 
> -There would have to be some way to pick out only certain params from a
> template, since I think there are almost certainly features that are
> configured using a subset of say puppet/controller.yaml which obviously
> can't just take the params from an entire file.  Although maybe this is
> an indication that we could/should refactor the templates to move some
> of these optional params into their own separate files (at this point I
> think I should take a moment to mention that this is somewhat of a brain
> dump, so I haven't thought through all of the implications yet and I'm
> not sure it all makes sense).
> 
> The nice thing about generating these programmatically is we would
> formalize the interface of the templates somewhat, and it would be
> easier to keep sample envs in sync with the actual implementation.
> You'd never have to worry about someone adding a param to a file but
> forgetting to update the env (or at least it would be easy to catch and
> fix when they did, just run "tox -e genconfig").
> 
> I think having such a tool is an excellent idea.
> 
> 
> I'm not saying this is a simple or short-term solution, but I'm curious
> what people think about setting this as a longer-term goal, because as I
> think our discussion in Tokyo exposed, we're probably going to have a
> bit of an explosion of sample envs soon and we're going to need some way
> to keep them sane.
> 
> Some more comments inline.
> 
> On 11/19/2015 10:16 AM, Steven Hardy wrote:
> > On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
> >> On 11/16/2015 04:25 PM, Steven Hardy wrote:
> >>> Hi all,
> >>>
> >>> I wanted to start some discussion re $subject, because it's been
> apparrent
> >>> that we have a lack of clarity on this issue (and have done ever
> since we
> >>> started using parameter_defaults).
> >>
> >> [...]
> >>
> >>> How do people feel about this example, and others like it, where
> we're
> >>> enabling common, but not mandatory functionality?
> >>
> >> At first I was thinking about something as simple as: "don't use
> top-level
> >> params for resources which the registry doesn't enable by default".
> >>
> >> It seems to be somewhat what we tried to do with the existing
> pluggable
> >> resources.
> >>
> >> Also, not to hijack the thread but I wanted to add another
> question related
> >> to a similar issue:
> >>
> >>   Is there a reason to prefer use of parameters: instead of
> >> parameter_defaults: in the environment files?
> >>
> >> It looks to me that by defaulting to parameter_defaults: users
> won't need to
> >> update their environment files in case the parameter is moved
> from top-level
> >> into a specific nested stack so I'm inclined to prefer this. Are
> there
> >> reasons not to?
> >
> > The main reason is scope - if you use "parameters", you know the
> data flow
> > happens via the parent template (e.g overcloud-without-mergepy)
> and you
> > never have to worry about naming collisions outside of that template.
> >
> > But if you use parameter_defaults, all parameters values 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

On 11/23/2015 06:58 AM, Jay Dobies wrote:
> On 11/20/2015 07:05 PM, Ben Nemec wrote:
>> Thinking about this some more makes me wonder if we need a sample config
>> generator like oslo.config.  It would work off something similar to the
>> capabilities map, where you would say
>>
>> SSL:
>>templates:
>>  -puppet/extraconfig/tls/tls-cert-inject.yaml
>>output:
>>  -environments/enable-ssl.yaml
>>
>> And the tool would look at that, read all the params from
>> tls-cert-inject.yaml and generate the sample env file.  We'd have to be
>> able to do a few new things with the params in order for this to work:
>>
>> -Need to specify whether a param is intended to be set as a top-level
>> param, parameter_defaults (which we informally do today with the Can be
>> overridden by parameter_defaults comment), or internal, to define params
>> that shouldn't be exposed in the sample config and are only intended as
>> an interface between templates.  There wouldn't be any enforcement of
>> the internal type, but Python relies on convention for its private
>> members so there's precedent. :-)
> 
> There is new functionality in Heat that will let you pass in a series of 
> templates and environments and it will return:
> 
> - The list of top-level parameters, the same way template-validate 
> always did
> - A list of all nested parameters, keyed by resource.
> 
> Take a look at 
> https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst
>  
> for details and an example.
> 
> That's not entirely what you're getting at, I realize that. I'm glad to 
> see you suggest a convention-based approach because I think that's the 
> only way we're going to be able to convey some of this information.
> 
> I think at the same time we add a mechanism to distinguish between 
> internal and external parameters, we need to add something to indicate 
> required v. optional.
> 
> With a nested stack, anything that's not part of the top-level parameter 
> contract is defaulted. The problem is that it loses information on what 
> is a valid default v. what's simply defaulted to pass validation.

I thought the nested validation spec was supposed to handle that though?
 To me, required vs. optional should be as simple as "Does the parameter
definition have a 'default' key?  If yes, then it's optional, if no,
then it's required for the user to pass a value via a parameter or
parameter_default".  I realize we may not have been following that up to
now for various reasons, but it seems like Heat is already providing a
pretty explicit mechanism for marking params as required, so we ought to
use it.

> 
> I've been noticing this more and more on the vendor integrations. They 
> have parameters that are required (such as a username) and others that 
> are less likely to be changed (I can't think of an example, but I think 
> everyone can see where I'm going with this).
> 
> So I think there are two sorts of things (at least, I'm also thinking 
> off the top of my head) we'd like this tool/sample file to convey:
> 
> - Parameters a user would want to change, as compared to those used for 
> internal data shuffling
> - Information on if the user must supply a value, as compared to 
> parameters with an actual default
> 
> All that said, I dig this idea of a tool that would generate a skeleton 
> environment file.

Cool, I'll try to get a spec written up at some point where we can
bikeshed^W discuss the details.

> 
>> -There would have to be some way to pick out only certain params from a
>> template, since I think there are almost certainly features that are
>> configured using a subset of say puppet/controller.yaml which obviously
>> can't just take the params from an entire file.  Although maybe this is
>> an indication that we could/should refactor the templates to move some
>> of these optional params into their own separate files (at this point I
>> think I should take a moment to mention that this is somewhat of a brain
>> dump, so I haven't thought through all of the implications yet and I'm
>> not sure it all makes sense).
> 
>> The nice thing about generating these programmatically is we would
>> formalize the interface of the templates somewhat, and it would be
>> easier to keep sample envs in sync with the actual implementation.
> 
> You could go so far as to put CI on top of it like we do with the oslo 
> config stuff, which would be neat.
> 
>> You'd never have to worry about someone adding a param to a file but
>> forgetting to update the env (or at least it would be easy to catch and
>> fix when they did, just run "tox -e genconfig").
>>
>> I'm not saying this is a simple or short-term solution, but I'm curious
>> what people think about setting this as a longer-term goal, because as I
>> think our discussion in Tokyo exposed, we're probably going to have a
>> bit of an explosion of sample envs soon and we're going to need some way
>> to keep them sane.
>>
>> Some more comments inline.
>>
>> On 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

On 11/20/2015 07:05 PM, Ben Nemec wrote:

Thinking about this some more makes me wonder if we need a sample config
generator like oslo.config.  It would work off something similar to the
capabilities map, where you would say

SSL:
   templates:
 -puppet/extraconfig/tls/tls-cert-inject.yaml
   output:
 -environments/enable-ssl.yaml

And the tool would look at that, read all the params from
tls-cert-inject.yaml and generate the sample env file.  We'd have to be
able to do a few new things with the params in order for this to work:

-Need to specify whether a param is intended to be set as a top-level
param, parameter_defaults (which we informally do today with the Can be
overridden by parameter_defaults comment), or internal, to define params
that shouldn't be exposed in the sample config and are only intended as
an interface between templates.  There wouldn't be any enforcement of
the internal type, but Python relies on convention for its private
members so there's precedent. :-)


There is new functionality in Heat that will let you pass in a series of 
templates and environments and it will return:


- The list of top-level parameters, the same way template-validate 
always did

- A list of all nested parameters, keyed by resource.

Take a look at 
https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst 
for details and an example.


That's not entirely what you're getting at, I realize that. I'm glad to 
see you suggest a convention-based approach because I think that's the 
only way we're going to be able to convey some of this information.


I think at the same time we add a mechanism to distinguish between 
internal and external parameters, we need to add something to indicate 
required v. optional.


With a nested stack, anything that's not part of the top-level parameter 
contract is defaulted. The problem is that it loses information on what 
is a valid default v. what's simply defaulted to pass validation.


I've been noticing this more and more on the vendor integrations. They 
have parameters that are required (such as a username) and others that 
are less likely to be changed (I can't think of an example, but I think 
everyone can see where I'm going with this).


So I think there are two sorts of things (at least, I'm also thinking 
off the top of my head) we'd like this tool/sample file to convey:


- Parameters a user would want to change, as compared to those used for 
internal data shuffling
- Information on if the user must supply a value, as compared to 
parameters with an actual default


All that said, I dig this idea of a tool that would generate a skeleton 
environment file.



-There would have to be some way to pick out only certain params from a
template, since I think there are almost certainly features that are
configured using a subset of say puppet/controller.yaml which obviously
can't just take the params from an entire file.  Although maybe this is
an indication that we could/should refactor the templates to move some
of these optional params into their own separate files (at this point I
think I should take a moment to mention that this is somewhat of a brain
dump, so I haven't thought through all of the implications yet and I'm
not sure it all makes sense).



The nice thing about generating these programmatically is we would
formalize the interface of the templates somewhat, and it would be
easier to keep sample envs in sync with the actual implementation.


You could go so far as to put CI on top of it like we do with the oslo 
config stuff, which would be neat.



You'd never have to worry about someone adding a param to a file but
forgetting to update the env (or at least it would be easy to catch and
fix when they did, just run "tox -e genconfig").

I'm not saying this is a simple or short-term solution, but I'm curious
what people think about setting this as a longer-term goal, because as I
think our discussion in Tokyo exposed, we're probably going to have a
bit of an explosion of sample envs soon and we're going to need some way
to keep them sane.

Some more comments inline.

On 11/19/2015 10:16 AM, Steven Hardy wrote:

On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:

On 11/16/2015 04:25 PM, Steven Hardy wrote:

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).


[...]


How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?


At first I was thinking about something as simple as: "don't use top-level
params for resources which the registry doesn't enable by default".

It seems to be somewhat what we tried to do with the existing pluggable
resources.

Also, not to hijack the thread but I wanted to add another question related
to a similar issue:

   Is there a reason to prefer use of parameters: instead 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Juan Antonio Osorio]

On Sat, Nov 21, 2015 at 2:05 AM, Ben Nemec  wrote:

> Thinking about this some more makes me wonder if we need a sample config
> generator like oslo.config.  It would work off something similar to the
> capabilities map, where you would say
>
> SSL:
>   templates:
> -puppet/extraconfig/tls/tls-cert-inject.yaml
>   output:
> -environments/enable-ssl.yaml
>
> And the tool would look at that, read all the params from
> tls-cert-inject.yaml and generate the sample env file.  We'd have to be
> able to do a few new things with the params in order for this to work:
>
> -Need to specify whether a param is intended to be set as a top-level
> param, parameter_defaults (which we informally do today with the Can be
> overridden by parameter_defaults comment), or internal, to define params
> that shouldn't be exposed in the sample config and are only intended as
> an interface between templates.  There wouldn't be any enforcement of
> the internal type, but Python relies on convention for its private
> members so there's precedent. :-)
>
Perhaps a convention could be done in a similar fashion to how things are
done in
python. Where parameters passed from top-level could be defined as they are
defined
now (with camel-case type of definition) and non-top-parameters could be
defined with
lowercase and underscores (or with an underscore prefix). That could make
things
clearer and allow us to have a more programmatic approach in the future.

> -There would have to be some way to pick out only certain params from a
> template, since I think there are almost certainly features that are
> configured using a subset of say puppet/controller.yaml which obviously
> can't just take the params from an entire file.  Although maybe this is
> an indication that we could/should refactor the templates to move some
> of these optional params into their own separate files (at this point I
> think I should take a moment to mention that this is somewhat of a brain
> dump, so I haven't thought through all of the implications yet and I'm
> not sure it all makes sense).
>
> The nice thing about generating these programmatically is we would
> formalize the interface of the templates somewhat, and it would be
> easier to keep sample envs in sync with the actual implementation.
> You'd never have to worry about someone adding a param to a file but
> forgetting to update the env (or at least it would be easy to catch and
> fix when they did, just run "tox -e genconfig").
>
I think having such a tool is an excellent idea.

>
> I'm not saying this is a simple or short-term solution, but I'm curious
> what people think about setting this as a longer-term goal, because as I
> think our discussion in Tokyo exposed, we're probably going to have a
> bit of an explosion of sample envs soon and we're going to need some way
> to keep them sane.
>
> Some more comments inline.
>
> On 11/19/2015 10:16 AM, Steven Hardy wrote:
> > On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
> >> On 11/16/2015 04:25 PM, Steven Hardy wrote:
> >>> Hi all,
> >>>
> >>> I wanted to start some discussion re $subject, because it's been
> apparrent
> >>> that we have a lack of clarity on this issue (and have done ever since
> we
> >>> started using parameter_defaults).
> >>
> >> [...]
> >>
> >>> How do people feel about this example, and others like it, where we're
> >>> enabling common, but not mandatory functionality?
> >>
> >> At first I was thinking about something as simple as: "don't use
> top-level
> >> params for resources which the registry doesn't enable by default".
> >>
> >> It seems to be somewhat what we tried to do with the existing pluggable
> >> resources.
> >>
> >> Also, not to hijack the thread but I wanted to add another question
> related
> >> to a similar issue:
> >>
> >>   Is there a reason to prefer use of parameters: instead of
> >> parameter_defaults: in the environment files?
> >>
> >> It looks to me that by defaulting to parameter_defaults: users won't
> need to
> >> update their environment files in case the parameter is moved from
> top-level
> >> into a specific nested stack so I'm inclined to prefer this. Are there
> >> reasons not to?
> >
> > The main reason is scope - if you use "parameters", you know the data
> flow
> > happens via the parent template (e.g overcloud-without-mergepy) and you
> > never have to worry about naming collisions outside of that template.
> >
> > But if you use parameter_defaults, all parameters values defined that way
> > are effectively global, and you then have to be much more careful that
> you
> > never shadow a parameter name and get an unexpected value passed in to
> it.
> >
> > Here's another example of why we need to decide this btw:
> >
> > https://review.openstack.org/#/c/229471/
> >
> > Here, we have some workers parameters, going only into controller.yaml -
> > this is fine, but the new options are completely invisible to users who
> > look at the 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jiri Tomasek]

On 11/16/2015 04:25 PM, Steven Hardy wrote:

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).

Some context:

- Historically TripleO has provided a fairly comprehensive "top level"
   parameters interface, where many per-role and common options are
   specified, then passed in to the respective ResourceGroups on deployment

https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-without-mergepy.yaml#n14

The nice thing about this approach is it gives a consistent API to the
operator, e.g the parameters schema for the main overcloud template defines
most of the expected inputs to the deployment.

The main disadvantage is a degree of template bloat, where we wire dozens
of parameters into each ResourceGroup, and from there into whatever nested
templates consume them.

- When we started adding interfaces (such as all the OS::TripleO::*ExtraConfig*
   interfaces, there was a need to enable passing arbitrary additional
   values to nested templates, with no way of knowing what they are (e.g to
   enable wiring in third-party pieces we have no knowledge of or which
   require implementation-specific arguments which don't make sense for all
   deployments.

To do this, we made use of the heat parameter_defaults interface, which
(unlike normal parameters) have global scope (visible to all nested stacks,
without explicitly wiring in the values from the parent):

http://docs.openstack.org/developer/heat/template_guide/environment.html#define-defaults-to-parameters

The nice thing about this approach is its flexibility, any arbitrary
values can be provided without affecting the parent templates, and it can
allow for a terser implementation because you only specify the parameter
definition where it's actually used.

The main disadvantage of this approach is it becomes very much harder to
discover an API surface for the operator, e.g the parameters that must be
provided on deployment by any CLI/UI tools etc.  This has been partially
addressed by the new-for-liberty nested validation heat feature, but
there's still a bunch of unsolved complexity around how to actually consume
that data and build a coherent consolidated API for user interaction:

https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst

My question is, where do we draw the line on when to use each interface?

My position has always been that we should only use parameter_defaults for
the ExtraConfig interfaces, where we cannot know what reasonable parameters
are.  And for all other "core" functionality, we should accept the increased
template verbosity and wire arguments in from overcloud-without-mergepy.

However we've got some patches which fall into a grey area, e.g this SSL
enablement patch:

https://review.openstack.org/#/c/231930/46/overcloud-without-mergepy.yaml

Here we're actually removing some existing (non functional) top-level
parameters, and moving them to parameter_defaults.

I can see the logic behind it, it does make the templates a bit cleaner,
but at the expense of discoverablility of those (probably not
implementation dependent) parameters.

How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?

In particular I'm keen to hear from Mainn and others interested in building
UIs on top of TripleO as to which is best from that perspective, and how
such arguments may be handled relative to the capabilities mapping proposed
here:

https://review.openstack.org/#/c/242439/

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


I think I'll try to do a bit of a recap to make sure I understand 
things. It may shift slightly off the topic of this thread but I think 
it is worth it and it will describe what the GUI is able/expecting to 
work with.


Template defines parameters and passes them to child templates via 
resource properties.

Root template parameter values are set by (in order of precedence):
1. 'parameters' param in 'stack create' api call or 'parameters' section 
in environment

2. 'parameter_defaults' section in environment
3. 'default' in parameter definition in template

Non-root template parameter values are set by (in order of precedence):
1. parent resource properties
2. 'parameter_defaults' in environment
3. 'default' in parameter definition in template

The name collisions in parameter_defaults should not be a problem since 
the template author should make sure, the parameters names he defines 
don't collide with other templates.


The GUI's main goal (same as CLI and tripleo-common) is not to hardcode 
anything and use THT (or 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Ben Nemec]

Thinking about this some more makes me wonder if we need a sample config
generator like oslo.config.  It would work off something similar to the
capabilities map, where you would say

SSL:
  templates:
-puppet/extraconfig/tls/tls-cert-inject.yaml
  output:
-environments/enable-ssl.yaml

And the tool would look at that, read all the params from
tls-cert-inject.yaml and generate the sample env file.  We'd have to be
able to do a few new things with the params in order for this to work:

-Need to specify whether a param is intended to be set as a top-level
param, parameter_defaults (which we informally do today with the Can be
overridden by parameter_defaults comment), or internal, to define params
that shouldn't be exposed in the sample config and are only intended as
an interface between templates.  There wouldn't be any enforcement of
the internal type, but Python relies on convention for its private
members so there's precedent. :-)
-There would have to be some way to pick out only certain params from a
template, since I think there are almost certainly features that are
configured using a subset of say puppet/controller.yaml which obviously
can't just take the params from an entire file.  Although maybe this is
an indication that we could/should refactor the templates to move some
of these optional params into their own separate files (at this point I
think I should take a moment to mention that this is somewhat of a brain
dump, so I haven't thought through all of the implications yet and I'm
not sure it all makes sense).

The nice thing about generating these programmatically is we would
formalize the interface of the templates somewhat, and it would be
easier to keep sample envs in sync with the actual implementation.
You'd never have to worry about someone adding a param to a file but
forgetting to update the env (or at least it would be easy to catch and
fix when they did, just run "tox -e genconfig").

I'm not saying this is a simple or short-term solution, but I'm curious
what people think about setting this as a longer-term goal, because as I
think our discussion in Tokyo exposed, we're probably going to have a
bit of an explosion of sample envs soon and we're going to need some way
to keep them sane.

Some more comments inline.

On 11/19/2015 10:16 AM, Steven Hardy wrote:
> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
>> On 11/16/2015 04:25 PM, Steven Hardy wrote:
>>> Hi all,
>>>
>>> I wanted to start some discussion re $subject, because it's been apparrent
>>> that we have a lack of clarity on this issue (and have done ever since we
>>> started using parameter_defaults).
>>
>> [...]
>>
>>> How do people feel about this example, and others like it, where we're
>>> enabling common, but not mandatory functionality?
>>
>> At first I was thinking about something as simple as: "don't use top-level
>> params for resources which the registry doesn't enable by default".
>>
>> It seems to be somewhat what we tried to do with the existing pluggable
>> resources.
>>
>> Also, not to hijack the thread but I wanted to add another question related
>> to a similar issue:
>>
>>   Is there a reason to prefer use of parameters: instead of
>> parameter_defaults: in the environment files?
>>
>> It looks to me that by defaulting to parameter_defaults: users won't need to
>> update their environment files in case the parameter is moved from top-level
>> into a specific nested stack so I'm inclined to prefer this. Are there
>> reasons not to?
> 
> The main reason is scope - if you use "parameters", you know the data flow
> happens via the parent template (e.g overcloud-without-mergepy) and you
> never have to worry about naming collisions outside of that template.
> 
> But if you use parameter_defaults, all parameters values defined that way
> are effectively global, and you then have to be much more careful that you
> never shadow a parameter name and get an unexpected value passed in to it.
> 
> Here's another example of why we need to decide this btw:
> 
> https://review.openstack.org/#/c/229471/
> 
> Here, we have some workers parameters, going only into controller.yaml -
> this is fine, but the new options are completely invisible to users who
> look at the overcloud_without_mergepy parameters schema as their interface
> (in particular I'm thinking of any UI here).
> 
> My personal preference is to say:
> 
> 1. Any templates which are included in the default environment (e.g
> overcloud-resource-registry-puppet.yaml), must expose their parameters
> via overcloud-without-mergepy.yaml
> 
> 2. Any templates which are included in the default environment, but via a
> "noop" implementation *may* expose their parameters provided they are
> common and not implementation/vendor specific.

This seems like a reasonable approach, although that "may" still leaves
a lot of room for bikeshedding. ;-)

It might be good to say that in this case it is "preferred" to use a
top-level param, but if there's a 

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

My personal preference is to say:

1. Any templates which are included in the default environment (e.g
overcloud-resource-registry-puppet.yaml), must expose their parameters
via overcloud-without-mergepy.yaml

2. Any templates which are included in the default environment, but via a
"noop" implementation *may* expose their parameters provided they are
common and not implementation/vendor specific.


I think this makes sense. Two combination of these two represent what 
TripleO views as the common* API for deploying.


* "common" in the sense that you may not use all of them every time, but 
they are part of the expected deployments.


I still have concerns with us not treating this strongly enough as a 
versioned API, but we can discuss that in a different thread. That's 
more of a "when" we change the parameters v. the conventions on "how" we 
do it.



3. Any templates exposing vendor specific interfaces (e.g at least anything
related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
parameters via the top level template.

How does this sound?

This does mean we suffer some template bloat from (1) and (2), but it makes
the job of any UI or other tool requiring user input much easier, I think?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Steven Hardy]

On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
> On 11/16/2015 04:25 PM, Steven Hardy wrote:
> >Hi all,
> >
> >I wanted to start some discussion re $subject, because it's been apparrent
> >that we have a lack of clarity on this issue (and have done ever since we
> >started using parameter_defaults).
> 
> [...]
> 
> >How do people feel about this example, and others like it, where we're
> >enabling common, but not mandatory functionality?
> 
> At first I was thinking about something as simple as: "don't use top-level
> params for resources which the registry doesn't enable by default".
> 
> It seems to be somewhat what we tried to do with the existing pluggable
> resources.
> 
> Also, not to hijack the thread but I wanted to add another question related
> to a similar issue:
> 
>   Is there a reason to prefer use of parameters: instead of
> parameter_defaults: in the environment files?
> 
> It looks to me that by defaulting to parameter_defaults: users won't need to
> update their environment files in case the parameter is moved from top-level
> into a specific nested stack so I'm inclined to prefer this. Are there
> reasons not to?

The main reason is scope - if you use "parameters", you know the data flow
happens via the parent template (e.g overcloud-without-mergepy) and you
never have to worry about naming collisions outside of that template.

But if you use parameter_defaults, all parameters values defined that way
are effectively global, and you then have to be much more careful that you
never shadow a parameter name and get an unexpected value passed in to it.

Here's another example of why we need to decide this btw:

https://review.openstack.org/#/c/229471/

Here, we have some workers parameters, going only into controller.yaml -
this is fine, but the new options are completely invisible to users who
look at the overcloud_without_mergepy parameters schema as their interface
(in particular I'm thinking of any UI here).

My personal preference is to say:

1. Any templates which are included in the default environment (e.g
overcloud-resource-registry-puppet.yaml), must expose their parameters
via overcloud-without-mergepy.yaml

2. Any templates which are included in the default environment, but via a
"noop" implementation *may* expose their parameters provided they are
common and not implementation/vendor specific.

3. Any templates exposing vendor specific interfaces (e.g at least anything
related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
parameters via the top level template.

How does this sound?

This does mean we suffer some template bloat from (1) and (2), but it makes
the job of any UI or other tool requiring user input much easier, I think?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Juan Antonio Osorio]

On Thu, Nov 19, 2015 at 6:16 PM, Steven Hardy  wrote:

> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
> > On 11/16/2015 04:25 PM, Steven Hardy wrote:
> > >Hi all,
> > >
> > >I wanted to start some discussion re $subject, because it's been
> apparrent
> > >that we have a lack of clarity on this issue (and have done ever since
> we
> > >started using parameter_defaults).
> >
> > [...]
> >
> > >How do people feel about this example, and others like it, where we're
> > >enabling common, but not mandatory functionality?
> >
> > At first I was thinking about something as simple as: "don't use
> top-level
> > params for resources which the registry doesn't enable by default".
> >
> > It seems to be somewhat what we tried to do with the existing pluggable
> > resources.
> >
> > Also, not to hijack the thread but I wanted to add another question
> related
> > to a similar issue:
> >
> >   Is there a reason to prefer use of parameters: instead of
> > parameter_defaults: in the environment files?
> >
> > It looks to me that by defaulting to parameter_defaults: users won't
> need to
> > update their environment files in case the parameter is moved from
> top-level
> > into a specific nested stack so I'm inclined to prefer this. Are there
> > reasons not to?
>
> The main reason is scope - if you use "parameters", you know the data flow
> happens via the parent template (e.g overcloud-without-mergepy) and you
> never have to worry about naming collisions outside of that template.
>
> But if you use parameter_defaults, all parameters values defined that way
> are effectively global, and you then have to be much more careful that you
> never shadow a parameter name and get an unexpected value passed in to it.
>
> Here's another example of why we need to decide this btw:
>
> https://review.openstack.org/#/c/229471/
>
> Here, we have some workers parameters, going only into controller.yaml -
> this is fine, but the new options are completely invisible to users who
> look at the overcloud_without_mergepy parameters schema as their interface
> (in particular I'm thinking of any UI here).
>
> My personal preference is to say:
>
> 1. Any templates which are included in the default environment (e.g
> overcloud-resource-registry-puppet.yaml), must expose their parameters
> via overcloud-without-mergepy.yaml
>
> 2. Any templates which are included in the default environment, but via a
> "noop" implementation *may* expose their parameters provided they are
> common and not implementation/vendor specific.


I guess the TLS patch falls into this category. I would say that this needs
to be
documented somewhere though. So if there are parameters that would be
available via parameter defaults, then that should be visible somewhere.
Is there any preferred place to add such a thing?


>
> 3. Any templates exposing vendor specific interfaces (e.g at least anything
> related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
> parameters via the top level template.
>
> How does this sound?
>

This sounds reasonable. Though the same question as above applies.

>
> This does mean we suffer some template bloat from (1) and (2), but it makes
> the job of any UI or other tool requiring user input much easier, I think?
>
> Steve
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Juan Antonio Osorio R.
e-mail: jaosor...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Steven Hardy]

On Thu, Nov 19, 2015 at 06:50:35PM +0200, Juan Antonio Osorio wrote:
>On Thu, Nov 19, 2015 at 6:16 PM, Steven Hardy  wrote:
> 
>  On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:
>  > On 11/16/2015 04:25 PM, Steven Hardy wrote:
>  > >Hi all,
>  > >
>  > >I wanted to start some discussion re $subject, because it's been
>  apparrent
>  > >that we have a lack of clarity on this issue (and have done ever
>  since we
>  > >started using parameter_defaults).
>  >
>  > [...]
>  >
>  > >How do people feel about this example, and others like it, where
>  we're
>  > >enabling common, but not mandatory functionality?
>  >
>  > At first I was thinking about something as simple as: "don't use
>  top-level
>  > params for resources which the registry doesn't enable by default".
>  >
>  > It seems to be somewhat what we tried to do with the existing
>  pluggable
>  > resources.
>  >
>  > Also, not to hijack the thread but I wanted to add another question
>  related
>  > to a similar issue:
>  >
>  >A  A Is there a reason to prefer use of parameters: instead of
>  > parameter_defaults: in the environment files?
>  >
>  > It looks to me that by defaulting to parameter_defaults: users won't
>  need to
>  > update their environment files in case the parameter is moved from
>  top-level
>  > into a specific nested stack so I'm inclined to prefer this. Are there
>  > reasons not to?
> 
>  The main reason is scope - if you use "parameters", you know the data
>  flow
>  happens via the parent template (e.g overcloud-without-mergepy) and you
>  never have to worry about naming collisions outside of that template.
> 
>  But if you use parameter_defaults, all parameters values defined that
>  way
>  are effectively global, and you then have to be much more careful that
>  you
>  never shadow a parameter name and get an unexpected value passed in to
>  it.
> 
>  Here's another example of why we need to decide this btw:
> 
>  https://review.openstack.org/#/c/229471/
> 
>  Here, we have some workers parameters, going only into controller.yaml -
>  this is fine, but the new options are completely invisible to users who
>  look at the overcloud_without_mergepy parameters schema as their
>  interface
>  (in particular I'm thinking of any UI here).
> 
>  My personal preference is to say:
> 
>  1. Any templates which are included in the default environment (e.g
>  overcloud-resource-registry-puppet.yaml), must expose their parameters
>  via overcloud-without-mergepy.yaml
> 
>  2. Any templates which are included in the default environment, but via
>  a
>  "noop" implementation *may* expose their parameters provided they are
>  common and not implementation/vendor specific.
> 
>A 
>I guess the TLS patch falls into this category. I would say that this
>needs to be
>documented somewhere though. So if there are parameters that would be
>available via parameter defaults, then that should be visible somewhere.
>Is there any preferred place to add such a thing?

So far, we've just been documenting them via example environment files,
such as you have done e.g in environments/enable-tls.yaml

The problem with this is it's not something that's easily discoverable by
any user/UI, the values in the example aren't necessarily exhaustive, and
it's not convenient to have heat validate example environments like this,
because they typically contain dummy values the user must know to modify.

There is work going on to build a schema around these example environments:

https://review.openstack.org/#/c/242439/

But even with that in place, it doesn't provide a convenient interface for
the "common to most deployments" type options most deployers will want to
be exposed to.

>  3. Any templates exposing vendor specific interfaces (e.g at least
>  anything
>  related to the OS::TripleO::*ExtraConfig* interfaces) must not expose
>  any
>  parameters via the top level template.
> 
>  How does this sound?
> 
>This sounds reasonable. Though the same question as above applies.

Same as above, example environments is all we have right now.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Steven Hardy]

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).

Some context:

- Historically TripleO has provided a fairly comprehensive "top level"
  parameters interface, where many per-role and common options are
  specified, then passed in to the respective ResourceGroups on deployment

https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-without-mergepy.yaml#n14

The nice thing about this approach is it gives a consistent API to the
operator, e.g the parameters schema for the main overcloud template defines
most of the expected inputs to the deployment.

The main disadvantage is a degree of template bloat, where we wire dozens
of parameters into each ResourceGroup, and from there into whatever nested
templates consume them.

- When we started adding interfaces (such as all the OS::TripleO::*ExtraConfig*
  interfaces, there was a need to enable passing arbitrary additional
  values to nested templates, with no way of knowing what they are (e.g to
  enable wiring in third-party pieces we have no knowledge of or which
  require implementation-specific arguments which don't make sense for all
  deployments.

To do this, we made use of the heat parameter_defaults interface, which
(unlike normal parameters) have global scope (visible to all nested stacks,
without explicitly wiring in the values from the parent):

http://docs.openstack.org/developer/heat/template_guide/environment.html#define-defaults-to-parameters

The nice thing about this approach is its flexibility, any arbitrary
values can be provided without affecting the parent templates, and it can
allow for a terser implementation because you only specify the parameter
definition where it's actually used.

The main disadvantage of this approach is it becomes very much harder to
discover an API surface for the operator, e.g the parameters that must be
provided on deployment by any CLI/UI tools etc.  This has been partially
addressed by the new-for-liberty nested validation heat feature, but
there's still a bunch of unsolved complexity around how to actually consume
that data and build a coherent consolidated API for user interaction:

https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst

My question is, where do we draw the line on when to use each interface?

My position has always been that we should only use parameter_defaults for
the ExtraConfig interfaces, where we cannot know what reasonable parameters
are.  And for all other "core" functionality, we should accept the increased
template verbosity and wire arguments in from overcloud-without-mergepy.

However we've got some patches which fall into a grey area, e.g this SSL
enablement patch:

https://review.openstack.org/#/c/231930/46/overcloud-without-mergepy.yaml

Here we're actually removing some existing (non functional) top-level
parameters, and moving them to parameter_defaults.

I can see the logic behind it, it does make the templates a bit cleaner,
but at the expense of discoverablility of those (probably not
implementation dependent) parameters.

How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?

In particular I'm keen to hear from Mainn and others interested in building
UIs on top of TripleO as to which is best from that perspective, and how
such arguments may be handled relative to the capabilities mapping proposed
here:

https://review.openstack.org/#/c/242439/

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Giulio Fidente]

On 11/16/2015 04:25 PM, Steven Hardy wrote:

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).


[...]


How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?


At first I was thinking about something as simple as: "don't use 
top-level params for resources which the registry doesn't enable by 
default".


It seems to be somewhat what we tried to do with the existing pluggable 
resources.


Also, not to hijack the thread but I wanted to add another question 
related to a similar issue:


  Is there a reason to prefer use of parameters: instead of 
parameter_defaults: in the environment files?


It looks to me that by defaulting to parameter_defaults: users won't 
need to update their environment files in case the parameter is moved 
from top-level into a specific nested stack so I'm inclined to prefer 
this. Are there reasons not to?

--
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).

Some context:

- Historically TripleO has provided a fairly comprehensive "top level"
   parameters interface, where many per-role and common options are
   specified, then passed in to the respective ResourceGroups on deployment

https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-without-mergepy.yaml#n14

The nice thing about this approach is it gives a consistent API to the
operator, e.g the parameters schema for the main overcloud template defines
most of the expected inputs to the deployment.

The main disadvantage is a degree of template bloat, where we wire dozens
of parameters into each ResourceGroup, and from there into whatever nested
templates consume them.

- When we started adding interfaces (such as all the OS::TripleO::*ExtraConfig*
   interfaces, there was a need to enable passing arbitrary additional
   values to nested templates, with no way of knowing what they are (e.g to
   enable wiring in third-party pieces we have no knowledge of or which
   require implementation-specific arguments which don't make sense for all
   deployments.

To do this, we made use of the heat parameter_defaults interface, which
(unlike normal parameters) have global scope (visible to all nested stacks,
without explicitly wiring in the values from the parent):

http://docs.openstack.org/developer/heat/template_guide/environment.html#define-defaults-to-parameters

The nice thing about this approach is its flexibility, any arbitrary
values can be provided without affecting the parent templates, and it can
allow for a terser implementation because you only specify the parameter
definition where it's actually used.

The main disadvantage of this approach is it becomes very much harder to
discover an API surface for the operator, e.g the parameters that must be
provided on deployment by any CLI/UI tools etc.  This has been partially
addressed by the new-for-liberty nested validation heat feature, but
there's still a bunch of unsolved complexity around how to actually consume
that data and build a coherent consolidated API for user interaction:

https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst

My question is, where do we draw the line on when to use each interface?

My position has always been that we should only use parameter_defaults for
the ExtraConfig interfaces, where we cannot know what reasonable parameters
are.  And for all other "core" functionality, we should accept the increased
template verbosity and wire arguments in from overcloud-without-mergepy.

However we've got some patches which fall into a grey area, e.g this SSL
enablement patch:

https://review.openstack.org/#/c/231930/46/overcloud-without-mergepy.yaml

Here we're actually removing some existing (non functional) top-level
parameters, and moving them to parameter_defaults.

I can see the logic behind it, it does make the templates a bit cleaner,
but at the expense of discoverablility of those (probably not
implementation dependent) parameters.

How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?

In particular I'm keen to hear from Mainn and others interested in building
UIs on top of TripleO as to which is best from that perspective, and how
such arguments may be handled relative to the capabilities mapping proposed
here:

https://review.openstack.org/#/c/242439/

Thanks!

Steve



(in re-reading this, I realize I'm not providing an answer to how I feel 
about the example as I am adding some more thoughts in general; 
apologies for that)


I see there being a few issues with the current approach:

- I'll get this one out of the way first, even though it's not the 
biggest issue. The name 'parameter_defaults' tends to confuse new 3rd 
party integrators since we're not using it as a default per se. I 
understand from Heat's point of view it's defaulting a parameter value, 
but from the user's standpoint, they are setting an actual value to be 
used. Perhaps this can be solved with better docs, and I largely mention 
it because I can see most of what you wrote in here turning into 
documentation, so it'll be good to have it mentioned.


- Back to the point of your e-mail, there are two ways to view it.

-- If you define too few parameters at the top level, you end up with a 
lot of comments like the following inside of nested templates:


ControlPlaneSubnetCidr: # Override this via parameter_defaults

or

# To be defined via a local or global environment in parameter_defaults
  rhel_reg_activation_key:
type: string

There are other examples too, but the thing to note is that we've so far 
been pretty good about adding those comments. It's not a programmatic 
marker, which may be a problem for a UX, but at least the