[Puppet Users] Re: Puppet 2.7, hiera 1.0 and hiera as an ENC
John,
Many thanks for your long reply, it is much appreciated. You have assured
me in teh conclusions and course of action and i am glad that my
appreciation of teh problem I have was not so far from the reality.
Thanks again.
Guillem
On Wednesday, 26 September 2012 16:52:19 UTC+1, Guillem Liarte wrote:
This is the situation I have:
All my hosts are the* same OS.*
All my host are in the* same puppet environment,* so I cannot use
%{environment}
I have a module that sets all the *basic* functionality for the OS,
resolution, authentication, security, packages, etc
I have a module for each application hosted.
At the moment all the 'data' is in Puppet, mostly in parametrised classes
in site.pp.
What I want to get is a hiera set-up that allows me to use this structure:
:hierarchy:
- global # source of application names (classes? modules?) and
environments list
- basic # data for the basic class
- prod/%{application}/%{hostname}# hostname files for specific
data
- prod/%{application}/%{env} # environmental data for
each application (module)
- prod/%{application}/default # default data for an
application
- nonprod/%{sysclass}/%{hostname}
- nonprod/%{sysclass}/%{env}
- nonprod/%{sysclass}/default
Then to have something like this under the datadir:
#├── global.yaml
#├── basic.yaml
#├── nonprod
#│ ├── app1
#│ │ ├── common-integration.yamlAlfresco common
integration
#│ │ ├── continuous-integration.yaml
#│ │ ├── dev.yaml
#│ │ ├── default.yaml
#│ │ ├── host1.yaml
#│ │ ├── host2.yaml
#│ │ ├── performance.yaml
#│ │ ├── qa.yaml
#│ │ ├── test.yaml
#│ │ └── uat.yaml
#│ └── app2
#└── prod
#├── app1
#└── app2
#
# etc.
In global.yaml
---
:classes:
basic:
app1:
app2:
app3:
app4:
:env:
test:
dev:
commonint:
continuousint:
dev:
performance:
qa:
test:
uat:
in app1 default.yaml:
---
classes:
app1:
app1_version: 'latest'
in app1 dev.yaml:
---
app1_version: '3.0'
If I wanted host1 and host2 to be part of dev for app1:
host1.yaml:
---
classes:
basic:
app1:
env:
dev:
maybe in host2 I want to override version too:
host2.yaml
---
classes:
basic:
app1:
env:
dev:
app1_version: '3.1'
So in short, I would like hiera to be a source of facts, where I can get
information that feeds Puppet in order to classify the nodes and to feed
the parametrised classes.
I recently found this blog entry:
http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
Gary has been very helpful and I have got an idea of what needs doing. I
can query all the data the way I want using the hiera command. Something
like these:
hiera app1_version sysclass=app1 env=dev
Returns the expected '3.0' and if I query by adding teh host:
hiera app1_version sysclass=app1 env=dev hostname=host1
I get 3.1. Cool!
Example using Gary's approach:
/opt/puppet-data/nonprod/hieratest/default.yaml
---
classes: hieratest
env: hieratest_default
/opt/puppet-data/nonprod/hieratest/host1.yaml
---
classes:
hieratest:
env: 'hieratest_performance'
# hiera env sysclass=hieratest --debug
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source
nonprod/hieratest/default
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Found env in
nonprod/hieratest/default
hieratest_default
# hiera type sysclass=hieratest hostname=host1 --debug
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source
nonprod/hieratest/host1
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Found env in nonprod/hieratest/host1
hieratest_performance
But when it comes to use this in Puppet the results are not as I expect,
nothing happens, it just does a run no classes are used. I see that the
basic class custom facts are loaded, but nothing gets executed, as if the
catalogue for host1 would not include it.
In Puppet I expect to just have:
in site.pp:
node default {}
And then in each application’s init.pp:
$env = hiera(env) this allows me to get the right config files
(with are maintained in a git repo)
$app1_version = hiera(app1_version) this allows me to set the right
RPM version (from satellite/spacewalk/RHN)
As per Gary's post, I can use hiera as node terminus, and
[Puppet Users] Re: Puppet 2.7, hiera 1.0 and hiera as an ENC
All,
Do I understand I have hit a dead-end?
On Wednesday, 26 September 2012 16:52:19 UTC+1, Guillem Liarte wrote:
This is the situation I have:
All my hosts are the* same OS.*
All my host are in the* same puppet environment,* so I cannot use
%{environment}
I have a module that sets all the *basic* functionality for the OS,
resolution, authentication, security, packages, etc
I have a module for each application hosted.
At the moment all the 'data' is in Puppet, mostly in parametrised classes
in site.pp.
What I want to get is a hiera set-up that allows me to use this structure:
:hierarchy:
- global # source of application names (classes? modules?) and
environments list
- basic # data for the basic class
- prod/%{application}/%{hostname}# hostname files for specific
data
- prod/%{application}/%{env} # environmental data for
each application (module)
- prod/%{application}/default # default data for an
application
- nonprod/%{sysclass}/%{hostname}
- nonprod/%{sysclass}/%{env}
- nonprod/%{sysclass}/default
Then to have something like this under the datadir:
#├── global.yaml
#├── basic.yaml
#├── nonprod
#│ ├── app1
#│ │ ├── common-integration.yamlAlfresco common
integration
#│ │ ├── continuous-integration.yaml
#│ │ ├── dev.yaml
#│ │ ├── default.yaml
#│ │ ├── host1.yaml
#│ │ ├── host2.yaml
#│ │ ├── performance.yaml
#│ │ ├── qa.yaml
#│ │ ├── test.yaml
#│ │ └── uat.yaml
#│ └── app2
#└── prod
#├── app1
#└── app2
#
# etc.
In global.yaml
---
:classes:
basic:
app1:
app2:
app3:
app4:
:env:
test:
dev:
commonint:
continuousint:
dev:
performance:
qa:
test:
uat:
in app1 default.yaml:
---
classes:
app1:
app1_version: 'latest'
in app1 dev.yaml:
---
app1_version: '3.0'
If I wanted host1 and host2 to be part of dev for app1:
host1.yaml:
---
classes:
basic:
app1:
env:
dev:
maybe in host2 I want to override version too:
host2.yaml
---
classes:
basic:
app1:
env:
dev:
app1_version: '3.1'
So in short, I would like hiera to be a source of facts, where I can get
information that feeds Puppet in order to classify the nodes and to feed
the parametrised classes.
I recently found this blog entry:
http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
Gary has been very helpful and I have got an idea of what needs doing. I
can query all the data the way I want using the hiera command. Something
like these:
hiera app1_version sysclass=app1 env=dev
Returns the expected '3.0' and if I query by adding teh host:
hiera app1_version sysclass=app1 env=dev hostname=host1
I get 3.1. Cool!
Example using Gary's approach:
/opt/puppet-data/nonprod/hieratest/default.yaml
---
classes: hieratest
env: hieratest_default
/opt/puppet-data/nonprod/hieratest/host1.yaml
---
classes:
hieratest:
env: 'hieratest_performance'
# hiera env sysclass=hieratest --debug
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source
nonprod/hieratest/default
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Found env in
nonprod/hieratest/default
hieratest_default
# hiera type sysclass=hieratest hostname=host1 --debug
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source
nonprod/hieratest/host1
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Found env in nonprod/hieratest/host1
hieratest_performance
But when it comes to use this in Puppet the results are not as I expect,
nothing happens, it just does a run no classes are used. I see that the
basic class custom facts are loaded, but nothing gets executed, as if the
catalogue for host1 would not include it.
In Puppet I expect to just have:
in site.pp:
node default {}
And then in each application’s init.pp:
$env = hiera(env) this allows me to get the right config files
(with are maintained in a git repo)
$app1_version = hiera(app1_version) this allows me to set the right
RPM version (from satellite/spacewalk/RHN)
As per Gary's post, I can use hiera as node terminus, and so it is set in
puppet.conf.
I would like to make emphasis in this: Gary's hiera as an ENC works, but
for a more simple scenario than the one I am proposing, if I only wanted to
classify
[Puppet Users] Re: Puppet 2.7, hiera 1.0 and hiera as an ENC
On Wednesday, September 26, 2012 10:52:19 AM UTC-5, Guillem Liarte wrote:
This is the situation I have:
All my hosts are the* same OS.*
All my host are in the* same puppet environment,* so I cannot use
%{environment}
I have a module that sets all the *basic* functionality for the OS,
resolution, authentication, security, packages, etc
I have a module for each application hosted.
At the moment all the 'data' is in Puppet, mostly in parametrised classes
in site.pp.
What I want to get is a hiera set-up that allows me to use this structure:
I suspect that one of the reasons you are having trouble is that you are
trying to employ a usage paradigm that is inconsistent with hiera's design
(more below).
:hierarchy:
- global # source of application names (classes? modules?) and
environments list
- basic # data for the basic class
There's nothing wrong with those levels.
- prod/%{application}/%{hostname}# hostname files for specific
data
- prod/%{application}/%{env} # environmental data for
each application (module)
- prod/%{application}/default # default data for an
application
But there *is* a problem with those. It may be possible to make it work,
but it's shaky to use variable hierarchy levels for data *selection*.
That's what keys are for. With that said, recent Puppet releases provide
automatic $calling_module and $calling_class variables, one of which you
could probably use in place of $application. As I understand it, that's
intended to provide (better) support for module-specific data, which might
be a good way to cast that part of your problem.
- nonprod/%{sysclass}/%{hostname}
- nonprod/%{sysclass}/%{env}
- nonprod/%{sysclass}/default
You additionally have a fundamental problem with %{env}. Hiera will
attempt to resolve that as a *Puppet* variable, to which the presence of a
matching key somewhere in the Hiera hierarchy is irrelevant. Hiera needs
to know the value to resolve the hierarchy (as you have defined it), and it
would need, in principle, to resolve the hierarchy before it could look up
the value in your data store.
What actually happens, I'm sure, is that hiera uses the value of $::env
that it looks up in Puppet at function entry. You might be able to work
around that by setting that variable in Puppet before looking up other
data, such as by putting
$env = hiera('env')
at top scope near the beginning of your site.pp.
So in short, I would like hiera to be a source of facts, where I can get
information that feeds Puppet in order to classify the nodes and to feed
the parametrised classes.
As an aside, throwing parametrized classes into this mix has only downside
as far as I am concerned, except inasmuch as you may want to use
parametrized classes that are (unwisely) provided by modules written by
others. Since you want to rely on hiera (which is good), it is superior to
write your classes like this wherever you are in control of module
interfaces:
class mymodule::class1 {
$param1 = hiera('mymodule::class1::param1')
$param2 = hiera('mymodule::class1::param2')
# or with simpler keys enabled by use of
# %{calling_module} and/or %{calling_class}
}
There are several advantages, among them that you can encode interclass
parse-order dependencies via the built-in 'include' function, and that you
can use hiera's 'hiera_include()' function to assign such classes to nodes.
I recently found this blog entry:
http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
Gary appears to have done some cool work there, but as you have discovered,
it's not going to overcome the inherent problem with self-referrential
data. It might be possible to work around this by augmenting Gary's hiera
additions/modifications with a separate pre-lookup of needed extra
variables, but you're then talking about a distinctly non-trivial effort
and a substantial branch away from stock hiera.
I would like to make emphasis in this: Gary's hiera as an ENC works, but
for a more simple scenario than the one I am proposing, if I only wanted to
classify Classes and Hosts, it does work fine. Where I have not been able
to succeed is in adding an 'env' layer after the application (classes,
organised in modules).
You are classifying based only on hostname, because that's the only data
you want to consider that actually originates from the node being
classified. Everything else is logic and structure of the ENC you are
trying to build.
The problem is that you are trying to implement a data structure that Hiera
does not natively support. You can make it work, but you will need either
significant changes in hiera, or a different usage mode. I would suggest
the latter.
Were I you, I would consider writing a separate, hiera-based ENC instead of
trying to build all the ENC features you want directly into hiera itself.
Among other
Re: [Puppet Users] Re: Puppet 2.7, hiera 1.0 and hiera as an ENC
Guillem,
Sorry for the delayed response. Anyway, John is correct about what you're
trying to do with Hiera. I can say as far as the ENC Gary has written follows
all the rules of what an ENC 'should do' as per the documentation.
http://docs.puppetlabs.com/guides/external_nodes.html
Keep in mind also, when the ENC is run, the only information it has access to
is the Facts from the node. The manifests are not compiled until after the ENC
returns the classes it has determined should be declared. This ENC does it in
one call to hiera for classes. When this Hiera ENC processes, it never adds the
discovered parameters and other variables to the current running context. While
that could achieve what you want, it could also complicate things in unexpected
ways. So as I said above, when the ENC runs, you only have just the facts from
the host as your current context.
John's suggestion of a new usage model that aligns better with Hiera's design
is probably the best answer. That being a case, I would suggest custom facts to
help you navigate your hiera tree in a more controlled/granular manner.
--
Tom Linkin
Professional Services Engineer
http://puppetlabs.com/
twitter: @trlinkin
On Monday, October 1, 2012 at 10:47 AM, jcbollinger wrote:
On Wednesday, September 26, 2012 10:52:19 AM UTC-5, Guillem Liarte wrote:
This is the situation I have:
All my hosts are the same OS.
All my host are in the same puppet environment, so I cannot use
%{environment}
I have a module that sets all the basic functionality for the OS,
resolution, authentication, security, packages, etc
I have a module for each application hosted.
At the moment all the 'data' is in Puppet, mostly in parametrised classes
in site.pp.
What I want to get is a hiera set-up that allows me to use this structure:
I suspect that one of the reasons you are having trouble is that you are
trying to employ a usage paradigm that is inconsistent with hiera's design
(more below).
:hierarchy:
- global # source of application names (classes? modules?) and
environments list
- basic # data for the basic class
There's nothing wrong with those levels.
- prod/%{application}/%{hostname}# hostname files for specific
data
- prod/%{application}/%{env} # environmental data for
each application (module)
- prod/%{application}/default # default data for an
application
But there is a problem with those. It may be possible to make it work, but
it's shaky to use variable hierarchy levels for data selection. That's what
keys are for. With that said, recent Puppet releases provide automatic
$calling_module and $calling_class variables, one of which you could probably
use in place of $application. As I understand it, that's intended to provide
(better) support for module-specific data, which might be a good way to cast
that part of your problem.
- nonprod/%{sysclass}/%{hostname}
- nonprod/%{sysclass}/%{env}
- nonprod/%{sysclass}/default
You additionally have a fundamental problem with %{env}. Hiera will attempt
to resolve that as a Puppet variable, to which the presence of a matching key
somewhere in the Hiera hierarchy is irrelevant. Hiera needs to know the
value to resolve the hierarchy (as you have defined it), and it would need,
in principle, to resolve the hierarchy before it could look up the value in
your data store.
What actually happens, I'm sure, is that hiera uses the value of $::env that
it looks up in Puppet at function entry. You might be able to work around
that by setting that variable in Puppet before looking up other data, such as
by putting
$env = hiera('env')
at top scope near the beginning of your site.pp.
So in short, I would like hiera to be a source of facts, where I can get
information that feeds Puppet in order to classify the nodes and to feed
the parametrised classes.
As an aside, throwing parametrized classes into this mix has only downside as
far as I am concerned, except inasmuch as you may want to use parametrized
classes that are (unwisely) provided by modules written by others. Since you
want to rely on hiera (which is good), it is superior to write your classes
like this wherever you are in control of module interfaces:
class mymodule::class1 {
$param1 = hiera('mymodule::class1::param1')
$param2 = hiera('mymodule::class1::param2')
# or with simpler keys enabled by use of
# %{calling_module} and/or %{calling_class}
}
There are several advantages, among them that you can encode interclass
parse-order dependencies via the built-in 'include' function, and that you
can use hiera's 'hiera_include()' function to assign such classes to nodes.
I recently found this blog entry:
http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
Gary appears to have done some cool
