Re: [Puppet Users] Puppet 3 and hiera

[Justin Stoller]

On Wed, Jan 9, 2019 at 9:06 AM Peter Berghold 
wrote:

> Hi folks,
>
> I know... I know... get off of Puppet 3.  I'll be getting there soon.
>
> Right now I have Puppet 3 in our production environment where I work.
> Long political story as to why we are still on 3 that I won't get into.   I
> just did a release of the Puppet code and one of the "features" of the
> release is moving all the data out of the code into hiera.
>
> This has been shaken out and tested thoroughly and works fine. So far so
> good.  Superficially at least both the production and lab environments
> match each other in terms of Puppet version OS etc.   Both lab and
> production use RHEL 6.5 (?) and here's where the issue comes in.
>
> Puppet servers (I have a tiered environment, grand master -> manages ->
> remote masters -> manage clients) all seem to be working fine.  On the
> client nodes the Puppet agent terminates with an error:
>
> Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error
> +RuntimeError: Hiera terminus not supported without hiera library at
> +/etc/puppet/environments/Production/manifests/site.pp:24 on node
> +
>
> I did some Googling looking for a solution and I found one mention of the
> need for a "ruby-hiera" package.  This is not installed in our test lab so
> this seems to be a red herring to me but I'm very willing to be wrong.
>

I think that package provides the library that can also be installed via
the hiera gem. You might want to see whether the hiera gem is installed in
Puppet/Puppet Server's gempath in one place vs the other (maybe someone
installed it via gem in the lab?).

>
> The line of code the error is happening on cited above is simple
> hiera_include('classes');
>
> Thoughts anybody?
> --
>
> Peter L. Berghold   salty.cowd...@gmail.com
>
> h ttp://science-fiction.berghold.net
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAArvnv281q9BHN5eY22MCcS_rtQ4utkmfYffY15b0cRtuUCg_A%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXWRjRag3KkCsGDdVAYuMP73hPa%2BKy4tTyaB6hNszOn-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Puppet 3 and hiera

[Peter Berghold]

Hi folks,

I know... I know... get off of Puppet 3.  I'll be getting there soon.

Right now I have Puppet 3 in our production environment where I work.  Long
political story as to why we are still on 3 that I won't get into.   I just
did a release of the Puppet code and one of the "features" of the release
is moving all the data out of the code into hiera.

This has been shaken out and tested thoroughly and works fine. So far so
good.  Superficially at least both the production and lab environments
match each other in terms of Puppet version OS etc.   Both lab and
production use RHEL 6.5 (?) and here's where the issue comes in.

Puppet servers (I have a tiered environment, grand master -> manages ->
remote masters -> manage clients) all seem to be working fine.  On the
client nodes the Puppet agent terminates with an error:

Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error
+RuntimeError: Hiera terminus not supported without hiera library at
+/etc/puppet/environments/Production/manifests/site.pp:24 on node
+

I did some Googling looking for a solution and I found one mention of the
need for a "ruby-hiera" package.  This is not installed in our test lab so
this seems to be a red herring to me but I'm very willing to be wrong.

The line of code the error is happening on cited above is simple
hiera_include('classes');

Thoughts anybody?
-- 

Peter L. Berghold   salty.cowd...@gmail.com

h ttp://science-fiction.berghold.net

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAArvnv281q9BHN5eY22MCcS_rtQ4utkmfYffY15b0cRtuUCg_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] A GUI tool for Hiera

[Karsten Heymann]

Am Mi., 9. Jan. 2019 um 12:48 Uhr schrieb desertkun :

>
> ** Have you thoght about integrating support for hiera-enc?*
>
>
>
> Wasn’t aware of it, so I need time to investigate what it does.
>
>
It allows to encrypt entries (value) in hiera so that they don't end up in
clear text for example in the git history of the puppet code.


> ** Some kind of git commit function for the changes done would be a neat
> feature. This would certain parts of our coworkers to stay completely in
> this kind of editor as they probably only interact with hiera data. If you
> are interested I have some ideas how this could be implemented in a safe
> and convenient way.*
>
>
>
> Yes, that is definitely planned. I want it to the level the editor would
> highlight classes/yaml files/directories with green/blue/red colors to
> indicate staging status. Obviously, with commit/discard buttons. I also
> want to implement actual module installation (assuming everyone just adds
> them as submodules to their repo).
>

I would think that nowadays using a Puppetfile is the preferred way to
install modules.

** Is there support for changing a value at every depth of the configured
> hiera hierachy?*
>
>
>
> Not quite sure about that, it can edit classes (their fields) and
> resources (and their fields), would appreciate an example of what you mean.
>

The way we use hiera is that we have the following hierachy definded in
hiera.yaml (this is still hiera 3, but the principle stays the same with
hiera 5+):

hiera.yaml:
:hierarchy:
  - "hostname/%{fqdn}"
  - "roles_%{trusted.extensions.pp_apptier}/%{trusted.extensions.pp_role}"
  - "roles_%{trusted.extensions.pp_apptier}/default"
  - "roles_common/%{trusted.extensions.pp_role}"
  - common

The hiera data folder in our environments looks this way:
$ tree -d
.
├── hostname
├── roles_common
├── roles_labor
├── roles_prod
└── roles_staging

With yaml files in every folder and a common.yaml in the main folder. Now
when in the puppet code a variable is looked up, the hierachy list above is
evaluated and checked on which level a variable is set.

Example: A webserver with the hostname web-01.mydomain, the apptier staging
and the role web uses the class profiles::webserver and in there looks up
the variable profiles::webserver::servername.  Now when compiling the
catalog the following hiera files are sequentially checked if they contain
a "profiles::webserver::servername: whatever" variable:

hostname/web-01.mydomain.yaml
roles_staging/web.yaml
roles_staging/default.yaml
roles_common/web.yaml
common.yaml

Depending if I want to set a variable only for a specific server, for all
staging web server, for all staging servers, for all prod/lab/staging web
servers or for every system known, the variable can be edited on any of
these levels. This is the main benefit of using hiera, you can define a
variable as general and as specific as needed without repeating yourself.
The main feature I would expect from a hiera editor would be to make this
layered setup as approachable and transparent as possible,


> ** Are you aware of jerakia (http://jerakia.io/ ). We
> don't use it yet but are considering it to be able to use our "hiera" data
> from ansible too. But I think it's a very interesting project.*
>
>
>
> I am not but I am pretty sure we have different goals. I want make a tool
> for those who have no idea what Puppet is but need to use it.
>

This wasn't ment as a competing project, more as a hint to a possible
alternative backend that could be supported additionally to hiera.


> ** Is the hieraresources part optional? We don't want to use hiera to
> define arbitrary resources as it would work around the way we define roles
> and profiles.*
>
>
>
> It is, you can just do hiera_include('classes') instead if you only need
> to manage classes. But as long as you won’t create any resource in the tool
> it won’t populate the `resources` hash and thus the second line
> hiera_resources('resources')  of the hieraresources module won’t do
> anything. I implemented this because there is a case when you actually need
> your instances of profiles several times.
>

I think we use hiera in a different way. We don't load any classes from
hiera at all, instead we use an enc script for this. So the only purpose
hiera does serve us is to set the variables for the profiles loaded by the
roles our servers use. Every server "automatically" loads
roles.pp and in this file all the server's profiles are
included ('contain'd to be precise). But I understand that loading classes
not from hiera makes it much more difficult to decide which variables are
used by a server.

This all isn't ment to say that our solution is any better than what you
seem to propose, but just as a heads up that I think it's quite important
that you document the assumptions you have about how a user uses
puppet/hiera in order to be able to use your tool.

Best regards,
Karsten

-- 
You received this message because you are 

RE: [Puppet Users] A GUI tool for Hiera

[desertkun]

especially for our coworkers that aren't that deep into puppet and for which 
unerstanding hiera regularly is the most complicated part of the setup. 

Thanks for the feedback, that is exactly the goal of the project. It makes me 
happy to know that someone sees potential in this.

* Have you thoght about integrating support for hiera-enc?

Wasn’t aware of it, so I need time to investigate what it does.

* Some kind of git commit function for the changes done would be a neat 
feature. This would certain parts of our coworkers to stay completely in this 
kind of editor as they probably only interact with hiera data. If you are 
interested I have some ideas how this could be implemented in a safe and 
convenient way.

Yes, that is definitely planned. I want it to the level the editor would 
highlight classes/yaml files/directories with green/blue/red colors to indicate 
staging status. Obviously, with commit/discard buttons. I also want to 
implement actual module installation (assuming everyone just adds them as 
submodules to their repo).

* Is there support for changing a value at every depth of the configured hiera 
hierachy?

Not quite sure about that, it can edit classes (their fields) and resources 
(and their fields), would appreciate an example of what you mean.

* Does this include module-level data directories?

Not as of yet.

* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are 
considering it to be able to use our "hiera" data from ansible too. But I think 
it's a very interesting project.

I am not but I am pretty sure we have different goals. I want make a tool for 
those who have no idea what Puppet is but need to use it.

* Is the hieraresources part optional? We don't want to use hiera to define 
arbitrary resources as it would work around the way we define roles and 
profiles.

It is, you can just do hiera_include('classes') instead if you only need to 
manage classes. But as long as you won’t create any resource in the tool it 
won’t populate the `resources` hash and thus the second line 
hiera_resources('resources')  of the hieraresources module won’t do anything. I 
implemented this because there is a case when you actually need your instances 
of profiles several times.

From: Karsten Heymann
Sent: Wednesday, January 9, 2019 13:25
To: puppet-users@googlegroups.com
Subject: Re: [Puppet Users] A GUI tool for Hiera

Hi Alex,

that sounds definitely very interesting, especially for our coworkers that 
aren't that deep into puppet and for which unerstanding hiera regularly is the 
most complicated part of the setup. Some questions and remarks:

* Have you thoght about integrating support for hiera-enc?
* Some kind of git commit function for the changes done would be a neat 
feature. This would certain parts of our coworkers to stay completely in this 
kind of editor as they probably only interact with hiera data. If you are 
interested I have some ideas how this could be implemented in a safe and 
convenient way.
* Is there support for changing a value at every depth of the configured hiera 
hierachy?
* Does this include module-level data directories?
* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are 
considering it to be able to use our "hiera" data from ansible too. But I think 
it's a very interesting project.
* Is the hieraresources part optional? We don't want to use hiera to define 
arbitrary resources as it would work around the way we define roles and 
profiles.

So thank you for starting this, we are definitely interested in what progress 
you will make.

Best regards
Karsten

Am Di., 8. Jan. 2019 um 18:57 Uhr schrieb desertkun :
Hello, everyone.

I have made a small useful open source project for Puppet/Hiera, so I hope you 
can excuse me a bit of advertisement of it for greater good.

https://github.com/desertkun/hiera-editor

Basically it takes editing Hiera configurations to a new level. 
It parses modules with puppet-strings to extract class information like field 
names, types and doc strings, and retrieves default values of class fields 
by doing best-effort compilation (with puppet-parser) of Puppet AST on your 
machine. So no more typos and less of "commit-deploy-check" cycles.



The goal of the project is to help manage servers with Puppet to those who far 
away from the back-end, including Puppet itself, 
like "I need to deploy nglinx but I have installed debian for the first time". 
So if you have a project that complex that requires Puppet to deploy it, having 
some
tool to introduce Puppet to end users of your project might improve the 
learning curve.

Would really appreciate any input on the idea, including concerns like "there's 
no need for this" as I just have made the project public and still not sure if 
I should continue.

Regards.
-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an 

Re: [Puppet Users] A GUI tool for Hiera

[Karsten Heymann]

Hi Alex,

that sounds definitely very interesting, especially for our coworkers that
aren't that deep into puppet and for which unerstanding hiera regularly is
the most complicated part of the setup. Some questions and remarks:

* Have you thoght about integrating support for hiera-enc?
* Some kind of git commit function for the changes done would be a neat
feature. This would certain parts of our coworkers to stay completely in
this kind of editor as they probably only interact with hiera data. If you
are interested I have some ideas how this could be implemented in a safe
and convenient way.
* Is there support for changing a value at every depth of the configured
hiera hierachy?
* Does this include module-level data directories?
* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but
are considering it to be able to use our "hiera" data from ansible too. But
I think it's a very interesting project.
* Is the hieraresources part optional? We don't want to use hiera to define
arbitrary resources as it would work around the way we define roles and
profiles.

So thank you for starting this, we are definitely interested in what
progress you will make.

Best regards
Karsten

Am Di., 8. Jan. 2019 um 18:57 Uhr schrieb desertkun :

> Hello, everyone.
>
> I have made a small useful open source project for Puppet/Hiera, so I hope
> you can excuse me a bit of advertisement of it for greater good.
>
> https://github.com/desertkun/hiera-editor
>
> Basically it takes editing Hiera configurations to a new level.
> It parses modules with puppet-strings to extract class information like
> field names, types and doc strings, and retrieves default values of class
> fields
> by doing best-effort compilation (with puppet-parser) of Puppet AST on
> your machine. So no more typos and less of "commit-deploy-check" cycles.
>
>
>
> The goal of the project is to help manage servers with Puppet to those who
> far away from the back-end, including Puppet itself,
> like "I need to deploy nglinx but I have installed debian for the first
> time". So if you have a project that complex that requires Puppet to deploy
> it, having some
> tool to introduce Puppet to end users of your project might improve the
> learning curve.
>
> Would really appreciate any input on the idea, including concerns like
> "there's no need for this" as I just have made the project public and still
> not sure if I should continue.
>
> Regards.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/1eb43a86-8f65-42d6-910f-9d45a8789256%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAL017hCAj_XsOfG%3D7aaJAV56JbBXm96U1p32DqCXW1GXK--Z0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.