Re: [Puppet Users] Re: Puppet 6 client lookup secret from Hashicorp Vault
We now have Vault lookups working using the Deferred functionality that
debuted in Puppet6x.
Here are my high level note on "how", hoping they help someone else in the
future (lots of assumptions, but if you have questions please reach out...)
#export VAULT_SKIP_VERIFY=true
#mkdir /var/lib/vault
#chown puppet:puppet /var/lib/vault
#vault operator init -key-shares=1 -key-threshold=1
# (securely record keys and tokens)
#export VAULT_TOKEN=TheTokenFromAbove
#vault operator unseal
# (unseal key from above)
#vault auth enable cert
#vault write auth/cert/certs/puppetserver display_name=puppet
policies=prod,test certificate=@/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
ttl=3600
#vault kv put secret/test hello=world
#echo "path \"secret/test\" {capabilities = [\"read\",\"list\"]}" > test.hcl
#vault policy write test test.hcl
# (enable vault profile::vaulttest in Hiera & deploy, test Puppet and see
the KV output)
Our next challenge is how to isolate clients to see ONLY their own secrets
within Vault based on the incoming name and a policy to suit (or a similar
workflow that results in client isolation).
If any Vault users has any advice it would be welcome.
On Monday, November 5, 2018 at 4:56:36 PM UTC+11, comport3 wrote:
>
> Hi Lindsay and Thomas,
>
> Thanks for your documentation - I'm having some problems getting the
> client lookup to work.
>
> I have the Puppetserver CA setup in Vault, and the Vault servers Puppet
> certificate and private key configured.
>
> I have added the Puppetserver CA to the trusted roots, per:
> https://github.com/hashicorp/vault/issues/438
>
> I have configured Vault ('auth enable cert', 'vault write
> auth/cert/certs/puppetserver...') successfully (or so it seems) and Vault
> is unlocked.
>
> Now I can get a test lookup to work using this CURL command -
> curl -X GET -H "X-Vault-Token:$VAULT_TOKEN"
> https://vault1.domain.com:8200/v1/secret/test
>
> But configuring via Puppet code, I get -
> Error: Failed to apply catalog: Received 403 response code from vault at
> vault1.domain.com for secret lookup (api errors: ["1 error occurred:\n\t*
> permission denied\n\n"])
>
> Any ideas what I'm missing?
>
> On Saturday, October 13, 2018 at 2:20:02 AM UTC+11, Lindsey Smith wrote:
>>
>>
>>
>> On Wed, Oct 10, 2018 at 5:28 AM Thomas Müller
>> wrote:
>>
>>>
>>>
>>> Am Dienstag, 9. Oktober 2018 14:12:39 UTC+2 schrieb comport3:
Mentioned in the Puppet 6 release notes are the ability for a client to
lookup secret data from Vault.
Is there any more info on how to implement this?
I have done extensive work on POC environments that use Vault as a top
level in Hierarchy and mark the secrets as 'sensitive' so they do not
appear in logs and reports, but do not want to continue deploying this
methodology if it's not the way the technology is headed.
https://github.com/comport3/puppet5-hiera-vault-poc
>>>
>>> from https://puppet.com/docs/puppet/6.0/using_a_deferred_function.html
>>> :
>>>
>>> The Forge already hosts some community modules that provide integrations
with secret store, like the following:
-
Azure Key Vault: works on both the master and the server
-
Cyberark Conjur: works on the master
-
Cyberark AIM: works on the agent
-
Hashicorp Vault: works on the agent
-
AWS Secrets Manager: works on the agent
but it does not directly link the modules.
>>>
>>
>> Apologies for not updating the docs in the last couple of days. The
>> agent-side Vault integration lives here:
>> https://github.com/voxpupuli/puppet-vault_lookup (coming soon to the
>> Forge)
>>
>> See also the related blog post:
>> https://puppet.com/blog/secret-agents-man-secrets-store-integrations-puppet-6
>>
>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to puppet-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/puppet-users/c44e5f05-fefe-40d6-90d0-4471fb33a9a0%40googlegroups.com
>>>
>>>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/0e3a4a35-aec0-4f30-959e-f8555f983a9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Bolt 1.9.0 now available
Greetings! We're happy to announce the release of Bolt 1.9.0. Highlights in this release include: - Improved out-of-the-box tasks - Support for multiple PuppetDB `server_urls` - Bug fixes For more information, check out the release notes: https://puppet.com/docs/bolt/1.x/bolt_release_notes.html To try this version of Bolt, follow the installation instructions for your operating system: https://puppet.com/docs/bolt/1.x/bolt_installing.html Thanks! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2B%2ByhkyzKbQnU4e2Kjg7RQDuCSTOBL0xJZR1yfmiTOp%3D5EN%2BBw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] PuppetDB exported resources with hiera-eyaml
On 2019-01-10 10:56, Jocelyn Thode wrote: Hey, I'm trying to use exported resources where one of the parameter of the resource is a variable from hiera. This variable is retrieved using automatic lookup and is encrypted in hiera using hiera-eyaml. However when the ressource is exported insted of the clear password being exported, I get the base64 encoded version of the clear password as argument. This does not happen if I don't use a hiera-eyaml encrypted password. Any idea why ? I think that is because hiera-eyaml ends up returning ASCII-8bit encoded clear text strings which is then interpreted as potentially being binary and non UTF-8 compliant and therefore sent as a Binary (which gets encoded as Base64 text). This problem should be fixed in hiera-eyaml as it should return Strings with UTF8 encoding. This may depend on the encoding of the original yaml file that hiera-eyaml read. It is a bit difficult to check if what I suspect is true. I would write a function (or call a simple function such as "with()"), do a lookup and pass the value to the function, I would then use a debugger, set a breakpoint in the function, and check the encoding of the string given to the function. To test what hiera-eyaml does requires debugging hiera-eyaml. Some background: Before puppet 6 the default format was JSON with fallback to PSON if strings were ascii-8bit. Since puppet 6, we use "rich-data encoding" by default and handle ascii-8bit as being Binary - and by not using PSON. We did work on issues related to export to PDB from puppet and it may be that a newer puppet versions does a better job with ascii-8bit that can be converted to UTF-8 without problems. In summary, I think this should be logged as a ticket for hiera-eyaml. Feel free to ping me on that ticket if the maintainers of hiera-eyaml needs a hand with figuring things out. Best, - henrik Puppet version: 6.0.4 Puppetdb version: 6.1.0 Puppetserver version: 6.0.2 -- Visit my Blog "Puppet on the Edge" http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/q18e65%24dpn%241%40blaine.gmane.org. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: Puppet 3 and hiera
>> Is there any chance that you have *too new* an Hiera installed?
I was actually worried at first that there was a regression in terms of
what hiera was on there given the host was "patched" recently. I may have
found the problem and yes I suspected right along the issue was on the
remote master not the client. (well except for a brief moment in time at
around 6AM the installation starting at midnight... I was a bit foggy then.)
I'm deploying a "fix" and if it works I'm going to post as much here.
On Thu, Jan 10, 2019 at 9:57 AM jcbollinger
wrote:
>
>
> On Wednesday, January 9, 2019 at 11:06:54 AM UTC-6, Salty Old Cowdawg
> wrote:
>>
>> Hi folks,
>>
>> I know... I know... get off of Puppet 3. I'll be getting there soon.
>>
>> Right now I have Puppet 3 in our production environment where I work.
>> Long political story as to why we are still on 3 that I won't get into. I
>> just did a release of the Puppet code and one of the "features" of the
>> release is moving all the data out of the code into hiera.
>>
>> This has been shaken out and tested thoroughly and works fine. So far so
>> good. Superficially at least both the production and lab environments
>> match each other in terms of Puppet version OS etc. Both lab and
>> production use RHEL 6.5 (?) and here's where the issue comes in.
>>
>> Puppet servers (I have a tiered environment, grand master -> manages ->
>> remote masters -> manage clients) all seem to be working fine. On the
>> client nodes the Puppet agent terminates with an error:
>>
>> Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error
>> +RuntimeError: Hiera terminus not supported without hiera library at
>> +/etc/puppet/environments/Production/manifests/site.pp:24 on node
>> +
>>
>>
>
> This may already be clear to you, but just in case, do note that although
> this error is *reported* on the clients, it *occurs* on their master. I
> guess that's one or more of the remote masters. If you're thinking in
> terms of what is installed on the agent machines then that could be
> throwing you off. Does the Hiera command-line tool run on the relevant
> masters?
>
> I'm afraid I don't know where to find P3 docs any longer, but if I did, I
> would be looking for clues among the installation instructions. The docs
> now available for both Puppet overall and for the Hiera subsystem both
> assume that you'll get Hiera in the same package that provides Puppet, or
> at least as a dependency of that package.
>
>
>
>> I did some Googling looking for a solution and I found one mention of the
>> need for a "ruby-hiera" package. This is not installed in our test lab so
>> this seems to be a red herring to me but I'm very willing to be wrong.
>>
>
>
> Hmm. I don't find a "ruby-hiera" in Fedora's Koji system. Their Hiera
> packages (i.e. those that go into EPEL) are named simply "hiera". And
> their Puppet packages, including for Puppet 3, have that "hiera" package as
> a dependency.
>
>
> The line of code the error is happening on cited above is simple
>> hiera_include('classes');
>>
>> Thoughts anybody?
>>
>
>
> Is there any chance that you have *too new* an Hiera installed?
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/d4efba93-eadd-4171-8493-e25500e7dcc8%40googlegroups.com
>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
Peter L. Berghold salty.cowd...@gmail.com
h ttp://science-fiction.berghold.net
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAArvnv3PY8BPHcoHzMOHG3eeuxGg-fSkhTw-cq8Ji7s-JD%3Dh-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[Puppet Users] PuppetDB exported resources with hiera-eyaml
Hey, I'm trying to use exported resources where one of the parameter of the resource is a variable from hiera. This variable is retrieved using automatic lookup and is encrypted in hiera using hiera-eyaml. However when the ressource is exported insted of the clear password being exported, I get the base64 encoded version of the clear password as argument. This does not happen if I don't use a hiera-eyaml encrypted password. Any idea why ? Puppet version: 6.0.4 Puppetdb version: 6.1.0 Puppetserver version: 6.0.2 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/bb8f4aa1-d52f-417c-a296-15560491d191%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: Puppet 3 and hiera
On Wednesday, January 9, 2019 at 11:06:54 AM UTC-6, Salty Old Cowdawg wrote:
>
> Hi folks,
>
> I know... I know... get off of Puppet 3. I'll be getting there soon.
>
> Right now I have Puppet 3 in our production environment where I work.
> Long political story as to why we are still on 3 that I won't get into. I
> just did a release of the Puppet code and one of the "features" of the
> release is moving all the data out of the code into hiera.
>
> This has been shaken out and tested thoroughly and works fine. So far so
> good. Superficially at least both the production and lab environments
> match each other in terms of Puppet version OS etc. Both lab and
> production use RHEL 6.5 (?) and here's where the issue comes in.
>
> Puppet servers (I have a tiered environment, grand master -> manages ->
> remote masters -> manage clients) all seem to be working fine. On the
> client nodes the Puppet agent terminates with an error:
>
> Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error
> +RuntimeError: Hiera terminus not supported without hiera library at
> +/etc/puppet/environments/Production/manifests/site.pp:24 on node
> +
>
>
This may already be clear to you, but just in case, do note that although
this error is *reported* on the clients, it *occurs* on their master. I
guess that's one or more of the remote masters. If you're thinking in
terms of what is installed on the agent machines then that could be
throwing you off. Does the Hiera command-line tool run on the relevant
masters?
I'm afraid I don't know where to find P3 docs any longer, but if I did, I
would be looking for clues among the installation instructions. The docs
now available for both Puppet overall and for the Hiera subsystem both
assume that you'll get Hiera in the same package that provides Puppet, or
at least as a dependency of that package.
> I did some Googling looking for a solution and I found one mention of the
> need for a "ruby-hiera" package. This is not installed in our test lab so
> this seems to be a red herring to me but I'm very willing to be wrong.
>
Hmm. I don't find a "ruby-hiera" in Fedora's Koji system. Their Hiera
packages (i.e. those that go into EPEL) are named simply "hiera". And
their Puppet packages, including for Puppet 3, have that "hiera" package as
a dependency.
The line of code the error is happening on cited above is simple
> hiera_include('classes');
>
> Thoughts anybody?
>
Is there any chance that you have *too new* an Hiera installed?
John
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/d4efba93-eadd-4171-8493-e25500e7dcc8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
