from:"Ken Barber"

Re: [Puppet Users] Re: Literal URIs in templates

2017-08-29 Thread Ken Barber

The appropriate attribute in this case should be 'content' instead of
'source' ... so instead of this:

file { '/etc/openldap/ldap.conf':
ensure  => file,
source  => template('ldap/ldap.conf.erb'),
require => Class["ldap::install"],
}

Use this:

file { '/etc/openldap/ldap.conf':
ensure  => file,
content  => template('ldap/ldap.conf.erb'),
require => Class["ldap::install"],
}

See: 
https://docs.puppet.com/puppet/latest/types/file.html#file-attribute-content

ken.

On Tue, Aug 29, 2017 at 1:00 PM, Michael Burling
 wrote:
> Me, too! How did you fix the above? I'm running into the same error.
>
> On Sunday, March 23, 2014 at 7:35:16 PM UTC-5, Grant wrote:
>>
>> Scrap that. I'm an idiot.
>>
>>
>>
>> On 24 March 2014 10:29, Grant Byers  wrote:
>>>
>>> Hi,
>>>
>>> I'm having some trouble using a template to construct URIs in a target
>>> file. How can I prevent the puppet templating engine from trying to
>>> interpret the URIs?
>>>
>>> For example, I've got the following class ;
>>>
>>> class ldap::config {
>>>   $ldap_sizelimit = extlookup("ldap_sizelimit", "500")
>>>   $ldap_timelimit = extlookup("ldap_timelimit", "15")
>>>   $ldap_deref = extlookup("ldap_deref", "never")
>>>   $ldap_uri = extlookup("ldap_uri", "ldap:// ldap://
>>> ldap://")
>>>   $ldap_basedn = extlookup("")
>>>   $ldap_referrals = extlookup("ldap_referrals", "off")
>>>
>>>   file { '/etc/openldap/ldap.conf':
>>> ensure  => file,
>>> source  => template('ldap/ldap.conf.erb'),
>>> require => Class["ldap::install"],
>>>   }
>>> }
>>>
>>> And my template ;
>>>
>>> #
>>> # LDAP Defaults
>>> #
>>>
>>> # See ldap.conf(5) for details
>>> # This file should be world readable but not world writable.
>>>
>>> SIZELIMIT   <%= @ldap_sizelimit %>
>>> TIMELIMIT   <%= @ldap_timelimit %>
>>>
>>> REFERRALS   <%= @ldap_referrals %>
>>> DEREF   <%= @ldap_deref %>
>>>
>>> URI <%= @ldap_uri %>
>>> BASE<%= @ldap_basedn %>
>>>
>>> TLS_CACERTDIR   /etc/pki/tls/certs
>>>
>>>
>>> However, this barfs when I attempt to apply ;
>>>
>>> err: Failed to apply catalog: Parameter source failed on
>>> File[/etc/openldap/ldap.conf]: Could not understand source #
>>> # LDAP Defaults
>>> #
>>>
>>> # See ldap.conf(5) for details
>>> # This file should be world readable but not world writable.
>>>
>>> SIZELIMIT   500
>>> TIMELIMIT   15
>>>
>>> REFERRALS   off
>>> DEREF   never
>>>
>>> URI ldap:// ldap:// ldap://
>>> BASE
>>>
>>> TLS_CACERTDIR   /etc/pki/tls/certs
>>> : bad URI(is not URI?):
>>> %23%0A%23%20LDAP%20Defaults%0A%23%0A%0A%23%20See%20ldap.conf(5)%20for%20details%0A%23%20This%20file%20should%20be%20world%20readable%20but%20not%20world%20writable.%0A%0ASIZELIMIT%09500%0ATIMELIMIT%0915%0A%0AREFERRALS%09off%0ADEREF%09%09never%0A%0AURI%09%09ldap://%20ldap://%20ldap://%0ABASE%20%09%09%0A%0ATLS_CACERTDIR%09/etc/pki/tls/certs%0A
>>> at /etc/puppet/environments/testing/modules/ldap/manifests/config.pp:16
>>>
>>>
>>> Thanks,
>>> Grant
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/7275ae03-e70d-46d1-aaa0-15cb3a73a87c%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D11n%2Bs5njb58hHm%2B5TpPtNu2t%3DSJM7EWQg9Gfigj%3Dj0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB resources are not available

2016-05-18 Thread Ken Barber

So at a high level 

The resources are populated from successful Puppet runs that submit
their catalogs to PuppetDB. So step 1, run puppet agent -t or
whatever, if you aren't seeing something like this in your
puppetdb.log:

2016-05-18 14:12:28,855 INFO  [command-proc-3055] [p.p.command]
[898d1f2d-f96b-450f-a627-7fb90eb7d491] [replace catalog]
macbook-pro-9.lan

Then the connection between puppet master & puppetdb is not configured
correctly, these instructions cover that:

https://docs.puppet.com/puppetdb/4.0/connect_puppet_master.html

(adjust for your version of PuppetDB, which looks like its a 2.x to me)

You should also check your puppet master logs to see if there are any
errors while submitting the catalog.

ken.

On Wed, May 18, 2016 at 7:21 AM, Harish Kothuri  wrote:
> Hi,
>
> I am trying to access resources from PuppetDB API as follows, API makes a
> successful call but empty response.
>
> API call:
> curl -X GET 'http://sdin-swt-ctf-01:8080/v3/resources/'
>
> Ouput:
> [ ]
>
> Is there anything that i need to enable to populate the same?
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/ca08dcc7-f58b-462e-b9ab-6c6dd5610fe8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D--8V37as-PiLLYuV%2BLu3ZjSJypO-HdSR_pToPAirn4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Avoiding duplicate exported resource

2016-03-07 Thread Ken Barber

>> Model wise, in an ideal world, the proxied/virtual address would be a
>> 'node' of sorts, and have that entry, but if no box exists to compile
>> that catalog, well then we're just talking crazy :-).
>>
>
>
> Well no, if the proxied / virtual address is not a property specific to any
> individual node, then it is a property of the overall site configuration.
> Puppet therefore does not need to determine this from the nodes; instead, it
> needs to *apply* it to them.  As such, it ought to be recorded in the Hiera
> data repository from which Puppet is working.  If it's in the data, then it
> does not need to be communicated between nodes via exported resources.
> Rather, Puppet should draw it from the same source for all nodes that need
> it for any purpose.

I think you've missed my modelling point or perspective, I was simply
expressing that if you could do it, you would record the intended
exported resource to a virtual node that maps to the virtual address,
but this isn't possible today. Hiera isn't part of the resulting
model, its just input that creates the graph. This is academic though,
its not possible anyway.

Irrespective of this imaginary world, one could store the data in
hiera to be consumed, if one chose to - or somewhere else, it matters
little for the resulting graph.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTk-D4w96-7729kOzZhx%2B4DnkR7VrR0Uat9x8HmnFribPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Avoiding duplicate exported resource

2016-03-03 Thread Ken Barber

Exported resources won't handle any resource de-duplication. You can
get around this by simply not using that to collect data, dalen's
puppetdb-query will help with this, and in PDB 4.0 we're introducing a
function for this purpose also into core. Once you have the data, then
you can do anything with it, using Puppet's latest iterator support
this should become easier also, as you can reduce the results.

As you mention, the other option is to put your proxied/external
address test, somewhere else, in one place - like on the nagios
machine itself or something like that. Then its not a duplicate. Model
wise, it probably doesn't belong on _all_ your nodes like you say, but
then again it shouldn't belong to one of the cluster nodes either,
later on if you remove that node it all stops working.

Model wise, in an ideal world, the proxied/virtual address would be a
'node' of sorts, and have that entry, but if no box exists to compile
that catalog, well then we're just talking crazy :-).

Whatever you're solution, the problem will repeat itself if you have
other virtual addresses, I'd make sure you're happy with whatever
solution for multiple clusters, at least then you have continuity,
people will know where to go to do look into problems etc.

ken.

On Thu, Mar 3, 2016 at 5:11 PM, Daniel Urist  wrote:
> I've created a module to configure a caching nginx proxy. I am running
> several of these proxies behind a load balancer. They all proxy the same
> external address. I'd like to export a nagios host/service for monitoring
> the external address, which will then be collected on my nagios server. The
> problem is, since I have several instances of the proxy managed by puppet,
> and the exported host/service is identical on each, I end up with duplicate
> resources. I could give the resources unique names (e.g. by appending the
> proxy's hostname to the resource name), but then I end up with multiple
> identical hosts/services in nagios, which doesn't work.
>
> The puppet stdlib module has an "ensure_resource" function, but there
> doesn't seem to be a way to use this on an exported resource collector.
>
> I guess one workaround would be to set a parameter in the proxy module, for
> example "export_address" and have that default to "false", and only set it
> to "true" for one node, but that's kind of ugly since one node then needs to
> be special.
>
> Surely this isn't an uncommon use case-- what's the best way to work around
> this?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAEo6%3DKbwrCYz3ovqo-E0u7EH-Jep%2BwrxEW3FhV1v8-G%3DRr%2B80w%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmjYc-MO4mnUmKGyv9cLSg0hpP%2BtwHOgyWSLqa-i1OpFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] upgrading puppedb, wrong schema number

2016-02-24 Thread Ken Barber

Oh hey, this package debacle is totally true. Confirmed it myself for EL6:

https://gist.github.com/kbarber/b0551d9aaffe2302a8dc

I've raised a bug on this with our release team, thanks.

As far as why there is a 40? I can't come up with another logical
explanation except for, at one point 3.2.3 was installed. We didn't
add a schema migration 40 anywhere else. Unless someone else added it,
or something else added it. If you don't think it was an accidental
package update, I'd double check the integrity of your database,
rather - if the package didn't do it, then I wouldn't trust the schema
to be in a good state.

In particular, check these function indexes were created using the
proper encoding, else performance is going to be slow:

https://github.com/puppetlabs/puppetdb/blob/3.2.3/src/puppetlabs/puppetdb/scf/migrate.clj#L1492-L1507

ken.

On Wed, Feb 24, 2016 at 11:37 AM, Fabrice Bacchella
<fabrice.bacche...@orange.fr> wrote:
> puppetdb 3.2.3 is not marked as an available update.
>
> # yum install puppetdb-3.2.3
> ...
> No package puppetdb-3.2.3 available.
>
> # yum repolist -v puppetlabs-pc1
> ...
> Repo-updated : Thu Feb  4 22:15:20 2016
> Repo-pkgs: 49
>
> And indeed :
> # find ...mirrors/puppetlabs/6/PC1/x86_64 -name '*.rpm' | wc -l
> 51
>
> 2 rpm are missing, did someone forgot a createrepo ? Running it on my local 
> mirror solved the problem.
>
> No one installed puppetdb 3.2.3, I don't know where this 40 is coming from.
>
>>
>> Le 24 févr. 2016 à 11:51, Ken Barber <k...@puppetlabs.com> a écrit :
>>
>> Migration number 40 is for version 3.2.3, not version 3.2.2. Looks
>> like someone has previously installed version 3.2.3 and pointed it at
>> your database.
>>
>> See:
>>
>> https://github.com/puppetlabs/puppetdb/blob/3.2.3/src/puppetlabs/puppetdb/scf/migrate.clj#L1553
>> versus
>> https://github.com/puppetlabs/puppetdb/blob/3.2.2/src/puppetlabs/puppetdb/scf/migrate.clj#L1534
>>
>> So ... why not just use 3.2.3 anyway? Since it looks like the
>> migration has taken place, and its the latest release.
>>
>> ken.
>>
>> On Wed, Feb 24, 2016 at 10:40 AM, Fabrice Bacchella
>> <fabrice.bacche...@orange.fr> wrote:
>>> I'm running puppetdb on ScientificLinux 6.7 (a RHEL clone).
>>>
>>> I upgraded from puppetdb-3.1.0 to puppetdb-3.2.2, but know I'm getting in 
>>> the log :
>>>
>>> 2016-02-24 11:22:58,585 ERROR [p.t.internal] Error during service start!!!
>>> java.lang.IllegalStateException: Your PuppetDB database contains a schema 
>>> migration numbered 40, but this version of PuppetDB does not recognize that 
>>> version.
>>>at 
>>> puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:1601) 
>>> ~[na:na]
>>>at 
>>> puppetlabs.puppetdb.cli.services$initialize_schema.invoke(services.clj:202) 
>>> ~[na:na]
>>>at 
>>> puppetlabs.puppetdb.cli.services$init_with_db.invoke(services.clj:220) 
>>> ~[na:na]
>>>
>>> I'm using postgresql for the backend.
>>>
>>> Any hint about the migration number 40 ?
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to puppet-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/puppet-users/5FF56C65-9335-41EC-A81E-EFA27E4BA3A4%40orange.fr.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to puppet-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/puppet-users/CAE4bNTkQ7ygsrRQ3amwXv_WKBd1SAckQc%2BGUbhMunC3uysDrVQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/DA3BD2D6-15C0-4132-B42A-A44A66E1E9C8%40orange.fr.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm-rO37w-Qd7pYwcUUw0tmY0M0XOFcGZ826cC-QugSZ-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] upgrading puppedb, wrong schema number

2016-02-24 Thread Ken Barber

Migration number 40 is for version 3.2.3, not version 3.2.2. Looks
like someone has previously installed version 3.2.3 and pointed it at
your database.

See:

https://github.com/puppetlabs/puppetdb/blob/3.2.3/src/puppetlabs/puppetdb/scf/migrate.clj#L1553
versus
https://github.com/puppetlabs/puppetdb/blob/3.2.2/src/puppetlabs/puppetdb/scf/migrate.clj#L1534

So ... why not just use 3.2.3 anyway? Since it looks like the
migration has taken place, and its the latest release.

ken.

On Wed, Feb 24, 2016 at 10:40 AM, Fabrice Bacchella
 wrote:
> I'm running puppetdb on ScientificLinux 6.7 (a RHEL clone).
>
> I upgraded from puppetdb-3.1.0 to puppetdb-3.2.2, but know I'm getting in the 
> log :
>
> 2016-02-24 11:22:58,585 ERROR [p.t.internal] Error during service start!!!
> java.lang.IllegalStateException: Your PuppetDB database contains a schema 
> migration numbered 40, but this version of PuppetDB does not recognize that 
> version.
> at 
> puppetlabs.puppetdb.scf.migrate$migrate_BANG_.invoke(migrate.clj:1601) 
> ~[na:na]
> at 
> puppetlabs.puppetdb.cli.services$initialize_schema.invoke(services.clj:202) 
> ~[na:na]
> at 
> puppetlabs.puppetdb.cli.services$init_with_db.invoke(services.clj:220) 
> ~[na:na]
>
> I'm using postgresql for the backend.
>
> Any hint about the migration number 40 ?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/5FF56C65-9335-41EC-A81E-EFA27E4BA3A4%40orange.fr.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkQ7ygsrRQ3amwXv_WKBd1SAckQc%2BGUbhMunC3uysDrVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Can puppet send email specific to a host?

2016-02-05 Thread Ken Barber

Perhaps you might want to modify the tagmail reporter, to handle this
yourself. A report handler gets passed the environment, so it
shouldn't be possible to massage it into shape. FWIW, tagmail has been
removed from the latest source of Puppet, full explanation here:
https://tickets.puppetlabs.com/browse/PUP-3463

ken.

On Mon, Feb 1, 2016 at 11:08 PM,   wrote:
> Please let me know if Puppet has a feature which can send emails specific to
> an environment?
> The tagmail feature sends emails of all the agents communicating to the
> master and that is not what we want in our implementation.
>
> Please advise.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/20f7b179-8f88-422c-85ad-87ceaaa2e16b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmdToaA9Arh-kBiN9HGkgULzRqD6h6DUSu%2B_LhJtP0S9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Announce: PuppetDB 3.2.0 is available!

2015-10-29 Thread Ken Barber

PuppetDB 3.2.0 - October 29, 2015

PuppetDB 3.2.0 Downloads


Available in native package format as part of Puppet Collection 1
(PC1). More information on the PC1 repositories is available here:
http://bit.ly/1HQJDNb

Binary tarball: http://downloads.puppetlabs.com/puppetdb/

Source: http://github.com/puppetlabs/puppetdb

Please report feedback via the Puppet Labs tickets site, using an
affected PuppetDB version of 3.2.0:
https://tickets.puppetlabs.com/browse/PDB

Documentation: http://docs.puppetlabs.com/puppetdb/3.2/

Puppet module: http://forge.puppetlabs.com/puppetlabs/puppetdb

PuppetDB 3.2.0 Release Notes


PuppetDB 3.2.0 is a backward-compatible feature release that
introduces some new API fields and parameters, better UTF-8 handling,
a new experimental way of performing subqueries and many more
enhancements and bug-fixes. More information on the specifics of the
release can be found in the official release notes:
https://docs.puppetlabs.com/puppetdb/3.2/release_notes.html

Contributors
---

Andrew Roetker, Ken Barber, Nick Fagerlund, Rob Browing, Russell Mull,
Ryan Senior, Tim Skirvin, Wayne Warren and Wyatt Alt.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkN1fGTrA2%2BxzaUTkLtW%3DwhYDEY25hzbteo%3DDK1aogc-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppet + puppetdb: No Catalog received

2015-10-27 Thread Ken Barber

> I have a running puppet installation (version 3.8.3)
> I installed and configured a puppetdb node (2.3.8 with postgresql).
> Configured puppet master to user the new puppetdb node.
>
> When I run puppet agent from any of the nodes I get a 'Invalid
> relationship Class doesn't seem to be in the catalog'

This is the error we should be focusing on. Can you post the full
error in context with the surrounding agent log? Can you double check
the relationship its pointing out in the error, and check for any
upper/lower case issues and ensure the name in the relationship is
100% correct, I've seen this kind of thing before and sometimes its
down to a typo. Try removing the relationship and seeing if it solves
it at least for test purposes perhaps.

> When I run puppet node status I get:
> root@puppet:/etc/puppet# puppet node status test.local
> test.local
> Currently active
> No catalog received
> Last facts: 2015-10-27T10:40:49.778Z
>
> In the puppetdb logs I see '[replace facts]', but no '[replace catalog]'
> like mentioned in the documentation.

The first error above means the catalog does not compile correctly, so
no catalog is ever submitted to PuppetDB.

ken.

>
> any idas?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/215a44ca-79d7-4a4f-be29-ed3266e07bb8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmjUVKscdORFt%2BErF-LKE7x85_tQ5waUoL4jgykJn5ACw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB Service Won't Install/Start

2015-10-09 Thread Ken Barber

On Fri, Oct 9, 2015 at 4:35 AM, Dan  wrote:
> Hi Wyatt,
>
> Thanks for the pointer!  I found the full stack trace which gives a better
> error:
>
> I just need to workout how to configure the SSL configuration now.

Try `puppetdb ssl-setup` on the command line. It requires that you've
done a full puppet agent run first, since it re-uses Puppet's SSL
certs/key files. Otherwise, you can create new ones manually using the
`puppet cert` tooling:
https://docs.puppetlabs.com/references/4.2.0/man/cert.html. Just be
mindful that normal SSL validation rules apply here with the
certificate hostnames. So you must match the real clients hostname,
much like a browser, or the client will reject the server.

>
>
> 2015-10-09 20:28:24,184 INFO  [p.p.pdb-routing] Starting PuppetDB, entering
> maintenance mode
> 2015-10-09 20:28:24,257 INFO  [p.t.s.n.nrepl-service] nREPL service
> disabled, not starting
> 2015-10-09 20:28:24,258 INFO  [p.t.s.w.jetty9-service] Starting web
> server(s).
> 2015-10-09 20:28:24,293 ERROR [p.t.internal] Error during service start!!!
> java.lang.IllegalArgumentException: Missing some SSL configuration; must
> provide either :ssl-cert, :ssl-key, and :ssl-ca-cert, OR :truststore,
> :trust-password, :keystor
> e, and :key-password.
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28763$get_jks_keystore_config_BANG___28764$fn__28768.invoke(jetty9_config.clj:288)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28763$get_jks_keystore_config_BANG___28764.invoke(jetty9_config.clj:282)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28787$get_keystore_config_BANG___28788$fn__28789.invoke(jetty9_config.clj:307)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28787$get_keystore_config_BANG___28788.invoke(jetty9_config.clj:300)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28963$maybe_get_https_connector__28964$fn__28965.invoke(jetty9_config.clj:399)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval28963$maybe_get_https_connector__28964.invoke(jetty9_config.clj:390)
> ~[na:na]
> at
> puppetlabs.trapperkeeper.services.webserver.jetty9_config$eval29004$maybe_add_https_connector__29005$fn__29006.invoke(jetty9_config.clj
>
>
> Thanks
>
> Dan
>
> On Friday, 9 October 2015 01:18:08 UTC+1, Wyatt Alt wrote:
>>
>> Hey Dan,
>>
>> I see the bottom of a java stacktrace in your log snippet there -- could
>> you get the full stacktrace from journalctl and stick it in a gist?
>>
>> Wyatt
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/03974e18-839d-4aab-bddc-f4c42a9928c7%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmdD6eUKrD34-4dP16ybhuZLO0Qw2d8NUBeut5MMb3jVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb - getting a list of specific facts for specific hosts?

2015-08-12 Thread Ken Barber

 Can you help with this query?  I am trying to get 2 facts from all of our
 puppet clients in PuppetDB.
 I tried variations of the following, but no luck: ('[or, [=, name,
 kernelversion], [=, name, instance_uuid]]')

For me this query works. Here is the full curl example in the latest
PDB (I replaced instance_uuid with operatingsystem, since I don't have
that fact and I wanted to show it working with _something_):

# curl -G 'http://localhost:8080/pdb/query/v4/facts' --data-urlencode
'query=[or,[=,name,kernelversion],[=,name,operatingsystem]]'
[ {
  certname : kb.local,
  environment : production,
  name : operatingsystem,
  value : Darwin
}, {
  certname : kb.local,
  environment : production,
  name : kernelversion,
  value : 14.4.0
} ]%

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm3wBtqEvcWnMjy63U7P3T5E4kOcv6%2BHQ8vMPLL41dQqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] How to make file_line autocreate files?

2015-08-04 Thread Ken Barber

 I may try to query the PuppetDB from a parser function to get the list
 of paths on the client; I am reading the docs at the moment.

 Here is what I came up with, and it works for me. It assumes the
 PuppetDB is on localhost:8080 as seen from the Puppet master, though; I
 don't know if it would be easy to make it generic.



 # 2015-08-04 Arnaud Gomes-do-Vale
 #This function returns the list of files needed for file_line
 #resources with a given tag.

 module Puppet::Parser::Functions
   newfunction(:files_with_lines, :type = :rvalue) do |args|
 # Argument parsing and some sanity checking.
 if args.length != 1
   raise Puppet::ParseError, Must provide a single argument.
 end
 tag = args[0]
 unless tag.instance_of? String
   raise Puppet::ParseError, I want a string.
 end

 # Initialise return value.
 file_list = []

 # Query PuppetDB.
 # FIXME Do not hardcode!
 uri = URI('http://localhost:8080/v3/resources/File_line')
 params = {:query = '[=, tag, dsh_group]'}
 uri.query = URI.encode_www_form(params)
 resp = Net::HTTP.get_response(uri)

 # Grab list of files from PuppetDB response.
 if resp.is_a?(Net::HTTPSuccess)
   entries = JSON.load(resp.body)
   entries.each do |ent|
 file_list.push ent[parameters][path]
   end
 end

 return file_list.uniq
   end
 end

 # EOF

Probably worth taking a look at how todays resource collection in the
terminus performs its HTTP queries:

https://github.com/puppetlabs/puppetdb/blob/master/puppet/lib/puppet/indirector/resource/puppetdb.rb#L18-L35

If you use that action facility, it does all the right stuff for you,
reads the puppetdb.conf file, does failover if necessary, uses SSL.
Its not necessarily something that is a public API per se, but worth
considering over doing your own HTTP work.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTn1CO6iL7MceaCcHZWhn3StT%2Bc%2BNbG3%2BvW_9%2BFayH%2B4Zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetserver puppetdb and storeconfigs=true

2015-07-23 Thread Ken Barber

I'm attempting a migration from a PuppetDB 2.x and rack Puppet 3.8.1
install over to the all new 'pc1' puppetserver puppet-agent PuppetDB v3
stack[0].

On the two nodes I've tried so far (the master itself and a test node)
I'm getting the following error:

Error 400 on SERVER: Attempt to assign to a reserved variable name: 'trusted'
on node node

This is broadly the same error as PDB-949[1] and at least one other user
[2] has had this issue with the same setup. If I disable storeconfigs
then my manifests run with both agents just fine. With storeconfigs
enabled (and even if the environment is completely empty) this is what
puppetserver logs

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in
`process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppet-server-lib/puppet/server/master.rb:39:in
`handleRequest'
Puppet$$Server$$Master_35370687.gen:13:in `handleRequest'
request_handler_core.clj:274:in `invoke'
request_handler_service.clj:14:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:626:in `invoke'
core.clj:2468:in `doInvoke'
master_core.clj:47:in `invoke'
ring.clj:22:in `invoke'
ring.clj:13:in `invoke'
comidi.clj:267:in `invoke'
ringutils.clj:106:in `invoke'
ringutils.clj:62:in `invoke'
ringutils.clj:68:in `invoke'
ringutils.clj:118:in `invoke'
jetty9_core.clj:408:in `invoke'
2015-07-23 17:09:45,728 ERROR [puppet-server] Puppet Attempt to assign to a
reserved variable name: 'trusted' on node node
2015-07-23 17:09:45,737 ERROR [puppet-server] Puppet Attempt to assign to a
reserved variable name: 'trusted' on node node

PuppetDB logs only successful updating of facts and the report:

2015-07-23 17:09:45,224 INFO [p.p.command]
[64604053-f1cb-4588-8c09-e7fc95c6b348] [replace facts] node
2015-07-23 17:09:45,833 INFO [p.p.command]
[4f7df7c3-715a-4b97-8eff-917329667a9f] [store report] puppet v4.2.1 - node

The above URLs both allude to trusted facts clobbering other facts
and/or timing issues. I definitely don't have timing issues (given one
of the agents is colocated with its master).

I've completely purge all of the old Puppet installations and even
dropped the Postgres DB so I think my stack reall should work.

Does anyone have any pointers as to how I should debug this further?

The problem is, that if you're contacting PuppetDB in this case for
your fact information, something is wrong. The 'bug' is that if PDB is
referenced for this information it tries to clobber trusted_facts, but
you should never get this far under a compilation workflow.

This can sometimes be caused by a timing issue, due to the cache
looking like its expired (even though it was just written) so it
reaches out to the PDB terminus for its answer.

The other cause can be a misconfigured routes.yaml, or some other
workflow problem. What does your routes.yaml look like? This can be
misconfigured to always use PDB for that information, which is bad.
Facts should come from agents, not from a cache inside PDB during
compilation :-).

Also - double check your facts yaml cache, wipe it if necessary (or
back it up just in case) to ensure there is nothing stale. This is
usually not necessary and probably a bit brute force, but worth a
check if you're game.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D7V-n%2BNfSKwu365kbePxTYZX3AG%2BBdeVxWNMa8TTwzFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Package versions in PuppetDB

2015-07-17 Thread Ken Barber

I can run 'puppet resource package' on a node to get a list of installed
packages and version numbers.

are those version numbers available through a PuppetDB(2.3) API query?

We don't currently store the version of pre-existing/unmanaged
resources. If you define a version explicitly, that gets compiled and
stored, but otherwise no.

how about the versions of managed Packages?

We only store, catalogs, facts reports.

Inside the catalog, it's going to look similar to how the resource was
typed out in your manifests, give or take.

So if you do this:

package { foo:
ensure = 2.1.4,
}

Yep, that ensure field gets stored. But only if it's defined like
that. So the answer is 'sort of' probably not what you want
generally. Most people just have ensure = installed or ensure =
latest or the like, which isn't a version so its useless.

The best way to think of it, is that today for catalogs we store the
intention, not the reality. Only reports come close to expressing
reality around resource state ...

Now in the reports, we do store perhaps if a field changed from X to
Y, but that's an event, a change - so also perhaps not what you want,
as it would be horrible to sort through and not-efficient etc.

An alternative is to have this data be collected in a fact. If you
want all the package data, a well structured fact is suitable for
this. That will then get submitted each run, and can be queried via
the various facts endpoints.

Ok, that sounds doable. I just turned up this sample code ,
http://www.uberobert.com/puppet-facts-with-package-versions/
and I'll season to taste for Ubuntu, but looks like it'll do the trick.

Thanks,Ken

No problemo.

FWIW ... Storing the 'unmanaged state', which we'd like to do one day,
is like storing the output from 'puppet resource package' (and any
other resource type you can think of, and want to cram in). Also known
as the RAL state, in some esoteric circles :-). This would be injected
and stored most probably in the report, or nearby the report in the
lifecycle - but this is all conjecture :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DjKtQOq94qDaO6_0sOtcKdPvsr2RaN27jpw0dkHJstfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Package versions in PuppetDB

2015-07-17 Thread Ken Barber

 I can run 'puppet resource package' on a node to get a list of installed
 packages and version numbers.

 are those version numbers available through a PuppetDB(2.3) API query?

We don't currently store the version of pre-existing/unmanaged
resources. If you define a version explicitly, that gets compiled and
stored, but otherwise no.

This is something we are considering for future work however.

 the  http://localhost:8080/v3/catalogs/${NODE}
 call shows the desired end-state

 type : Package,
 title : libc6,
 parameters : {
ensure : latest
 },

 and
 curl -G 'http://localhost:8080/v3/events' --data-urlencode 'query=[and,
 [=, resource-type,Package], [=,resource-title,libc6]]'

 will show what's changed or failed with libc6 .

 can I get the current package version number through an API call?

An alternative is to have this data be collected in a fact. If you
want all the package data, a well structured fact is suitable for
this. That will then get submitted each run, and can be queried via
the various facts endpoints.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkzsBKhWdfr95w_PCJ%3Dre3%3DJFggy6RwwJ1u4_a0q7RxOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Generate an array from PuppetDB

2015-07-16 Thread Ken Barber

Hello list.

I'm trying to use PuppetDB to link a webserver cluster to an EJB cluster.
Each webserver host need to reference a comma separated list of IPs provided
by each EJB host

Exporting and collecting resources works pretty nice but I think this isn't
the better approach. I just need a variable with a hash or an array to
iterate within an ERB template in order to build my list.

I tried the generate() function approach, using curl to request a JSON
array. Iterating such array I can populate my template. But this approach
sounds more a workaround instead a solution.

Do you have another ideas to this scenario?

Thanks, JM

Joao,

Take a look at this library, authored by my friend Eric:

https://github.com/dalen/puppet-puppetdbquery

It provides Puppet functions for querying PDB, its syntax is a wrapper
around our own PDB one, but should provide you with the tools to do
queries against PDB in a more arbitrary way hopefully avoiding the
need for your own curl/shell methodology. You might still need to
manipulate the data you receive using other tools, but it should be a
good start for what you want. If you get stuck further down the line,
respond to this thread.

We have plans to solidify this story, and make it closer to our core
offerings but I'm wary to get into a vapourware discussion until we
have something to show. For now, what I've provided is where you want
to look.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkmEHdTw%2BdR7myJYJcAt%2B3ukkfimW8V%2BRKmdJOkWvQbdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Possible to prevent submission to PuppetDB on certain runs

2015-07-12 Thread Ken Barber

 I have the need to run puppet in a 'stripped down' state, essentially
 turning off a large portion of an orchestration module.  The problem occurs
 when puppetdb receives a catalog that does not have any of the exported
 resources that the orchestration module would generate.  This obviously
 wreaks havoc on the system.

 I need a way to do puppet runs with storeconfigs turned off or otherwise
 gimped.

 I've tried modifying the config.ru adding

 ARGV  --storeconfigs   false

 but this does not seem to work as I thought it ought to.

 I tried to follow
 https://docs.puppetlabs.com/puppet/latest/reference/config_about_settings.html#settings-can-be-set-on-the-command-line

Hmm, why focus on command line args for this? You can just disable it
in your puppet.conf:

http://docs.puppetlabs.com/puppetdb/3.0/connect_puppet_master.html#edit-puppetconf

ie. storeconfigs = false in the [master] section.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnmH_fqS5Mwh%3DGT6G7qOnmj8RFrqQbxT1vzR9c2%2B1Lxmw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB doesn't work in Puppet 4: Error Executing http request

2015-06-09 Thread Ken Barber

I am trying to use PuppetDB with a Puppet 4 server that I am testing. I have
set it up as per the official docs but now I get this error when trying to
do a Puppet run (it worked before adding PuppetDB):

# puppet agent -t --noop
Warning: Unable to fetch my node definition, but the agent run will
continue:
Warning: Error 400 on SERVER: Could not retrieve facts for
ip-172-30-2-43.eu-west-1.compute.internal: Failed to find facts from
PuppetDB at puppetdb.solutions.exmaple.co.uk:8081: Error executing http
request
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Failed to submit 'replace facts' command for
ip-172-30-2-43.eu-west-1.compute.internal to PuppetDB at
puppetdb.solutions.example.co.uk:8081: Error executing http request
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

The error is a little generic, do you have the error and stack trace
from the master process, so we can see what line of code is causing
this perhaps? Unfortunately the agent error is just a mirror of the
error on the server, you need to look into the master process to dig
further - potentially even turn on debugging. The puppetdb termini
will also log its actions as well in the master log, so it would be
good to see a full run here, where it breaks and the stack trace to
figure out why.

I re-followed the instructions on a Puppet 3.8.1 master and PuppetDB worked
as expected.

DNS correctly resolves the hostname, security groups in AWS are open to
allow the connection. The only differing factor between the two setups was
the use of Puppet 4 for the one that is failing.

If you can provide the normal diagnostic info, like distro, version of
distro and exact version of all the elements (like puppetdb
puppetdb-terminus) this might be useful.

Also - what does your /etc/puppet/puppetdb.conf file look like?

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmogdjkh9SjxCA-Pc67S9doV1YLrctHo9SqXLhTRaEHCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb module fails to install or work with fresh node

2015-05-29 Thread Ken Barber

So Rob I've managed to do a successful install on a clean Ubuntu 14.04
box, you can see the full transcript from here:
https://gist.github.com/kbarber/837ff7e55e8940a7d1c8

What variations during the installation process do you think are here?

In regards to your other points yesterday:

I figured out the issue with the embedded database. For some reason letting
the system default to ::cert for the “ssl_listen_address” had it binding to
localhost instead of the actual interface it should have. Specifying
“0.0.0.0” for that option made puppetdb work just fine when using the
embedded database. It still does not add the firewall rules in though- I’m
doing that myself.

Its a silly default, I've raised a PR against master for it, should be
able to change it on the next major release 5.0.0:
https://github.com/puppetlabs/puppetlabs-puppetdb/pull/183

That being said, I did have to push through a change to postgres to allow it
to use the puppet labs apt-module, since the puppet labs team have not
managed to keep all their module dependencies in sync. You can see the pull
request I made for that here-
https://github.com/puppetlabs/puppetlabs-postgresql/pull/632/files

Are you using manage_repos or something like that in the postgresql
module to pull in the PGDG upstream repos? My setup just uses the
Ubuntu 14.04 packages, since they come with PG 9.3.

root@puppetdb01:/var/log# /usr/local/bin/validate_postgresql_connection.sh
2 10 '/usr/bin/psql --tuples-only --quiet -h localhost -U puppetdb -p
5432 --dbname puppetdb '

The result is that the script hangs while waiting for the user to put a
password in.

Yeah, you've got to pass through the password in the environment
variable like the manifest does:
https://github.com/puppetlabs/puppetlabs-postgresql/blob/master/manifests/validate_db_connection.pp#L40-L43

Like so:

root@localhost:~# PGPASSWORD=puppetdb
/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
--tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname
puppetdb '
1
1

root@localhost:~# echo $?
0

I’m using a PuppetFile with librarian-puppet.

Are you retrieving modules from git, or using forge versions like I
have? What are the particular versions of all these modules you are
using?

Also ... going back to your original error:

Error:
/Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate
puppetdb postgres connection]/Exec[validate postgres connection for
puppetdb@localhost:5432/puppetdb]/unless: Check
/usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
--tuples-only --quiet -h localhost -U puppetdb -p 5432 --dbname puppetdb
' exceeded timeout

You mention firewalling, is the port properly open? It sounds like
with your previous psql test it is, but it's odd that it's slow, which
brings me to Garrett's comment:

I put this in Hiera to extend the timeout to 30 seconds. This is often needed
when using underpowered VM's as PostgreSQL can take awhile to start.

Seem unlikely that a postgresql database, even a large one would need
30 seconds, but if you are seeing delays in starting postgresql then
it's worth a try, just seems odd for a small database, PG usually
takes less than a second to start.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnfWZZZbM%3DQYTOEBE89FGAb7FVpX6z6%2BuvvEez%3DYZmqqw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb module fails to install or work with fresh node

2015-05-28 Thread Ken Barber

 Even using the “embedded” database is apparently useless, as puppet is still
 not able to connect to puppetdb.

 In addition, puppetdb is very obviously not creating it’s firewall rules
 even though I haven’t disabled that feature.

This is interesting/surprising, but it sounds like the main pressing
issue is postgresql validation, perhaps we can focus on this to start
with, and I'll get back to this particular issue.

 Does anyone have an example of this module actually working? It’s hard to
 imagine a more simple setup than the one I have right here, but this module
 is not working or giving me any reasons why it’s not working. PuppetDB is
 just failing completely, and with all the default settings.

We've got it working for our acceptance tests for every patch, so this
is surprising as I said. Lets see if we can work out what is going on
together.

 class profiles::puppetdb {

   class { '::puppetdb':

 database_validate = false

   }

 }


 Results in:

 Error: Failed to apply catalog: Could not find dependent
 Class[Puppetdb::Server::Validate_db] for
 Class[Puppetdb::Database::Postgresql] at
 /etc/puppet/modules/puppetdb/manifests/init.pp:133

 Looking at the code in the puppetdb module I'm not even sure if the
 'database_validate = false ' option ever worked as advertised.

Yep, it looks like a bug, we're looking into it - thanks for pointing
it out, but still - it shouldn't be required.

 I'm creating a puppetdb machine using puppetdb module. My code is simple:

  class profiles::puppetdb {
class { '::puppetdb':
 
}
  }


 This, unfortunately, fails with the following error:

  Debug: Executing '/usr/local/bin/validate_postgresql_connection.sh 2 10
  '/usr/bin/psql --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432
  --dbname puppetdb ''

  Error:
  /Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate
  puppetdb postgres connection]/Exec[validate postgres connection for
  puppetdb@localhost:5432/puppetdb]/unless: Check
  /usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
  --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432  --dbname 
  puppetdb
  ' exceeded timeout


 I'm a bit at a loss as to what to do here, as there's nothing in the logs
 that can tell me why I'm getting this error.

Okay, so lets start with the basics: what is the distro, arch release
version and version of the PuppetDB module? Are you obtaining the
module via forge, git or some other mechanism? I'm asking for these
details so I can try and recreate this and see what is happening. It
shouldn't bomb this hard, and I'm wary we have a bug somewhere.

Can you also send me a full debug output run for this in a gist
(gist.github.com) from start to finish? This should be the full output
of puppet agent -t --debug for example.

You say this is on a clean machine, is this vagrant perchance? If so
is there a vagrantfile you can show me so I can reproduce? Any other
puppet manifests in the mix here that you can show us to help us
reproduce it?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm1CcN6-CD_YOrwZODjm-Lee_gdJiKxGVrUnVgV%3DA_SHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb module fails to install or work with fresh node

2015-05-28 Thread Ken Barber

Going back to postgres still results in the same failure. Since it’s related
to the validation script I thought I would run it manually to see what
happens-

root@puppetdb01:/var/log# /usr/local/bin/validate_postgresql_connection.sh
2 10 '/usr/bin/psql --tuples-only --quiet -h localhost -U puppetdb -p
5432 --dbname puppetdb '

The result is that the script hangs while waiting for the user to put a
password in.

Both machines are running Ubuntu 14.04 LTS. I am using Puppet 3.7.5 from the
PuppetLabs repository. This is on a VMWare ESXI box.

I’m using a PuppetFile with librarian-puppet. Although the puppetdb machine
is new, and is a “plank role” so to speak, it does include other modules as
there are other services which are up and running. This is the first time
I’ve used the postgres module (it’s not being used by other nodes) so there
shouldn’t be any issues with conflicts. That being said, I did have to push
through a change to postgres to allow it to use the puppet labs apt-module,
since the puppet labs team have not managed to keep all their module
dependencies in sync. You can see the pull request I made for that here-
https://github.com/puppetlabs/puppetlabs-postgresql/pull/632/files

Okay, I think this is enough to start with, thanks Rob. I'm going to
get something working now to try and reproduce, I'm in the UK though
so I may end up going to bed before I get it going and respond,
otherwise I'll pick it up again tomorrow. I suspect its because of
Ubuntu 14.04, which we don't have automated tests for yet ... and
we've got a bug we haven't noticed. At least this wouldn't surprise
me.

You definitely want to prefer PostgreSQL over embedded btw, so this is
the correct direction to follow.

Sorry about all the hassles again mate :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmjkPr9mcsXF3%2BWHbbmaPRfDC37YehCY0SRbRFxM5zkZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb module fails to install or work with fresh node

2015-05-28 Thread Ken Barber

 Even using the “embedded” database is apparently useless, as puppet is still
 not able to connect to puppetdb.

 In addition, puppetdb is very obviously not creating it’s firewall rules
 even though I haven’t disabled that feature.

 This is interesting/surprising, but it sounds like the main pressing
 issue is postgresql validation, perhaps we can focus on this to start
 with, and I'll get back to this particular issue.

 Does anyone have an example of this module actually working? It’s hard to
 imagine a more simple setup than the one I have right here, but this module
 is not working or giving me any reasons why it’s not working. PuppetDB is
 just failing completely, and with all the default settings.

 We've got it working for our acceptance tests for every patch, so this
 is surprising as I said. Lets see if we can work out what is going on
 together.

 class profiles::puppetdb {

   class { '::puppetdb':

 database_validate = false

   }

 }


 Results in:

 Error: Failed to apply catalog: Could not find dependent
 Class[Puppetdb::Server::Validate_db] for
 Class[Puppetdb::Database::Postgresql] at
 /etc/puppet/modules/puppetdb/manifests/init.pp:133

 Looking at the code in the puppetdb module I'm not even sure if the
 'database_validate = false ' option ever worked as advertised.

 Yep, it looks like a bug, we're looking into it - thanks for pointing
 it out, but still - it shouldn't be required.

FWIW: We've got a PR for this now, my colleague AJ has just raised it:
https://github.com/puppetlabs/puppetlabs-puppetdb/pull/182 you can try
this out yourself if you like, otherwise we'll be merging it once its
reviewed.

 I'm creating a puppetdb machine using puppetdb module. My code is simple:

  class profiles::puppetdb {
class { '::puppetdb':
 
}
  }


 This, unfortunately, fails with the following error:

  Debug: Executing '/usr/local/bin/validate_postgresql_connection.sh 2 10
  '/usr/bin/psql --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432
  --dbname puppetdb ''

  Error:
  /Stage[main]/Puppetdb::Server::Validate_db/Postgresql::Validate_db_connection[validate
  puppetdb postgres connection]/Exec[validate postgres connection for
  puppetdb@localhost:5432/puppetdb]/unless: Check
  /usr/local/bin/validate_postgresql_connection.sh 2 10 '/usr/bin/psql
  --tuples-only --quiet  -h localhost  -U puppetdb  -p 5432  --dbname 
  puppetdb
  ' exceeded timeout


 I'm a bit at a loss as to what to do here, as there's nothing in the logs
 that can tell me why I'm getting this error.

 Okay, so lets start with the basics: what is the distro, arch release
 version and version of the PuppetDB module? Are you obtaining the
 module via forge, git or some other mechanism? I'm asking for these
 details so I can try and recreate this and see what is happening. It
 shouldn't bomb this hard, and I'm wary we have a bug somewhere.

 Can you also send me a full debug output run for this in a gist
 (gist.github.com) from start to finish? This should be the full output
 of puppet agent -t --debug for example.

 You say this is on a clean machine, is this vagrant perchance? If so
 is there a vagrantfile you can show me so I can reproduce? Any other
 puppet manifests in the mix here that you can show us to help us
 reproduce it?

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnFOPHTArWv0yqO-JHEkZaoNB99-Az%3Dd83A%3DDnLokB6bA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Exported resource collection timing out

2015-05-28 Thread Ken Barber

 Documentation says

 configtimeout

 How long the client should wait for the configuration to be retrieved before
 considering it a failure. This setting is deprecated and has been replaced
 by http_connect_timeout and http_read_timeout. This setting can be a time
 interval in seconds (30 or 30s), minutes (30m), hours (6h), days (2d), or
 years (5y).

 Default: 2m

 I'll try tweaking http_read_timeout.

The other side of this coin is, improve the time it takes to return
the answer :-). I presume you are using PuppetDB for this? Perhaps
some performance improvements can be made on that end of the stack.
Have you looked into why the queries are taking so long to return yet?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DySMR64Xctos6iWhESVGE_b2z-aBuBt%3DiZZskWL9%2BqAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB, change how long it keeps events

2015-05-26 Thread Ken Barber

 When I make event request to PuppetDB, it seems that PuppetDB only keep 2
 weeks worth of events. How can I change how long this 2 weeks to longer
 period? Will there be any consequences other than disk space if I do that?

Change this setting and restart:

http://docs.puppetlabs.com/puppetdb/master/configure.html#report-ttl

Disk space is the main cost, however sometimes larger storage
requirements will sometimes slow down certain queries also, but YMMV.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkutnu3QzqO%3DvaOoxmDLSqP6BheQ7xsj%2B_3j-ozX%2BmTag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] [announce] The PuppetDB module master branch (5.x) gets breaking changes

2015-05-21 Thread Ken Barber

 So will bugfixes then land against the stable branch and get new bugfix
 releases there?

Absolutely, the 4.x module branch (now stable) is still alive and
while for now and will continue to take patches and release from that
branch. It doesn't even preclude minor feature releases (non breaking
ones). However, this might change once we release PuppetDB 3.0.0.

Generally speaking we don't really release bugfixes for FOSS PuppetDB
for older versions except for PE (and some major security bugs) - so
once 3.0.0 is released that will become our stable branch, and the
5.0.0 module release will be stable for that. To translate, once we
release PDB 3.0.0 we'll mostly just work on the 5.x module branch due
to our 3+ focus but as with most cases, we'll take things as they
come. If there is enough pressure to do an older 4.x releases after
that, we'll consider it, but most of our efforts should be put into
5.x of the module TBH.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmpORgAtpSV187K5RXTWsFJ3kH9CgLzpsUrzMc9ZiUhHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Upgrade puppetdb goes wrong

2015-05-14 Thread Ken Barber

 Here my setup :
 puppetdb-terminus-2.2.2-1.el6.noarch
 puppetdb-2.2.2-1.el6.noarch
 postgresql93-libs-9.3.6-1PGDG.rhel6.x86_64
 postgresql93-server-9.3.6-1PGDG.rhel6.x86_64
 postgresql93-contrib-9.3.6-1PGDG.rhel6.x86_64
 postgresql93-9.3.6-1PGDG.rhel6.x86_64
 Centos 6.6

 I try to upgrade to the latest puppetdb and puppetdb-terminus 3.3.4, but I 
 got this error:

 clojure.lang.ExceptionInfo: Input to init! does not match schema: [nil (named 
 {:ss-ca-cert disallowed-key} config)]

Check your configuration (probably /etc/puppetdb/conf.d/jetty.ini),
you are using the keyword ss-ca-cert, instead of ssl-ca-cert. Its a
typo basically.

 at 
 puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval22731$init_BANG___22732.invoke
  (jetty9_core.clj:891)
 
 puppetlabs.trapperkeeper.services.webserver.jetty9_service$reify__23105$service_fnk__17647__auto___positional$reify__23112.ini
 t (jetty9_service.clj:35)
 
 puppetlabs.trapperkeeper.services$eval17483$fn__17484$G__17475__17487.invoke 
 (services.clj:8)
 
 puppetlabs.trapperkeeper.services$eval17483$fn__17484$G__17474__17491.invoke 
 (services.clj:8)
 puppetlabs.trapperkeeper.internal$run_lifecycle_fn_BANG_.invoke 
 (internal.clj:152)
 puppetlabs.trapperkeeper.internal$run_lifecycle_fns.invoke 
 (internal.clj:180)
 puppetlabs.trapperkeeper.internal$build_app_STAR_$reify__19027.init 
 (internal.clj:444)
 puppetlabs.trapperkeeper.internal$boot_services_STAR_$fn__19039.invoke 
 (internal.clj:470)
 puppetlabs.trapperkeeper.internal$boot_services_STAR_.invoke 
 (internal.clj:469)
 puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke (core.clj:113)
 puppetlabs.trapperkeeper.core$run.invoke (core.clj:144)

 I follow the upgrade howto 
 (https://docs.puppetlabs.com/puppetdb/latest/configure.html#using-postgresql 
 ) without success.

 And I can connect to the BD:
 [root@puppetserver01 conf.d]# psql -h localhost puppetdb puppetdb
 Password for user puppetdb:
 psql (9.4.1, server 9.3.6)
 Type help for help.

 puppetdb=

 Regards,

  Dominique Arpin

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmiw1uvv_V020y6ZS_36zDdXv7OjZpkXQy2Y4dPBNEJSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb 2.2.0 on SLES 11.3: start failure Error: Could not find or load main class com.puppetlabs.puppetdb.core

2015-05-12 Thread Ken Barber

 we run puppet 3.6.2 on SLES 11 SP3 and downloaded puppetdb 2.2.0 from
 http://download.opensuse.org/repositories/systemsmanagement:/puppet:/devel/SLE_11_SP3/x86_64/puppetdb-2.2.0-14.34.x86_64.rpm
 .

 Trying to start puppetdb produces nothing more than this message Error:
 Could not find or load main class com.puppetlabs.puppetdb.core in
 puppetdb-daemon.log.

 root@DMS-P-SV1:/var/log/puppetdb
 (DMS-P-SV1)# rcpuppetdb status
 Checking for service puppetdb   dead
 root@DMS-P-SV1:/var/log/puppetdb
 (DMS-P-SV1)# rcpuppetdb start
 Starting puppetdb 8895
 done
 root@DMS-P-SV1:/var/log/puppetdb
 (DMS-P-SV1)# rcpuppetdb status
 Checking for service puppetdb   dead
 root@DMS-P-SV1:/var/log/puppetdb
 (DMS-P-SV1)# cat puppetdb-daemon.log
 Error: Could not find or load main class com.puppetlabs.puppetdb.core
 Error: Could not find or load main class com.puppetlabs.puppetdb.core
 Error: Could not find or load main class com.puppetlabs.puppetdb.core
 root@DMS-P-SV1:/var/log/puppetdb
 (DMS-P-SV1)# rpm -qa | grep puppetdb
 puppetdb-terminus-2.2.0-14.34
 python-pypuppetdb-0.1.1-1.5
 puppetdb-2.2.0-14.34


 All we could found about this error yet is a hint, that the compilation
 method AOT is no longer to be used. However, I doubt that SuSE publishes a
 version that was not correctly compiled.

Well, its not that they have compiled anything badly, it's just they
haven't adjusted their init script to reflect the changes since 2.x
when we removed AOT compilation. If you look at our version currently
in stable (2.3.4) the java arguments are different now:

https://github.com/puppetlabs/puppetdb/blob/stable/ext/templates/init_suse.erb#L36

In particular, Java launches PuppetDB using the clojure.main class,
and passes the entry point using the clojure specific argument -m.

 puppetdb 1.6.3 runs out of the box, but we would prefer to use a newer
 version.

 Does anyone have a hint how to solve this problem?

This boils down to a bug in the package I think.

You'll need to either fix this init script manually yourself, ask the
package maintainer to update their package (it's still a few revisions
behind anyhow, I wouldn't advise a 2.2.0 if you can avoid it) or roll
your own SLES specific package.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTktgdLnL%2BxgbZXqes0gOBS7FePgeUuSvfr4nRPLa%2BKgCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppet exported resources dramatic performance

2015-05-07 Thread Ken Barber

I know this has been discussed several times, but I did not find the
information/fix I’m looking for, so here I go…

I have a farm of ~370 servers.

I have a single puppet master (12 physical cores, lots of RAM) on which I
deployed puppet 3.7.5 this week (not using r10k, that’s on our huge todo
list).

I am running on Scientific Linux 6.2 (RHEL like) + Passenger + Foreman as
ENC.

More than puppet 3.7.5, I deployed the saz/ssh (2.5.0) module (required by
the openstack modules), which in turns by default enables puppet exported
resources collection for the ssh keys.

Since I was using a local hack with the generate function to achieve the
same goal, I let the ssh keys collection in place… and it worked OK until I
deployed that in production .

I then faced huge Passenger overloads, and nearly all puppet runs failed
until I raised the PassengerMaxPoolSize to 200 instead of 12 (yes…).

I have found out that enabling/disabling the SSH keys collection on a
specific host causes the compilation time to jump from 17s to 53s or even
more.

I have tried querying the puppetdb for those specific Sshkey resources using
curl and to my surprise, this was quite quick.

I am currently profiling the puppet master and will send that data to
puppetlabs if that’s still usefull to them (see :
https://puppetlabs.com/blog/tune-puppet-performance-profiler), but I would
like to know if someone would understand what’s wrong with that use of
exported resources ?

I think this direction is best, I would set up master profiling and
submit it to this thread for analysis (or take a look yourself, this
might be enough to figure it out). The PDB terminus has profiling
hooks which will also show the times it takes to perform queries as
they happen. However, like its been stated in other parts of the this
thread, it might not be the collection query at all, but an aspect of
the catalog compilation. We just can't presume until we see the
profile.

Also comparison between 'quiet' load and 'heavy' load is a good thing,
your delays in compilation could be due to lots of running threads,
keeping your profiling separate means you can compare catalog
compilation times on a single node much easier, and alleviates other
noise caused by 'general load'.

Also, if you haven't done so already - the new Clojure based puppet
server will potentially give you perf gains over the passenger
mechanism, I would definitely consider experimenting with it, if you
haven't already.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D6sp_jxJXWJ49_qPB2YqGKuvsZCUMKMLykzKg%2B9FDrEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Deprecation of HSQLDB for PuppetDB

2015-05-07 Thread Ken Barber

Hi all,

As a representative of the PuppetDB engineering team, I wanted to let
you all know we are deprecating support for HSQLDB (HyperSQL DataBase)
in the next major release of PuppetDB (version 3.0). We will drop
support in the major release after that (version 4.0).

Note: For those customers using Puppet Enterprise, this email should
not apply to you, since we have never supported HSQLDB there. However,
if you have any questions about this feel free to ask me or open a
support case.

Since PuppetDB was released in 2012 we've supported HSQLDB as an
alternative database store, for both development and smaller use
cases. The intention (and the reality) was that HSQLDB was never
really intended for middle to large production use-cases. Over the
years we've often struggled to maintain support, given its substantial
limitations as compared to PostgreSQL.

Largely this has been fine, but as time goes on and we push the
platform further, HSQLDB’s more limited capabilities have forced us to
make decisions that favor the lowest common denominator. Sometimes
this wasn't a major problem, other times it has slowed development,
and forced us to limit the features, performance, and design for
everyone.

For example:

* For queries that include child data (i.e. catalogs containing
resources) we have an excellent PostgreSQL solution using JSON
aggregation functions in 3.0, but HSQLDB simply can't support this
case efficiently, and so we are forced to perform multiple queries
instead.
* JSONB based storage offers substantial promise for our
document-style data, but since HSQLDB doesn’t support it, we haven't
been able to seriously pursue the benefits.
* Common Table Expressions could simplify and improve the performance
of some of our queries, and although HSQLDB has CTE’s, they're weak
enough that we've had to avoid using them.
* HSQLDB lacks some basic operational functions, like online backups
and online querying (for debugging purposes). Instead you must stop
the service entirely before proceeding.
* Performance tuning for HSQLDB is more difficult, since it doesn't
have a powerful query optimizer and since it’s more difficult to to
execute explain plans. We've hit various cases where we've had to
completely redesign the way we construct a query because HSQLDB’s
optimizer couldn't handle the work.

Our general opinion is that instead of compromising our overall
solution and making choices for a more ‘development’ or ‘smaller scale
only’ focused solution, we want to remove that from the equation and
to only support the more production-ready case, which is PostgreSQL
today.

What does this mean to you all? Well we'll start shipping deprecation
messages in the next major release, and the default setup will become
PostgreSQL for new installs.

For the more studious of you all, we recommend not waiting for this,
and migrating as soon as possible. However don't worry, HSQLDB will
still continue to work for the lifetime of the next major release.

For those wanting to migrate, we have supplied tooling and
documentation so that you can export your database from a HSQLDB based
system to something using PostgreSQL:

http://docs.puppetlabs.com/puppetdb/2.3/migrate.html#exporting-data-from-an-existing-puppetdb-database

For PostgreSQL setup, the PGDG team have made it super simple with
their new package repos to get the latest and greatest PostgreSQL on
most popular distributions:

https://wiki.postgresql.org/wiki/YUM_Installation
https://wiki.postgresql.org/wiki/Apt

In addition we supply a Puppet module designed for this purpose which
we highly recommend, that has had contributions from a large number of
good people over the years:

https://forge.puppetlabs.com/puppetlabs/postgresql

And for those of you who are already using our PuppetDB module,
consult the documentation on how to change your configuration to use
PostgreSQL:

https://forge.puppetlabs.com/puppetlabs/puppetdb

In any case, know that this decision hasn’t been made lightly, and
we’ve been weighing it for some time. About a year ago we had almost
45% of our reported users using HSQLDB, but this number is now less
than 20%. So while we understand that people may have legitimate
reasons for using HSQLDB, we don't feel that given the substantial
disadvantages, there’s sufficient justification to maintain support,
when doing so negatively affects the majority of users.

Of course, it goes without saying that if you have any questions or
trouble migrating to PostgreSQL, the puppet-users mailing list and the
#puppet IRC channel are watched by a number of us in the PuppetDB team
(not to mention, by other avid community users who are also helpful),
so we can help where necessary with any problems.

Regards

Ken Barber
PuppetDB Team
Puppet Labs Inc.

Re: [Puppet Users] puppetdb dies at random

2015-02-16 Thread Ken Barber

I have a 270MB puppetdb-oom.hprof.prev file in /var/log/puppetdb

This isn't unexpected behaviour per se, although it appears as such if
you haven't dealt much with Java applications. Memory usage is a hard
to predict thing, and if its too low, yes the JVM will crash itself
drop that hprof file. Not all languages have limits, they'll just
increase the usage until the operating system kills them, but Java
does have a hard limit that needs to be set up front, and if it
exceeds that limit we have it set to crash drop a hprof file in case
we need to analyze it. I don't believe in this case it needs analysis,
not yet anyway :-).

The main bug is in the fact that the daemon stdout isn't being logged
so its easier to determine its happening, at least not for Debian.
Basically Java is expressing the out of memory error via STDOUT, but
STDOUT is not being logged anywhere. We'll have this fixed in a future
release.

For now the recommendation is to increase your heap size until the
behaviour stabilises:

https://docs.puppetlabs.com/puppetdb/2.2/configure.html#configuring-the-java-heap-size

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmzR39EwU5aEhdYLGGq0j2_W0fHy93JFrGo1zdnDyc-cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb dies at random

2015-02-16 Thread Ken Barber

16850 puppetdb 20 0 12.697g 418684 14848 S 0.9 0.4 4:32.74 java

That's top now since it began running around 10.30 this morning (GMT). 12G
of ram? It's the only proc in the list having a 'g' against it. Seems
excessive..?

So, there is a difference in the columns here ... the column with the
'g' is the 'virtual' usage, that means its the amount of RAM allocated
for potential usage. Its not using all of that memory right now, but
it can under large circumstances change to use that much. I'm not
quite sure why this is so high, you'd need to show a full output of
your settings, perhaps a ps auxwww | grep java will give us the
settings that have been passed and will enable me to understand why
its so high. Either way, it's usually been set to do that by
someone/something - by default our settings shouldn't enable Java to
consume 12 GB out of the box, so I can only presume the heap setting
was changed at some point.

The column just after that is the RES column, indicating how much its
actually consuming now. This is usually the important one. I'm of
course trivialising the description of each column, but understand
virt versus res is important. There are lots of articles on the
internet about this subject that are definitely worth researching as a
sysadmin.

Another thing, if that is truly that high, you might want to check
your dmesg output to make sure the process isn't getting caught by the
OOMkiller in Linux. I have no other information about your system then
what you've given me, so I can't make a judgement on whether 12 GB is
high or not for you. It does seem high, although I could understand
this increase in the setting if you were processing a lot of data.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmhJFvS-cUg3UoUxpyCoKV8YX7WKC1fq72XwHQ5yPtnUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB 2.2.x with PostgreSQL 9.2 supported?

2015-02-16 Thread Ken Barber

In short, yes - yes 2.2.2 does support 9.2, although in the next major
release (3.x) we will be dropping that support. We are generally
telling people to utilise the PGDG set of packages to obtain the
latest PostgreSQL version:

http://yum.postgresql.org/repopackages.php
https://wiki.postgresql.org/wiki/Apt

ken.

On Mon, Feb 16, 2015 at 9:28 AM, Stefan Dietrich
stefan.dietr...@desy.de wrote:
Hi,

we are currently evaluating to update our Puppet infrastructure, one
open item is the required PostgreSQL version for PuppetDB 2.2.x.

Does PuppetDB 2.2.x still support PostgreSQL 9.2?

The upgrade docs only mention that Postgres 9.3 is recommended and pre
9.1 versions have been deprecated. No information about 9.2.

Looking at ticket PDB-769 [1] 9.2 is deprecated, but still supported?

Regards,
Stefan

[1] https://tickets.puppetlabs.com/browse/PDB-769

Stefan DietrichDeutsches Elektronen-Synchrotron (IT-Systems)
Ein Forschungszentrum der Helmholtz-Gemeinschaft
Notkestr. 85
phone: +49-40-8998-4696 22607 Hamburg
e-mail: stefan.dietr...@desy.de Germany

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/1424078884.19596.10.camel%40desy.de.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm%2Bxmbs%3DrMXbx5Oxm5vL%3DWmEvB-4z84FXE5YrU3odmNfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb dies at random

2015-02-16 Thread Ken Barber

It might be that PuppetDB is running out of heap? Check
/var/log/puppetdb for a file 'puppetdb-oom.hprof' for an indiciation
this is happening.

You can find instructions for how to adjust your heap space here:

https://docs.puppetlabs.com/puppetdb/2.2/configure.html#configuring-the-java-heap-size

ken.

On Mon, Feb 16, 2015 at 11:23 AM, James Green james.mk.gr...@gmail.com wrote:
 We have a puppet-master box with the following installed:

 root@puppet-master:/var/log/puppetdb# dpkg -l | grep puppet
 ii  facter   2.4.1-1puppetlabs1   all
 Ruby module for collecting simple facts about a host operating system
 ii  hiera1.3.4-1puppetlabs1   all
 A simple pluggable Hierarchical Database.
 ii  puppet   3.7.4-1puppetlabs1   all
 Centralized configuration management - agent startup and compatibility
 scripts
 ii  puppet-common3.7.4-1puppetlabs1   all
 Centralized configuration management
 ii  puppetdb 2.2.2-1puppetlabs1   all
 PuppetDB Centralized Storage.
 ii  puppetdb-terminus2.2.2-1puppetlabs1   all
 Connect Puppet to PuppetDB by setting up a terminus for PuppetDB.
 ii  puppetlabs-release   1.0-11   all
 Package to install Puppet Labs gpg key and apt repo
 ii  puppetmaster-common  3.7.4-1puppetlabs1   all
 Puppet master common scripts
 ii  puppetmaster-passenger   3.7.4-1puppetlabs1   all
 Centralised configuration management - master setup to run under mod
 passenger

 Occasionally puppetdb is found to no longer be running. The end of the log
 says that it is replacing facts. Then syslog shows puppet-master is unable
 to replace-facts as the connection to puppet-db is refused. The start of
 the log when we boot it again states:

 2015-02-16 10:34:28,202 INFO  [p.t.s.w.jetty9-core] Removing buggy security
 provider SunPKCS11-NSS version 1.7
 2015-02-16 10:34:28,537 INFO  [p.t.s.w.jetty9-service] Initializing web
 server.
 2015-02-16 10:34:28,604 INFO  [p.t.s.w.jetty9-service] Starting web server.
 2015-02-16 10:34:28,606 INFO  [o.e.j.s.Server] jetty-9.1.z-SNAPSHOT
 2015-02-16 10:34:28,638 INFO  [o.e.j.s.ServerConnector] Started
 ServerConnector@395ac93f{HTTP/1.1}{localhost:8080}
 2015-02-16 10:34:28,732 INFO  [o.e.j.s.ServerConnector] Started
 ServerConnector@298d54c9{SSL-HTTP/1.1}{0.0.0.0:8081}
 2015-02-16 10:34:28,787 INFO  [c.p.p.c.services] PuppetDB version 2.2.2
 2015-02-16 10:34:28,792 WARN  [c.j.b.BoneCPConfig] JDBC username was not set
 in config!
 2015-02-16 10:34:28,792 WARN  [c.j.b.BoneCPConfig] JDBC password was not set
 in config!
 2015-02-16 10:34:31,650 INFO  [c.p.p.s.migrate] There are no pending
 migrations
 2015-02-16 10:34:31,650 WARN  [c.p.p.s.migrate] Unable to install optimal
 indexing

 We are unable to create optimal indexes for your database.
 For maximum index performance, we recommend using PostgreSQL 9.3 or
 greater.

 2015-02-16 10:34:31,654 INFO  [c.p.p.c.services] Starting broker
 2015-02-16 10:34:31,899 INFO  [o.a.a.s.k.MessageDatabase] KahaDB is version
 4
 2015-02-16 10:34:31,931 INFO  [o.a.a.s.k.MessageDatabase] Recovering from
 the journal ...
 2015-02-16 10:34:31,931 INFO  [o.a.a.s.k.MessageDatabase] Recovery replayed
 2 operations from the journal in 0.026 seconds.
 2015-02-16 10:34:32,455 INFO  [c.p.p.c.services] Starting 12 command
 processor threads
 2015-02-16 10:34:32,471 WARN  [c.j.b.BoneCPConfig] JDBC username was not set
 in config!
 2015-02-16 10:34:32,473 WARN  [c.j.b.BoneCPConfig] JDBC password was not set
 in config!
 2015-02-16 10:34:32,479 INFO  [c.p.p.c.services] Starting query server
 2015-02-16 10:34:32,496 WARN  [o.e.j.s.h.ContextHandler] Empty contextPath
 2015-02-16 10:34:32,500 INFO  [o.e.j.s.h.ContextHandler] Started
 o.e.j.s.h.ContextHandler@41ec3132{/,null,AVAILABLE}
 2015-02-16 10:34:32,515 INFO  [c.p.p.c.services] Starting sweep of stale
 reports (threshold: 14 days)
 2015-02-16 10:34:32,530 INFO  [c.p.p.c.services] Finished sweep of stale
 reports (threshold: 14 days)
 2015-02-16 10:34:32,531 INFO  [c.p.p.c.services] Starting database garbage
 collection
 2015-02-16 10:34:32,752 INFO  [c.p.p.c.services] Finished database garbage
 collection

 And then we're back to replacing facts again.

 Any ideas where we should go from here?

 Thanks,

 James

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/puppet-users/CAMH6%2BazCSD-BPD%2ByAO1jV_36bExqhVJ98DBBPk3s3ex4iDVNvg%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google

Re: [Puppet Users] Exporting resource to a particular host

2015-02-04 Thread Ken Barber

 I am using  puppetdb and exported resource  to manage autmatic nagios setup.
 It works very well. Now I want to setup another nagios server for another
 set of machines using same puppetdb and puppet master.
 As for I understand,  a client exports @@nagios_host and nagios server
 collect it by Nagios_host |  |

 I want some Nagios_host to be collected by different nagios server. Is it
 possible ? I was thinking about 'tag' but  am not sure how to use with
 exported resources.

It's pretty easy:

Nagios_host | tag == 'nagios1' |

See:

https://docs.puppetlabs.com/puppet/latest/reference/lang_collectors.html

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTk0U3%3Dww_06rRFUoosVjJzsPWffQE-1BD0_b%3DuqdUvzaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Raspberry pi + facter problem: Could not retrieve local facts: can't convert nil into String

2015-01-15 Thread Ken Barber

 I have installed Puppet on a Raspberry pi (running Raspbian) and it seems to
 work (sort of). I managed to add it to a Puppetmaster and sign its
 certificate but a puppet run fails:

 Info: Retrieving pluginfacts
 Info: Retrieving plugin
 Info: Loading facts
 Error: Could not retrieve local facts: can't convert nil into String
 Error: Failed to apply catalog: Could not retrieve local facts: can't
 convert nil into String

 It doesn’t show the facts that generates the error (even with the debug
 option supplied).

 So I executed facter on the command line (with the -p option) and got the
 following error:

 # facter -d -p
 Found no suitable resolves of 1 for ec2_metadata
 value for ec2_metadata is still nil
 Found no suitable resolves of 1 for lsbdistid
 value for lsbdistid is still nil
 can't convert nil into String

 Which made me think the lsbdistid fact was causing the problem but when I
 ran it separately it didn’t generate the error.

 facter -d -p lsbdistid
 Found no suitable resolves of 1 for lsbdistid
 value for lsbdistid is still nil

 Is there any (simple) way to find out which fact is generating the error?

 FYI, the following package/versions are used:

 ii  facter2.3.0-1puppetlabs1
 all  Ruby module for collecting simple facts about a host operating
 system
 ii  hiera 1.3.4-1puppetlabs1
 all  A simple pluggable Hierarchical Database.
 ii  puppet   3.7.3-1puppetlabs1
 all  Centralized configuration management - agent startup and
 compatibility scripts
 ii  puppet-common 3.7.3-1puppetlabs1
 all  Centralized configuration management
 ii  puppetlabs-release 1.0-11
 all  Package to install Puppet Labs gpg key and apt repo

Its been a while since I've hacked on facter so excuse my ignorance,
but does adding --trace surface the full stack trace from that
exception?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTn1iavQ1mEAPxGSkAEnMnRoUbZ-NDEDo7wbnS7qHY5uKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetdb memory use

2015-01-13 Thread Ken Barber

 in the meantime I've added RAM and extended the heap to 2GB. But still I'm
 getting crashes of PuppetDB.
 Last time it was the kernel OOM that killed the java process as I saw in
 /var/log/messages
 kernel: Out of memory: Kill process 10146 (java) score 158 or sacrifice
 child
 kernel: Killed process 10146, UID 498, (java) total-vm:3649108kB,
 anon-rss:1821580kB, file-rss:72kB

This kind of crash is more to do with the tuning on your Linux
instance usually. The OOM killer formula is somewhat tricky, but as a
general rule it takes into account the amount of RAM + swap (which
most people don't expect). So if your swap is zero, or very low in
relation to your memory, you may find the OOM killer is killing
processes before your RAM fills up.

The thing you want to research is overcommit_ratio, or your swap
allocation. There are lots of articles online about this.

As a general rule, if you're running with low swap you need the
overcommit_ratio to be higher, by default its set to 50% of the total
virtual memory ordinarily I think, so if a process tries to allocate
memory, and you've exceeded 50% of your overall RAM + swap, OOM killer
will kick in.

Here's an example from one of my instances, so you can see how to analyze this:

root@puppetdb1:~# free
 total   used   free sharedbuffers cached
Mem:   2054120 9754641078656  0 169772 219876
-/+ buffers/cache: 5858161468304
Swap:   892924  0 892924
root@puppetdb1:~# cat /proc/sys/vm/overcommit_ratio
50
root@puppetdb1:~#

So in my case, the total virtual memory available is 2.8 GB (2 GB +
800 MB swap), and if a process tries to allocated more than 50% of it
(1.4 GB), the oom killer might kick in.

I'm obviously trivialising the whole story for brevity, the OOM killer
has a few little quirks that might affect all of this (and quite a few
independant tunables), but more often than not if you think you have
enough RAM but the oom killer is still killing your processes its
somewhere between your swap + overcomit_ratio. I've seen this a bit in
virtualised environments, and places that try to launch instances with
zero swap as an example.

 Does this mean I need to add additional heap space and/or RAM?
 I'm looking at the dashboard and it's as if the heap slowly increases. Right
 after startup it's 3 of 400 MB, after a day or 1 I'm over 1 GB 

If its not crashing in the JVM any more, I'd focus on the tuning issue
above. If the Java instance is still crashing you can try increasing
it to see where the stable point is. If its starts to get stupid, and
you still don't know why its possible to analyze the heap dumps that
get left behind for any clues potentially, but about 90% of the time I
find its a large catalog causing it, so I'd focus there first.

Heap memory can naturally fluctuate over time ... and yes sometimes it
can increase but usually it should be garbage collecting and going up
and down, so it depends on what you are monitoring exactly, and where
you got the number I guess.

 I have no clue in how to find out what exactly is wrong. When I was running
 PuppetDB 1.6 I could do with 256MB heap space.
 Does anyone have an idea how to investigate what's wrong?

Well, it shouldn't have gotten any worse in the later versions afaik,
what version are you running now?

So here is the common memory-bloat story as I know it.

During ingestion of new content, PuppetDB has to hold the new
catalogs/facts/reports in memory for a short period of time as it
decodes the JSON and stores it in the internal queue ... at that point
there are really two copies running around, one is JSON, the other is
in the internal Clojure data structure. After that there are command
listener threads that process these 'commands', storing them in the
database. Sometimes, a very large catalog can cause a problem with
memory bloat, and if you happen to receive more than 1 at a time, it
can be much worse. At the same time, a very large factset or report
can also cause issues, like if you are storing a lot of information in
facts for example.

But more often than not, its to do with a large catalog, and while
most catalogs are sane there are a few cases that bloat them. A
combinatorial edge problem can often cause this, so doing something
like:

File||-Package||

(whereby we are trying to create a relationship between all file 
package resources, as an example here)

Can cause a many to many edge to be created, thus bloating the catalog
size. This is because that kind of graph will have many edges
reflected in its catalog. So trying to locate a large catalog might be
useful. Not to mention, such a catalog would cause slower compilation
times on the master also :-). Be mindful that we can receive any
catalog at any time without throttling (while we have free threads),
so if we get a few at a time, it could cause a memory bloat. We also
have N backend command processes ordinarily listening to the internal
queue, and if

Re: [Puppet Users] puppetdb report processor failed

2015-01-05 Thread Ken Barber

I recently upgraded from puppetdb 1.6 to 2.2 but now it seems like I'm
having issues with the puppetdb report processor.
I keep getting the errors below:
puppet-master[23130]: Report processor failed: Environment is nil, unable
to submit report. This may be due a bug with Puppet. Ensure you are running
the latest revision, see PUP-2508 for more details.

I'm running puppet-server 3.6.2 on CentOS 6.x behind an Apache with
passenger.
The PuppetDB v 2.2.2 is running on a PostgreSQL 9.3

Before the upgrade it worked fine, does anyone have an idea on what might be
wrong? Could it be related to the modified SSL config in puppetdb;s
jetty.ini config file? How could I test this?
I noticed it since all the hosts now show as unreported in Puppetboard

I doubt it's related.

This error is meant to pick up on the bug outlined in
https://tickets.puppetlabs.com/browse/PUP-2508, which was fixed in
3.6.1, or at least it should have been. Basically internally the
environment is not being set in that case, that patch was meant to fix
that. In short, its not meant to happen with 3.6.2 at all, so this is
surprising.

I certainly haven't seen this error in later revisions, or seen this
particular problem in the wild. I would suspect that you hadn't
restarted your Puppet master but I doubt this is true, as the error
only occurs when you upgrade (which needs a restart). Its possibly
related to some other magic, are you using environments in anger in
your setup? ie. do you use environments that are not 'production', and
are you doing anything special like using an ENC or something?

I think at this point we could try to reproduce the problem ourselves
for 3.6.2, but that would take me a little time to set up. Before I do
this, have you tested this with Puppet 3.7.3 yet? Also, is this only
happening in 1 environment, can you reproduce this on another puppet
master perhaps?

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkZ8hKf5d-V0%3D%2BFCCwDeEH5SEu1RAdMZe2JhfyW-KQA%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Using Puppetdb-terminus via rubygems...?

2014-12-08 Thread Ken Barber

We have entirely-gem based Puppet masters (no Ubuntu packages installing
Puppet)... we're trying to add in the puppetdb-terminus gemfile. We have it
configured, and installed:

# gem list | grep -i puppet
hiera-puppet (1.0.0)
puppet (3.7.3)
puppet-catalog-test (0.3.1)
puppet-lint (1.0.1)
puppet-syntax (1.3.0)
puppetdb-terminus (1.0)
puppetlabs_spec_helper (0.8.2)
rspec-hiera-puppet (1.0.0)

That is not an official gem for puppetdb-terminus :-). Someone
uploaded it in the past, I think its a dud.

We don't really ship a rubygem, we do have a gemfile however:
https://github.com/puppetlabs/puppetdb/tree/master/contrib/gem

That can be used to create your own gem.

We're running into a problem with our Unicorn/Nginx config though...

puppet-master[17213]: Could not configure routes from
/etc/puppet/routes.yaml: Could not find terminus puppetdb for indirection
facts

Is there something special we need to do to the config.ru or something so
that it can find the puppetdb terminus from the gem thats installed?

That just looks like the necessary ruby library files aren't in place,
probably due to the bad gem.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnH4ZQXnsGJpxB-W6SAypsJAsXtU0F%3DFoFXtcoRmTJbTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Exporting custom fact to PuppetDB ok with command but KO with daemon

2014-11-28 Thread Ken Barber

Nearly everything is in the title.

When I manually run a puppet agent --test on a host (let's say host1) that
export @@something, I can get something on another host (let's say host2)
and I'm happy.

Later, when the daemon on host1 runs, it still exports @@something but
custom facts are empty instead of having their value.

What am I missing ?

Do you mean your custom facts don't appear in your manifest code? Or
in PuppetDB?

Sometimes this could indicate a misconfigured routes.yaml. It should
look something like this:

root@puppetdb1:~# cat /etc/puppet/routes.yaml
---
master:
facts:
terminus: puppetdb
cache: yaml

This file belongs on your puppet master host, that is configured to
connect to PuppetDB. If you change it, your puppet master requires a
restart.

Without this configuration, you may find PuppetDB is being used for
facts instead of using the received facts from an agent. That is, it
might get populated the first time, but subsequent runs will only
retrieve facts from the first run from PuppetDB and only those will
appear in your manifest scope.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkPnEp4M%2BYQtSQki3ewHv5nMuBAq2%3D_%3DEmrLUWuSPuHVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB Queries

2014-11-18 Thread Ken Barber

 Would like to use PuppetDB to find out more about the estate inventory. 
 Specifically at the moment I am trying to find out the amount of servers 
 using productname, and the physicalprocessorcount for each - both with 
 totals.

 Is this doable from the API at all, or easier from inside PSQL ?

Do-able, but there are no server side aggregate operators/handlers for
this case in particular, its on our radar however. You can fake it by
using the include-total pagination option, and making your query
specific:

https://gist.github.com/kbarber/8175145dcca982b5cd0c

Then pluck out the 'X-Records' field, but that would require two
queries, one for each fact, /v4/facts/fact1  /v4/facts/fact2.

Otherwise perform a proper query for both fields, and count the
results yourself programmatically.

I can't recommend going to PostgreSQL directly, that isn't an API
contract we have commitments to - ie. we will change it underneath you
without warning or remorse today, even in a Z release :-).

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmdh44T-d-A1O8e6-ycLc0Of5gOja4-yi3Z-UwJxH9Zgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] command 'store report' version 2 is deprecated?

2014-10-24 Thread Ken Barber

 I recently changed over to a postgres back end. Now puppetdb.log seems to be
 awash in these errors.

 I'm pretty sure everything is up to date:


 # rpm -qa | grep puppet
 puppetlabs-release-6-11.noarch
 puppet-server-3.7.2-1.el6.noarch
 puppetdb-2.2.2-1.el6.noarch
 puppet-3.7.2-1.el6.noarch
 puppetdb-terminus-2.2.2-1.el6.noarch
 vim-puppet-2.7.20-1.el6.rf.noarch


 Did I miss a step somewhere?

Very likely you haven't restart your puppet master yet. Thats a
necessary step for the terminus upgrade to take hold, and use the
latest commands.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnWe1AznanBBp%3DJVcrKOJfOm2EhZ69aetaXMmW6Zy2%3DGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PDB 1.6: Queries filtering facts with large values might fail

2014-10-15 Thread Ken Barber

 We are running PuppetDB 1.6.0. We have fact 'a' in puppetdb that has large
 numbers occasionally, such as 2930266584. When we launch a query /v3/facts/a
 with filter '[, value, 1950341121]' it returns code 200 with an empty
 body, while puppetdb.log shows a new error:

 2014-10-15 11:20:41,293 ERROR [clojure-agent-send-off-pool-15310]
 [v3.facts] Error streaming response
 org.postgresql.util.PSQLException: ERROR: value 2930266584 is out of
 range for type integer


 The table stores values in strings, and apparently the query cannot convert
 large values into integers. Could someone please check if this issue also
 exists with newer versions?

In future versions the coercement is gone and types are supported. So
in this case to do something similar you must be using structured
facts and you must store the values as integers, they get stored in a
big int column, and the comparison will be uncoerced. Only signed big
int is supported today, there is no support for arbitrary precision
decimals.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D8ZkkbH-vYQQ_40TtktC5WrHTDJForZx%2B-yfJNqYLcFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Automatically remove deactivated host from icinga/nagios config

2014-10-09 Thread Ken Barber

I'm using this snippet to build my icinga configuration out of my exported
facts

#Collect the nagios_host resources
Nagios_host || {
target = /etc/icinga/puppet.d/hosts.cfg,
require = File[/etc/icinga/puppet.d/hosts.cfg],
notify = Service[icinga],
}

If I now deactivate a host on my puppetdb with:

puppet node deactivate fqdn.of.host

I would assume that on the next run my hosts.cfg should be without the
deactivated host. But this doesn't work. The host stays in the file. I can
only remove it, if I delete the hosts.cfg file and than let puppet run
again.
Did I miss something or is it not possible to automatically remove the host?

Nope it should work in theory, are you using PuppetDB for this? If so
in the puppetdb.log you should see a corresponding log entry for the
deactivate command for that node. Can you grep against your
puppetdb.log to see if this arrives when you send the `puppet node
deactivate {foo}` command.

Also check to make sure no new commands have come in for that node, it
will tell you in the log if this has happened. We reactivate a node on
any new data, so this is worth checking.

After deactivation you should be able to query the node data with
something like:

curl 'http://localhost:8080/v3/nodes/node_name'

And you should see a date next to 'deactivated' that indicates when it
was deactivated. If its deactivated then it should not collect.

Finally, try using a tool like this to analyze what exports still exist:

https://forge.puppetlabs.com/zack/exports

This will help you understand if the node you are trying to deactivate
is still exporting this data, or if its coming from another place.
This is important, sometimes there is data coming from another node,
and its often what users don't expect. Make sure you check for typos
on the node name most importantly, this is the biggest contributing
factor to confusion around this :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnkw34zrdHoj4rEbNPnTP8MC4%2Bg5JpAHMWrS4C6WpQDUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Automatically remove deactivated host from icinga/nagios config

2014-10-09 Thread Ken Barber

Yes, I'm using puppetdb and the command arrives at the database:
2014-10-09 13:31:37,302 INFO [c.p.p.command]
[55a70ff1-3491-4885-9614-589b35756883] [deactivate node] fqdn.of.host

Good.

What caught my attention here is that the Timestamp here is different from
the timestamp in the log. Maybe the is something going wrong there? PuppetDB
and Puppetmaster are running on the same host, so there should be no time
difference.

The timestamp is in ISO-8601 format, which means its got a timezone
associated with it, in this case UTC. Could this be the cause of
confusion?

Finally, try using a tool like this to analyze what exports still exist:

https://forge.puppetlabs.com/zack/exports

The node doesn't show up with puppet node exports
But a puppet agent -t run on the icinga node still doesn't remove the node.

Maybe I should say that I am using foreman. But I also deactivated the node
in foreman. So my guess is that I'm good there.

Wait, are you actually purging the resources somewhere? If it becomes
unmanaged, that doesn't mean it cleans up after itself unless you are
purging also.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DpAnOoPp5LrSfuojGzO4bDYxSscpks3-PZHFHTi6T9fw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Automatically remove deactivated host from icinga/nagios config

2014-10-09 Thread Ken Barber

 Wait, are you actually purging the resources somewhere? If it becomes
 unmanaged, that doesn't mean it cleans up after itself unless you are
 purging also.


 I did this:

 Not sure if I can follow you though?!

 What happens if I manually add a (fake) host to my hosts.cfg file. The host 
 doesn't exist in the puppetdb, because it was never alive. On the next 
 puppet run, puppet should remove this false entry in my hosts.cfg, right?

No not necessarily, you need to enable resource purging with resources
like nagios_host:

resources { nagios_host:
  purge = true,
}

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTktrxTdvDGSYpANB_RkoydN-S8jEjOdLyDZDYBbxnjJGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Automatically remove deactivated host from icinga/nagios config

2014-10-09 Thread Ken Barber

No not necessarily, you need to enable resource purging with resources
like nagios_host:

resources { nagios_host:
purge = true,
}

Oh, I just did not now that. My manifest now looks like this:

resources { [nagios_host, nagios_service]:
purge = true,
}

#Collect the nagios_host resources
Nagios_host || {
target = /etc/icinga/puppet.d/hosts.cfg,
require = File[/etc/icinga/puppet.d/hosts.cfg],
notify = Service[icinga],
}

But the entries don't get purged. Looks like I'm still missing something :/

I think you are running into this:

You can purge Nagios resources using the resources type, but only in the
default file locations. This is an architectural limitation.

https://docs.puppetlabs.com/references/latest/type.html#nagioscommand

i.e. if you set the target parameter, you lose the ability to purge.

Well spotted Jonathan ... :-).

So Kai, you can provide fake this with soft-links to the icigna dir
from the expected nagios configuration directory. Or soft-link the
files themselves, up to you.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTme_hcT%3DWitVy3%3DLMqmP2rZu%3DB32MNidHovp8D%2B3s8kVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: puppetboard unable to reach puppetb

2014-10-09 Thread Ken Barber

Are you pushing reports into puppetdb or only into foreman?

Ok so I missed that. Sorry dude. And yeah as you point out I originally had
reports only going to foreman. But I changed the puppet.conf to this:

[root@puppet:/etc/puppet] #egrep -i reports|storeconfigs puppet.conf
reports= foreman puppetdb
storeconfigs = true
storeconfigs_backend = puppetdb

I wasn't sure if this was supposed to be comma dlimeted or space. I tried
using a space between the two.

And so far no change.

http://puppetboard.jokefire.com/

Its a comma, if you are unclear. Try it again and restart your puppet
master. Then check your PuppetDB logs (/var/log/puppetdb/puppetdb.log)
you should see logged commands for the 'store report' command be
submitted on a node after you do an agent run, if not - check your
puppet master logs to see if its logging any errors with sending the
reports. The puppet master usually logs to syslog FYI. Basically these
'store report' commands should be submitted in order from one host
like so:

* replace facts
* replace catalog
* store report

If none of that helps, we can explore turning on debug mode for Puppet
to see what's being logged.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DsYszRgMm_eBWqvkDH8UsZnEB06knoQZT4r_pjWx2X2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppet-jobs list?

2014-10-09 Thread Ken Barber

A cursory google search hasn't turned up much on this topic.

Is there a puppet-jobs list for jobs oriented around Puppet (not to be
confused with Work at Puppet Labs! style stuff?

Other projects seem to have similar lists, but I can't find one for Puppet;
I suspect it'd be constructive to have a place to throw a few Puppet
projects to the community without spamming the users list.

Howdy, a list moderator here. We don't have a formal list, the only
thing I can speak to is perhaps LinkedIn, which seems to be a popular
place. Perhaps others can chime in about what's available in the
industry if you can supply a location. Having said that, if all you
want to do is post a job for a company you actually work for then
thats fine if you are an active member of the list, or your members of
your company are. The rules are simple:

Recruiters are not permitted to post jobs to our mailing lists.
However, if you are an active community member and you are personally
hiring more people to work on Puppet, you may post relevant job
descriptions.

We try to be pragmatic about it obviously, and this is largely in
response to our users needs rather than any hard and fast rules
because this list is primarily about helping community members not
about commerce. Others have voiced concerns in the past about
recruiters, and its our responsibility to maintain that.

That means basically don't be annoying about it :-). I'd suggest
looking through the list archives to see the 'tone' that others try to
set when advertising jobs for their companies to get a feel for the
rough expectations. Most people put [JOB] in the title to designate it
as such.

Full details of our guidelines and the quote I supplied above are here FYI:

https://docs.puppetlabs.com/community/community_guidelines.html#mailing-list-guidelines

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkU2Gpq3EzyWfF5ZJiPQjuSb92heBg%3DB1%2BOFVxiB3Wfow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-08 Thread Ken Barber

The default.yaml is in the gist I supplied. I used the el6 and centos6 64
from your sample project.

Taking a look again, its even more confusing than that, the platform
name you have used is pointing at one of the very very old AMI's. The
newer ones we actually use are private, but I don't have the ability
to make it public. Looks like its locked up in someone elses older
amazon account. Sorry about that. This is why we are slowly replacing
these older ones, the management of them has slipped, my fault mainly
- we'll work on that for PuppetDB.

An Ami search on aws by the name or Ami ID didn't turn up any results which
is why I thought they were private.

I'll try building an Ami with packer.Are there any conventions that need
to go into the VM required by beaker?

I think the main requirement to make your life easier, is to allow
logging in from root (which can be often disabled) this is often done
by changing just sshd_config (and hupping/restarting sshd), and
removing the /root/.ssh/authorized_keys files also. Some distros
support doing this kind of thing via cloud-init as well, so you can
pass something like:

#cloud-config
disable-root: false

Into the user_data field of packer for example, but not this will not
persist on its own, you can change the configuration in /etc/cloud/
and bake that setting into the image. This usually affects
Debian/Ubuntu images I believe, I haven't built one of the newer
centos images in a while so try firing up the image and checking if it
has /etc/cloud/ in its build, this usually indicates its got some sort
of cloud-init. Having said that, someone has a patch up for beaker to
allow logging in as non-root and 'fixing' this:

https://github.com/puppetlabs/beaker/pull/478

Beyond that it depends on the tests and helpers you want to use. Both
'curl' and 'git' are good tools to have around on an image, since a
lot of tests can utilize these commands, having said that some tools
on top of the image can most certainly be installed by beaker in the
spec_helper_acceptance.rb there is a section for doing this _before_
the suite runs.

In fact, the base beaker stuff will attempt to install items for you,
like timing is important so the code goes to lengths to make sure the
the image has the right tools before it uses ntp to update the time.
Having the tooling there (I believe it uses ntpdate? but look at your
log output from beaker it will tell you) should make that exercise
mildly faster.

Starting simple, and trying to run tests against an image is usually a
good move in general, and being additive when you need it. Like I said
you have two choices most of the time - bake in a tool, or do it
during the testing run _before_ suite ... its really dependant on the
wider problem you are going to solve with the image. Baking items into
the image will of course speed up the exercise at test time, at the
cost of forcing you to remake it when it needs a change. Starting
conservatively with what goes into the image I think is a good idea.

After that you'll start to see the wider problems when you run your
tests a lot ... such as the flakiness of public mirrors :-). But thats
a story for another day :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DmaOQ9QvoVC-e9DkQm%2Bs7iRDrgt3%3Dvn%2BHYU%3D_Zvsw%3DAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-07 Thread Ken Barber

I was just testing the host config file from puppetdb coupled with the
documentation on the beaker documentation.

Those docs honestly look old, they are still mentioning blimpy which I
effectively deprecated/superseded with the aws_sdk driver.

I was actually going to omit the error message. That's actually all of it
except for the json output of the compiled beaker configs. I can send the
full output in the morning.

Send the full output and the configuration and I can take a closer
look. Anything less, I'll probably struggle.

It looks like the Google Compute Engine docs are more complete... It
doesn't matter where it runs. Mostly looking for a free tier cloud to get
started with. I'm not sure aws micro would even be big enough anyways. But
it'd be cool to get it working.

Sure, well we use EC2 heavily so I'm happy to help you there, I know
some people use Google Compute Engine also, but I have no intimate
knowledge of how this one works.

Actually Brett, maybe this is a better approach. I've got a working
new project here showing beaker + beaker-rspec with EC2 support:

https://github.com/kbarber/sample-beaker

And you can see how I've launched it here:

https://gist.github.com/kbarber/850a7d88fce409592bab

Perhaps a better example will set you straight :-). It does fail
incidentally, but it is kind of meant to as an example. Perhaps you
can start with this project skeleton and modify to taste.

Now as Justin mentioned, you do need a ~/.fog file - this was
primarily to be compatible with the old Blimpy driver, but alas, we
don't use Fog any more. The file should look something like:

:default:
:aws_access_key_id: AA
:aws_secret_access_key: BB

(And obviously match your own EC2 access keys and secrets)

Also pay close attention to the config/image_templates/ec2.yaml file
... these map names of images to AMIs today, and the AMIs provided are
the ones we use for our own testing, but you might want to maintain
your own list. This is entirely up to you, just be aware each AMI
needs a small amount of pre-setup if you want to create your own (that
is certain minimal things might need 'baking' into the image, but
nothing drastic). Of course, you are free to use the ones here, but if
you need customisation I'd suggest forking your own images :-).

Let me know how you get on with that. I think generally speaking all
of this needs an overhaul in regards to usability, a lot of this
awkward layout is due to backwards compatibility from legacy elements.
That aside, once you get the fundamental elements right it should be
okay.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DzPBPSQi04EWH8J7sbMqmV0OXititEcgbPGM8utnjzGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-07 Thread Ken Barber

 That's great Ken.

 I'll have a look.   My .fog file was correct but I was missing that
 ec2.yaml.

 I get the user experience thing, it'll evolve and I'll help if I can.

 Would I be right to assume you built your images with packer?

All of those images predate packer, but we're using packer now. For
example my colleague Wyatt is about to add Debian 7 testing using a
packer template he has developed. So going forward yes, packer is the
trick.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnp56EsgFj9W2c%3DDnWFUWEz34asHdC06%2BvBV1cmEhYpLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb has no tables

2014-10-06 Thread Ken Barber

When I have a look at the logs I see that I'm getting password
authentication failures for the puppetdb user:

[root@puppet:/etc/puppet] #tail -30 /var/log/puppetdb/puppetdb.log
2014-10-05 16:25:36,339 ERROR [c.j.b.h.AbstractConnectionHook] Failed to
acquire connection Sleeping for 7000ms and trying again. Attempts left: 1.
Exception: null
2014-10-05 16:25:43,340 ERROR [c.j.b.PoolWatchThread] Error in trying to
obtain a connection. Retrying in 7000ms
org.postgresql.util.PSQLException: FATAL: password authentication failed for
user puppetdb

This is what I have in my /etc/puppetdb/conf.d/database.ini

[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //127.0.0.1:5432/puppetdb
username = puppetdb
password = secret
log-slow-statements = 10

Yet when I try to log into the postgres database using those credentials I
get no error:

[root@puppet:~] #su - postgres
-bash-4.1$ psql -h 127.0.0.1 -p 5432 -U puppetdb -W puppetdb
Password for user puppetdb:
psql (8.4.20)
Type help for help.

puppetdb=

So this might potentially be a problem with your pg_hba.conf, I'm
guessing though.

Try the above again, but as the root user, not as postgres, and show
us the results.

Here is my pg_hba.conf fwiw:
https://gist.github.com/kbarber/0fd98ec424687c1bea9d. Send us yours
and we can take a look.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnmXHx2vWp8GuYRQkXNrrya9HT5SFeqr0TDQjb629G-AQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb has no tables

2014-10-06 Thread Ken Barber

When I have a look at the logs I see that I'm getting password
authentication failures for the puppetdb user:

This is what I have in my /etc/puppetdb/conf.d/database.ini

[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //127.0.0.1:5432/puppetdb
username = puppetdb
password = secret
log-slow-statements = 10

Yet when I try to log into the postgres database using those credentials I
get no error:

[root@puppet:~] #su - postgres
-bash-4.1$ psql -h 127.0.0.1 -p 5432 -U puppetdb -W puppetdb
Password for user puppetdb:
psql (8.4.20)
Type help for help.

puppetdb=

So this might potentially be a problem with your pg_hba.conf, I'm
guessing though.

Try the above again, but as the root user, not as postgres, and show
us the results.

Here is my pg_hba.conf fwiw:
https://gist.github.com/kbarber/0fd98ec424687c1bea9d. Send us yours
and we can take a look.

Having said this, and actually tested my theory myself, I don't
believe this is the case - but worth looking into. When I tested
'breaking' my pg_hba.conf I get this result specifically:

2014-10-04 11:52:53,300 ERROR [c.j.b.h.AbstractConnectionHook] Failed
to acquire connection Sleeping for 7000ms and trying again. Attempts
left: 5. Exception: null
2014-10-04 11:53:00,310 ERROR [c.j.b.h.AbstractConnectionHook] Failed
to acquire connection Sleeping for 7000ms and trying again. Attempts
left: 4. Exception: null
2014-10-04 11:53:07,320 ERROR [c.j.b.h.AbstractConnectionHook] Failed
to acquire connection Sleeping for 7000ms and trying again. Attempts
left: 3. Exception: null
2014-10-04 11:53:14,330 ERROR [c.j.b.h.AbstractConnectionHook] Failed
to acquire connection Sleeping for 7000ms and trying again. Attempts
left: 2. Exception: null
2014-10-04 11:53:21,341 ERROR [c.j.b.h.AbstractConnectionHook] Failed
to acquire connection Sleeping for 7000ms and trying again. Attempts
left: 1. Exception: null
2014-10-04 11:53:28,347 ERROR [c.j.b.PoolWatchThread] Error in trying
to obtain a connection. Retrying in 7000ms
org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for
host 192.168.182.132, user puppetdb, database puppetdb, SSL off
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
~[puppetdb.jar:na]
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
~[puppetdb.jar:na]
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
~[puppetdb.jar:na]
at
org.postgresql.jdbc2.AbstractJdbc2Connection.init(AbstractJdbc2Connection.java:136)
~[puppetdb.jar:na]

Which is very specific, and definitely not a password auth error. Hmm.
Of course this is with 9.3, there is a chance the protocol has
changed.

I know it probably seems very dumb at this point, but have you tried
changing your password for the puppetdb user to something simple ...
and trying again?

http://www.postgresql.org/docs/8.4/static/sql-alteruser.html

I'm not saying you're typing the password incorrectly or anything, but
the process of changing the password for the `puppetdb` pg user might
shake something up. I'm really grasping at straws, an incorrect
password error is usually just that ... but your tests don't align
with this potentiality :-). I guess what I'm saying is that it feels
like something really silly that we are missing.

Still, try the `psql` auth test as root as I mentioned, pass us your
pg_hba.conf, you never know it might also shake something up that we
didn't notice before.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkzErKhRLjVPqB%3D_tp%2BLm5TkudiyAWrJ0OsFypQ6fmNiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb has no tables

2014-10-06 Thread Ken Barber

  Thanks again for your help. I changed the password on a temporary basis to
 an absurdly simple one. I'm both happy to say that puppetdb is working now.
 And sad to have taken up your time with this. Sorry about that.

No problem mate, glad it was something simple in the end :-).

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmmFnR_qRCuFWf60Pig%3DnBCCdy_qm0ihW1OFRWpTY3wNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-06 Thread Ken Barber

I've seen how the puppetdb module uses ec2 to execute beaker tests. I've
tried setting this up as well and am getting some errors.

Is there a working example of using the different hypervisors?

I see this:
https://github.com/puppetlabs/beaker/wiki/Creating-A-Test-Environment#ec2-support
but the documentation doesn't suggest it's a polished feature :)

It might be helpful to document a project that has an example of all the
hypervisors, as each one will have different required / optional parameters
in the nodeset configuration file.

Indented below is the error when running `beaker --hosts
spec/acceptance/nodesets/ec2-west-el6-64mda-el6-64a.cfg` - I pulled that
example from the puppetdb project, and as per the documentation have a .fog
file set up.. I was hoping to get an authentication or configuration error
on first run.

Beaker::Hypervisor, found some ec2 boxes to create
Failed: errored in CLI.provision
#TypeError: no implicit conversion of Symbol into Integer
/Projects/live_modules/puppet-shawlib/.bundle/ruby/2.1.0/gems/beaker-1.19.1/bin/beaker:6
/Users/bswift/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15
/Users/bswift/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15

I don't have much to go on there..

The exception seems a little truncated, is that it? BTW Instead of
annotating what has happened in a paragraph, just provide the full
example in your shell in a gist or something, it says the message much
more simply :-). Sort of like this:
https://gist.github.com/kbarber/97f45eba0f922497901a. Or better yet,
if you can create a repository with the basics of the above so I can
take a look that would be much much easier.

I must warn you, the aws_sdk.rb code was written in a massive hurry,
and thus has very little error protection so if everything isn't spot
on, it might break.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D9knDc5Ovg3_ws7K_4UifvPV1Djo%3D8FJFsZO3EPLDpjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-06 Thread Ken Barber

I've seen how the puppetdb module uses ec2 to execute beaker tests. I've
tried setting this up as well and am getting some errors.

Is there a working example of using the different hypervisors?

I see this:
https://github.com/puppetlabs/beaker/wiki/Creating-A-Test-Environment#ec2-support
but the documentation doesn't suggest it's a polished feature :)

It might be helpful to document a project that has an example of all the
hypervisors, as each one will have different required / optional parameters
in the nodeset configuration file.

I don't have much to go on there..

I must warn you, the aws_sdk.rb code was written in a massive hurry,
and thus has very little error protection so if everything isn't spot
on, it might break.

FWIW also - I wouldn't test the state of the puppetdb module testing
pieces, we've been waiting for our QA Module teams to automate those
tests for a about a year now :-(. But honestly, they haven't been ran
for quite some time - so I'd be dubious.

The PuppetDB source code itself (ie. puppetdb server, not the module)
however does use the AWS EC2 test stuff, and thats a better example to
use.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DgHWf5zD3j8GD8%2BYXsf5HCwYPvSXcH2Veka2O2_b5w_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] beaker ec2 example, should this work?

2014-10-06 Thread Ken Barber

 I was just testing the host config file from puppetdb coupled with the 
 documentation on the beaker documentation.

Those docs honestly look old, they are still mentioning blimpy which I
effectively deprecated/superseded with the aws_sdk driver.

 I was actually going to omit the error message.  That's actually all of it 
 except for the json output of the compiled beaker configs.   I can send the 
 full output  in the morning.

Send the full output and the configuration and I can take a closer
look. Anything less, I'll probably struggle.

 It looks like the Google Compute Engine docs  are more complete...  It 
 doesn't matter where it runs. Mostly looking for a free tier cloud to get 
 started with.   I'm not sure aws micro would even be big enough anyways. But 
 it'd be cool to get it working.

Sure, well we use EC2 heavily so I'm happy to help you there, I know
some people use Google Compute Engine also, but I have no intimate
knowledge of how this one works.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTng8oLjKCaVS6RV%2BjBSHqWAgYSatP69fpxcNWF1Upmz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb has no tables

2014-10-05 Thread Ken Barber

I've installed puppetdb on my puppetmaster. I have puppet-server-3.7.1,
puppetdb-2.2 and puppetdb-terminus-2.2.

I've setup puppetdb like this:

[root@puppet:/etc/puppet] #cat /etc/puppetdb/conf.d/database.ini
[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //127.0.0.1:5432/puppetdb
username = puppetdb
password = secret
log-slow-statements = 10

[root@puppet:/etc/puppet] #cat /etc/puppet/puppetdb.conf
[main]
server = puppet.jokefire.com
port = 8081

[root@puppet:/etc/puppet] #cat /etc/puppet/routes.yaml
---
master:
facts:
terminus: puppetdb
cache: yaml

[root@puppet:/etc/puppet] #grep storeconfigs puppet.conf
storeconfigs = true
storeconfigs_backend = puppetdb

I'm running puppet via passenger and using foreman 1.6.0. And I've selected
postgres as the database for puppetdb as you can see above. When I perform
puppet runs everything seems to go OK. Which I thought meant that everything
as good between puppet and the puppetdb.

However, when I connect to the postgres database and try to show tables for
the puppetdb database, it says that 'no relations are found'.

[root@puppet:/etc/puppet] #su - postgres
-bash-4.1$ psql
psql (8.4.20)
Type help for help.

postgres=# \c puppetdb
psql (8.4.20)
You are now connected to database puppetdb.
puppetdb=# \dt
No relations found.

So my question is, if I have everything setup correctly, why am I not seeing
any tables inside the puppetdb database? Am I missing any steps that I need
in order for this to work?

Is PuppetDB actually working at all? Or is this just a symptom you are
trying to track down.

Well couple of things, can you show your puppetdb.log for starters,
perhaps the initial startup might be useful.

I'm hoping this isn't it, but there is a small chance you have a
duplicate entry in your conf.d directory, I've always found this
methology to be a good one for checking this:

# cd /etc/puppetdb/conf.d
# grep '' *

Can you show us the results of this (with secrets removed also) it
might give us another clue.

It sounds to me like if its working its either using a different
database than we think it is, or potentially HSQLDB which is the
embedded one. You should be able to see clues if you are accidentally
using HSQLDB in this regard in /var/lib/puppetdb/db:

root@puppetdb1:~# ls /var/lib/puppetdb/db/
db.log db.properties db.script db.tmp

None of what you have provided shows this to be true so far, but worth
double checking :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnUcKLxxfCdp2NjkVdSekNL-%2BwSOvUvVpu_w%3DgvzZFaiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Puppet facts uploading to PuppetDB

2014-10-02 Thread Ken Barber

 We do this, but could probably live without it. But we do it using the facts
 indirector and setting it up to cache to puppetdb.

So in both cases you use 'puppet facts upload'?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkmoq_Casx%2B%2B_-tVnmBtryvwzqbR32%3D0Pq298W73zbXKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: PuppetDB 2.2 and pg_trgm extension

2014-10-01 Thread Ken Barber

 I tried the same thing and got the error below. Any ideas?

 puppetdb=# create extension pg_trgm;

 ERROR:  could not open extension control file
 /usr/share/postgresql/9.3/extension/pg_trgm.control: No such file or
 directory

Seems odd, pg_trgm should be shipped with PostgreSQL. Maybe its a bug
in that package or pg_trgm is shipped separately?

I can probably take a look here. What distro and distro release is
this exactly? Is this the PG shipped with the distro or is it from
PGDG upstream?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm4-2Yb5aOFcgxLha2AyRJD2HkJrL%3DqDA_Arj1TJsfBvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Announce: Puppet Server 0.2.0

2014-09-29 Thread Ken Barber

More information along these lines, highlighting ease of use and
tools for users to see their catalogs, will go along way towards soothing us
touchy sysadmins.

Totally understand, I was a very touchy admin myself before working at
Puppet Labs and when the tools let you down it can be frustrating.
Like chair-throwing frustrating sometimes :-). This is why I switched
to be a full time dev for PL, I wanted to make the experience better
in my own small way.

The performance gains while nice don't have the appeal of
better troubleshooting. I'm happy to learn yet another stack, but I'd like
to be sure I'm getting some thing more than the status quo.

Absolutely, and this is where I think we wouldn't have it any other
way. I can think of at least a few things I hope to gain now the
Puppet platform is changing, one of them at least being that I'd like
to start considering fallback caches and queues on the master for
PuppetDB delivery (ie. a change to the puppedb-terminus application -
this was a little harder and less clean in a Ruby/Passenger world), so
that when a remote PDB instance is down the message is not lost, but
queued up for delivery later.

Not to mention the opening up of monitoring facilities via JMX and
such, which should expose a fair amount of metrics just on its own,
but allows us to poke holes into our application expose to users what
things are doing for monitoring/alerting purposes. In PDB we have a
great HTTP/JMX bridge for those who don't want to use JMX itself also
(a consideration for admins btw - as pure JMX isn't always desirable),
hopefully we'll port that over to our other services in time (we'll
make it a trapperkeeper plugin most probably - so all our applications
get it) but for now JMX is there and able to be used right now.

We hope to hear more about where people want to go with this tooling
as well, so please let us know where we are going wrong. Its very easy
to create an insular application that doesn't expose enough, but
another thing to create a great tooled eco-system.

The information around tuning Passenger/Puppet explicitly provided
by Puppet Labs was mostly crap.

Indeed, it was a bit of a black art because of this. It wasn't until
later that Passenger even added the ability to reasonably introspect
what was going on in Passenger.

It would be extremely useful for everyone if
there were 4-8 pages of serious and indepth docs specifically about running
puppet_server on the JVM. If that doesn't happen, you'll be fighting the
supposed poor performance of every un-tuned puppet_server installation for
years.

Sounds like something ticket-worthy to mention. We already have some
of this for PuppetDB, a lot of it is similar for this platform as
well. I'm pretty sure this will become a hot topic, so I doubt it will
be left alone. I expect the new puppet-server to incur more traffic
than PuppetDB for example, so they'll probably see issues we have not.

Not only that, as you see issues you should be bringing them up in
these forums - and then we can discuss and feed that back into the
docs as we go (as you know most of our docs are user-contributable as
well, to make it easier). We can do as much benchmarking as we like in
the lab, but the real world is the only true learning ground. I've
been following Puppet for gosh, 7-8 years or something now, in the
forums and such - and one thing I've always enjoyed is how well we
taught each other.

At the very least I can tell you the web server is Jetty, and will
have tuning similar to PuppetDB. It has all the predominant tuning
items one would expect, like # of threads, but hopefully the reduction
in moving parts should actually help. Not only that, I've recently
patched the trapperkeeper plugin for Jetty so it exposes its JMX
monitoring capabilities:

https://github.com/puppetlabs/trapperkeeper-webserver-jetty9/commit/cb727a4731bc7e2df7151c93a1b9f91461823a91

Which I really wanted for PuppetDB, but the beauty of trapperkeeper is
that we all get it now. This means you get all kinds of introspection
into what the web server is doing, much like you would with Apache,
but in a potentially more curated/monitoring prepared way (I'm not
trying to upset an Apache fans here, I love Apache as well, but we
couldn't easily embed it :-).

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnKUSwOp1jRb4-67Rd%2Bcc2%2Bc8NxBtKoVKzfYGjqsmMCTw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Announce: Puppet Server 0.2.0

2014-09-29 Thread Ken Barber

 And further, I'd really like to see non-Ruby scripting languages enabled to
 participate as first-class citizens for the extension points - this (coupled
 with better definition of core APIs) would really make the on-ramp for new
 puppet users much lower friction.

Python support would be lovely.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnXauh0EXPR_udPw7kF24St%2B7AYoPzVKSAQtZM4%3D27M2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Trouble extracting raw data from custom fact

2014-09-29 Thread Ken Barber

I'm new to puppet and hope somebody can help me. I have a master server with
a few Mac and Windows Agents. I created a custom fact for windows that will
show the current logged in users Sid. I have a manifest that will edit the
HK Users registry Keys. My issue is when I run the agent it fails because it
wants to place the header and other stuff not needed from the string in the
path. I need the Sid only for it to work. It works if I manually insert the
Sid. I'm thinking my answer is regsubst but I really don't know much about
it. what I need to do is remove the first few characters and the last few.
Can anyone help me out or point me in a direction? Or is there something I
can do in my fact.rb to only show the raw data? Thank you in advance this
one has me pulling out my hair.

regsubstr as a function will help
(https://docs.puppetlabs.com/references/latest/function.html#regsubst),
or you can do it in the fact itself using Ruby's string substitution:
http://www.ruby-doc.org/core-2.1.3/String.html#method-i-sub

The idea with both is that you need the incoming variable, a regular
expression to do the match, and then a replacement. Having said that
however, there are many options available to do string manipulation
that might help, such as split (which will convert a string into an
array by splitting on a common delimiter).

If you are struggling with the solution, perhaps you can post your
code so far, an example of what the data looks like and what you want
it to look like. I'm sure someone can then provide an example and an
explanation.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkJhYecfk0VtmQsqurK9xhrgrFAPZpPurK8sOatSpVHCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Puppet facts uploading to PuppetDB

2014-09-27 Thread Ken Barber

Hmm... I didn't even know this existed. Ironically, given your question, it
sounds like something I'd want to use. But if it's going away, I guess I'll
just totally forget that I heard it...

Oh to be clear Jason, the functionality is of course still staying for
PuppetDB :-). Its just the transmission via the master that has been
removed. There are a number of future architecture cases that might
allow this via an alternate path, that are being debated however - but
we don't want to rush into something like this, or make promises we
can't keep :-). Understanding the use-cases in the real world is
obviously important to these considerations.

Use case: We still have a bunch of legacy systems that aren't puppetized and
probably never will be (lots of base stuff that we do on every host that
would break these snowflakes). For everything else, PuppetDB is our one and
only inventory system. These legacy boxes exist only in a... spreadsheet. If
I knew there was a way to get facts from them into PuppetDB without risking
what would happen with a full puppet run, I probably would've done it by
now...

So this sounds like you could just get away with something that
doesn't require the Puppet runtime, that can submit facts directly to
PuppetDB. At the moment this is completely possible with our commands
submission API, but afaik tooling is generally done by users only in a
bespoke way, at least I don't know anything that has been published.

Truth is its actually dead easy to do this, (*hint hint* for those who
have been looking for a personal project to work on, I'm sure we'd
love to see such a thing - and would happily promote it here:
https://docs.puppetlabs.com/puppetdb/2.2/community_add_ons.html). I
have something ~10 lines that already does this for example in Ruby
(using the Facter class from the facter gem/library), but since we're
moving to a cfacter world, might be good to consider doing it
different for future compatibility (not to mention a C++11 based
facter will provide more portability opportunities).

The main complication as ever, is the SSL authentication and the PKI
handling of keys/certs etc. Obviously with Puppet you get the key/cert
signing baked in - although one might consider yet another tool or
bespoke methodology to make this work as an alternate to the Puppet
client.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DKfbj-4f4BUgX2PgrM%3DKbcNt6RXxhTKfHL%3D4xpyrpSxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Announce: Puppet Server 0.2.0

2014-09-27 Thread Ken Barber

(1) at my current shop, there's an immense hatred of everything JVM. That's
going to be a hard transition. Not to mention Puppet is the only place we
run Ruby, so it's nice and easy to let puppet do whatever it wants with Ruby.
Not so much for installing JVMs that may break production (improperly
configured and installed, I'll grant) applications.

And rightly so - its had a bad history, but I must argue that largely
my hatred of JVM in the past wasn't the JVM per se, it was the
applications written for it. But I would gladly blame the JVM most
times. Also - most of the hatred I see in the industry is a lack of
understanding around the JVM. For me, I'm an old Perl programmer and
certainly making the transition over the last ~17 years was one I
fought against, more because of my own stubbornness I guess. But once
I started to actually study and learn about the tooling for JVM and
accepts its place in the application stack instead of just hating it,
my attitude began to change.

For example, I could never have understood memory usage in PuppetDB if
it was written in Ruby - never is probably too strong - but its hard
in Ruby to do this ... I have tried and it kind of sucks. But hey,
with clojure/jvm, I can use YourKit which gives me an almost
ludicrously simple way of seeing the memory flow. Point in case, we
used to use the urlencoded way of doing POST submissions for commands,
but when I analyzed command submission in Yourkit (live service mind
you) I quickly realized we had 2 objects, the encoded one and the
unencoded. Just think about that for a second - 2 copies of a very
large catalog in memory ... very wasteful :-). So yeah, we stopped
encoding, it wasn't needed anyway and halved our memory consumption
for command submissions and removed that processing need completely -
again thanks to JVM tooling. This work took at best a day or two,
including the patch I believe.

Same again for queries, we switched to streaming for this same reason
... versus loading up the answer and serving it all in one go ... we
now open a cursor on the db, and as answers come back we stream it via
HTTP. The Java core libraries and Clojure in particular are actually
very very good at doing streaming ... and on our platform streaming
becomes critical to reducing memory usage.

For me, I would only see the Erlang runtime coming close to this as a
serious contender (and perhaps the .Net framework/CLR might have
something here, but this isn't my area of expertise), and while the
tooling there for Erlang is pretty awesome, its not as evolved as the
JVM stuff. Don't get me wrong, I love Erlang too :-).

(2) I've gotta say, I'll really miss dropping log statements directly in the
puppet source when something seems wonky (and not having to compile
something).

Our answer to this for Clojure is usually a combination of NREPL and
(log/spy original item you want to see) from the
clojure.tools.logging library or #spy/d statements from the spy scope
library. Works great, and can wrap just about any variable as a nice
piece of magic to drop debug statements.

The nice thing here that we didn't have in Ruby is that NREPL allows
changes to a running service. So no need to stop/start the service to
see your debug lines.

I do this quite often for PuppetDB while developing, that is I have a
running PuppetDB instance and the PDB source code open in Emacs (with
the cider plugin for nrepl support already bootstrapped of course) ...
I modify the code ... save it ... hit Ctrl-C Ctrl-K ... and I see the
debug lines start to appear in the log. Its a far more rapid workflow
(to be clear: Emacs is only my choice, I believe there is NREPL
support in vim, eclipse, intellij and various other editors as well).

Oh yeah, and this can be done on real running systems also it doesn't
just have to be a dev workflow, you just need to have the NREPL port
exposed in your PDB config.ini:
https://docs.puppetlabs.com/puppetdb/2.2/configure.html#repl-settings.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmTPZhL1gRKrV9BfGBJ7CL0qvamHr21Q2yMHp2OyWD16A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb and oracle java

2014-09-26 Thread Ken Barber

 The requirements for puppetdb specify that it supports 1.7 from either
 openjdk or oracle. I've got oracle installed (RHEL6) but the rpm insists on
 openjdk (which I can't install for other reasons). Anyone know of a way
 around this, or am I going to have to hack the package?

Huh, I guess you are right looking at the spec:

https://github.com/puppetlabs/puppetdb/blob/master/ext/redhat/puppetdb.spec.erb

Honestly, you are the first one to pick complain about this :-). Worth
a bug: https://tickets.puppetlabs.com/browse/PDB

Why can't you install OpenJDK? Understanding this would help us
understand the priority of such a fix.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmhZhFDpJpo61u7xRAL5oOHZGxfYLiqPREyfyPsZJYvaw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb - how far back is it supposed to keep data?

2014-09-26 Thread Ken Barber

 Just wondering. I was messing about with some queries this morning.

 Asking for 'server/v3/reports --data-urlencode
 'query=[=,certname,client_name]' only returned about 2 weeks worth of
 reports. This system has been up and running for 6-8 months.

Whatever this value is set to:

https://docs.puppetlabs.com/puppetdb/2.2/configure.html#report-ttl

Or the default, which is 14 days.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DEJHHmwEsMw_kobJPv_1u%3DVXmr562X3SjytbNnGHo9dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Puppet facts uploading to PuppetDB

2014-09-26 Thread Ken Barber

Do many people use or care about the ability to upload facts out of
band to PuppetDB from a machine without the need for a full catalog
compilation? Its not a highly documented facility, ie. doesn't work
out of the box without configuration changes, but I know some people
have asked me this on IRC etc.

If you are using this facility, are you running masterless or with a
puppet master out of curiosity? Also - why are you using this
facility, what does it give you - what problems are you trying to
solve beyond just relying on the facts from a catalog compilation?

I'm just curious because some of this functionality is changing in a
future Puppet (basically it will stop working), trying to determine
whether its worth the time to port this functionality over to
something new. Or whether we should promote a new project outside of
the PL core items to do this (like a new module or plugin for
example).

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm%2BMVkwrhdMZ4JxHhEKrK_Pvxn%3D8egLvFkqRcngCO7kkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] PuppetDB 2.2.0 final now available

2014-08-27 Thread Ken Barber

PuppetDB 2.2.0 final - August 27th, 2014

PuppetDB 2.2.0 Downloads


Available in native package format in the release repositories at:
http://yum.puppetlabs.com and http://apt.puppetlabs.com

For information on how to enable the Puppet Labs repos, see:
http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#open-source-repositories

Binary tarball: http://downloads.puppetlabs.com/puppetdb/

Source: http://github.com/puppetlabs/puppetdb

Please report feedback via the Puppet Labs tickets site, using an
affected PuppetDB version of 2.2.0:
https://tickets.puppetlabs.com/browse/PDB

Documentation: http://docs.puppetlabs.com/puppetdb/2.2/

Puppet module:
http://forge.puppetlabs.com/puppetlabs/puppetdb

Release notes: https://docs.puppetlabs.com/puppetdb/2.2/release_notes.html

PuppetDB 2.2.0 Release Notes


This release was primarily focused on providing structured facts
support for PuppetDB. Structured facts
allow a user to include hashes and arrays in their fact data, but also
it provides the availability of
proper typing to include the storage of integers, floats, booleans as
well as strings.

This release introduces the ability to store structured facts in
PuppetDB, and use some new enhanced API's
to search and retrieve that data also.

With this change we have also introduced the capability to store and
retrieve trusted facts, which
are stored and retrieved in the same way as structured facts.

For more detailed information and upgrade advice, consult the detailed
release notes here:
https://docs.puppetlabs.com/puppetdb/2.2/release_notes.html

Contributors


Brian Cain, Eric Timmerman, Justin Holguin, Ken Barber, Nick
Fagerlund, Ryan Senior and Wyatt Alt.


Changelog
-

Brian Cain (1):
  5375b3a (PDB-550) Update puppetDB docs to include info on LibPQFactory

Eric Timmerman (1):
  9ee6c76 (PDB-823) Remove saucy from build defaults

Justin Holguin (1):
  d71dda5 DOCUMENT-97: Mention updating puppetdb module

Ken Barber (34):
  8f1d98c (PDB-47) Enable structured facts from facts terminus
  7b8ee3e Fix example for nodes endpoint to show 'certname' in response
  b3199e8 (PDB-747) Initial schema for structured facts support
  03fc9f9 (PDB-708) Structured fact storage
  6ae5257 (PDB-708) Add backwards compatible querying capability for facts
  33ce968 Allow empty arrays and hashes in structured facts
  6e2f055 Ensure factsets endpoint unescapes any delimiters
  1b0893a (maint) Split out acceptance and unit test gems in a better way
  bbe1976 (maint) Split out acceptance and unit test gems in a better way
  42f6ddf (maint) Switch confine for basic test during acc
dependency installation
  a6f3564 (PDB-709) Fact nodes endpoint
  631817c (maint) Switch confine for basic test during acc
dependency installation
  d1c2ab2 (maint) Fix an intermittent testing failure with time for factsets
  895032d (PDB-746) Add capability to use globs when querying fact-nodes
  f52959d (maint) Fix fact-node globbing tests
  6ab8c09 (PDB-763) Provide trgm index handling for fact_paths
  c68cc48 (PDB-798) Add capability to use regexp when querying
fact-nodes path types
  09f2f58 (maint) Fix old acceptance test refspec issue
  4e6faa0 (PDB-816) Unstringify facts
  8f271f8 (PDB-826) Fix pathing for puppetdb-legacy
  318a75c (PDB-809) Store top level structured fact data as JSON
  8a34686 (PDB-809) Use json stored branch for factsets also
  4c88393 (PDB-820) Remove glob operator
  580292f (maint) Remove more dead code from facts.clj
  35a58a2 (PDB-830) Regexp array operator is too greedy
  faed002 (PDB-769) Deprecate PostgreSQL 9.2 and older
  1fa294d (maint) Fix regexp array operator test failures for PostgreSQL
  31810cd (maint) Fix reference to greedy regexp operator
  44ddfe4 (PDB-834) Update documentation titles for release 2.2
  efa24b6 (docs) Update API Index to reflect all endpoints
  60312b5 Change name of fact-nodes operator to fact-contents
  7769948 Deprecate PG 9.1 instead, and remove nagging about 9.3
for PE users
  182b76f (PDB-783) Release notes for PuppetDB 2.2.0
  dee1433 (maint) Remove duplicate line after merge up

Nick Fagerlund (4):
  eb20757 (docs) Revise API docs for updated info, clarity,
consistency, and formatting
  a749705 (docs) Fix glitches noticed while backporting recent
revisions to 1.6.x
  69b95aa (docs) Port recent docs revisions to 1.6.x branch
  14ef0c8 (docs) Change some old URLs, remove mentions of inventory service.

Ryan Senior (2):
  2ac19ec Updated the release notes for PDB-675
  53a86bc (PDB-711) Initial work on factset end-point

wkalt (19):
  dd8d22c (PDB-488) Add producer timestamps to factsets and catalogs.
  c6c3041 Add deprecation warnings
  881e49a (PDB-762) Fix broken export
  dce93fa (PDB-565) Support PDB storage

Re: [Puppet Users] PuppetDB Catalog Duplication and Multiple Masters

2014-08-18 Thread Ken Barber

 Using multiple Puppet masters behind SRV records is working well although I
 suspect the low duplication rates I am seeing is down to the fact the load
 balancing is split between the nodes and the servername/serverip being
 recorded is different when hitting the other Puppet master in the pool ? I
 saw this by switching on the debug setting in PuppetDB and diffing the
 resulting JSON files.

Can you share the pertinent part of the diffs so we can see exactly
what differences you mean? I'm not sure I am aware of the
servername/serverip cases you mention in any core modules or core
resources, perhaps these are being introduced to a custom resource?

Can you also share a picture of your dashboard, so we can see the
duplication rate?

 Would that be the case, or is it something else I am not seeing ?

 Any ideas on how to improve this ?

Perhaps, with more details.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnWmUKiQfLFZQw2V9i5LCAHEvpdqexaTPcrGC-qmf%3DjQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppet CLA

2014-08-14 Thread Ken Barber

 I'm trying to sign this new github linked CLA and it's saying the my
 email address is already taken, which I'm guessing is because my
 puppetlabs and github accounts share a common email address. How can I
 get around this annoyance?

Can you try logging a ticket here?

https://tickets.puppetlabs.com/browse/CLA

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnpy-J33CjSuvJDoNJYky6noROBF0GOc%3DFaJpUuah6cjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetdb query. Search two classes

2014-08-11 Thread Ken Barber

 Thanks, but I says about query like this:

 [and,
   [and,
 [=, type, Class],
 [=, title, Php]],
   [and,
 [=, type, Class],
 [=, title, Nginx]]]

 Think about what this does behind the scenes on the resources endpoint
 (see the query here:
 https://gist.github.com/kbarber/d557da843b363ce3af3a#file-gistfile1-sql-L9-L13)
 , it ultimately is just going to attempt to find a class resource that
 has the title php AND nginx. Since a column can't have two values,
 this will return nothing. If you had a base OR at the top, it would
 return both the php and nginx classes, but this still isn't going to
 provide you everything, but at least it will give you enough
 information to break this down on the client end.

 In the SQL world, the way I might construct such a query, is to get
 all the class resources with nginx and php as the title, and then you
 would need a list of distinct certificate names for both, and you
 would want to find the overlap between the two, using an intersection
 query for example
 (http://www.postgresql.org/docs/9.3/static/queries-union.html). None
 of this is supported by the PuppetDB query API today.

Contrary to my last statement, this might work:
https://gist.github.com/kbarber/61d7c04f4d898148a06f

Deepak pointed that last one out. Let us know if it works.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmQX-E%2Bd8t71vVgENos9pXQR1VahSMmucHo_VoebgHjAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetdb query. Search two classes

2014-08-08 Thread Ken Barber

 Thanks, but I says about query like this:

 [and,
   [and,
 [=, type, Class],
 [=, title, Php]],
   [and,
 [=, type, Class],
 [=, title, Nginx]]]

Think about what this does behind the scenes on the resources endpoint
(see the query here:
https://gist.github.com/kbarber/d557da843b363ce3af3a#file-gistfile1-sql-L9-L13)
, it ultimately is just going to attempt to find a class resource that
has the title php AND nginx. Since a column can't have two values,
this will return nothing. If you had a base OR at the top, it would
return both the php and nginx classes, but this still isn't going to
provide you everything, but at least it will give you enough
information to break this down on the client end.

In the SQL world, the way I might construct such a query, is to get
all the class resources with nginx and php as the title, and then you
would need a list of distinct certificate names for both, and you
would want to find the overlap between the two, using an intersection
query for example
(http://www.postgresql.org/docs/9.3/static/queries-union.html). None
of this is supported by the PuppetDB query API today.

 Unfortunately, this query doesn't work.

 I use generate function and my own python script for puppetdb query. My
 question is how get list of nodes with two resources in one query.

Take another look at my suggestion:
https://gist.github.com/kbarber/6840f5b1a74d985d8167. It _is_ one
query. There is no shame in doing some work on the client side to
finish the job, you should be able to accomplish this last step in
Python quite easily.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTnm0N038Xg6vScyzwvU%3Dfz9DpBnxVnp7E6c5bCFU7DUag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetdb query. Search two classes

2014-08-07 Thread Ken Barber

I might be thinking about this the wrong way, but I think the API can
only do so much on the server side to achieve this. In particular I
want to do a distinct aggregation but we don't support that.

Fortunately, this is achievable on the command line with a tool like
JGrep: http://jgrep.org/#howto

Here is how I would do it:

https://gist.github.com/kbarber/6840f5b1a74d985d8167

Any entry that has a 2 next to it, meets the criteria basically.

ken.

On Thu, Aug 7, 2014 at 10:02 AM, Maksim Podlesniy cryptspi...@gmail.com wrote:
How I may find all nodes with two specified classes? Example: find all nodes
with class Nginx and Php

Sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/8ae479f4-c700-490f-83d3-4e6ec1df292a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DDwVDBUyC7VT8XmyQqoqJ-T-gbZzfY-MqaERCGP9ZVRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Reports from puppet

2014-07-21 Thread Ken Barber

Hi Maxim,

This is not directly reproducible by myself today:
https://gist.github.com/kbarber/c6941099bea07096361e ...

Perhaps something in your puppet.conf is doing this, I could imagine
something like:

usecacheonfailure = true

Causing this to happen, but I can't reproduce the exact same conditions myself.

Here is my basic puppet.conf fwiw:

# cat /etc/puppet/puppet.conf
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter

[agent]
report = true

[master]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb
trusted_node_data = true

Perhaps you can share your so we can see what settings may be causing it.

ken.

On Mon, Jul 21, 2014 at 12:55 PM, Maxim Nikolaev m...@maximnik.com wrote:
Hi

I'm using Puppet with Dashboard and PuppetDB and Puppetdb board. I can see
all nodes and rfeports.
Problem is that when puppet fail to run on instance - i get report
unchanged instead of fail.

For ex. I've changed postfix manifest to install package postfix1. Puppet
failed to run:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Invalid relationship: File[/etc/postfix/main.cf] { require =
Package[postfix] }, because Package[postfix] doesn't seem to be in the
catalog
But I got unchanged report instead of failed also in dashboard and in
puppetdb.

Puppet: 3.6.2
Facter: 2.1.0
OS: Amazon Linux

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/29057799-8d81-4272-b02d-8df2f8ba38f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3Dx%3DO0k2vHeuUJ7x%3DNsu5aOux_sY6OzJuKj9e%3DiRyqCUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Reports from puppet

2014-07-21 Thread Ken Barber

Are we even sure this is a PuppetDB problem? And not an issue with the
Dashboard itself? As I've shown, the report gets stored in PuppetDB as
a failure. Having said that, if there is an issue in the summary of
the report we don't store those stats ourselves in PuppetDB.

Maxim, I'd do the test I provided in my gist, that is - the curl
request against PuppetDB to confirm this is true for PuppetDB at
least. For 2.1.0 and above this should work fine. At the very least we
need to determine where this fault is. It sounds like PUP-283, but the
latest version of Puppet Board shouldn't suffer this issue since its
not querying against the Dashboard, but PuppetDB directly.

ken.

On Mon, Jul 21, 2014 at 5:31 PM, José Luis Ledesma
joseluis.lede...@gmail.com wrote:
Which version of puppedb are you running? If I'm not wrong, from puppetdb
2.1 catalog compilation errors are stored , but not for previous versions.

Regards,

El 21/07/2014 17:17, Maxim Nikolaev m...@maximnik.com escribió:

No. I haven't this feature.
my puppet.conf looks like this:

[main]
modules = /etc/puppet/modules
hiera_config = /etc/puppet/hiera.yaml
pluginsync= true
server = HOSTNAME
#port = 8081

[agent]
server = HOSTNAME
report = true
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig

[master]
certname = HOSTNAME
dns_alt_names = HOSTNAME
autosign = true
reports = store, http, puppetdb, tagmail
reporturl = http://HOSTNAME/reports/upload
hiera_config = /etc/puppet/hiera.yaml
storeconfigs = true
storeconfigs_backend = puppetdb

I found discussion about this bug:

http://projects.theforeman.org/issues/3851

But I want to know if there is any workaround.

If catalog fails on client side - I can see error reports.

On Monday, July 21, 2014 4:01:43 PM UTC+3, Ken Barber wrote:

Hi Maxim,

This is not directly reproducible by myself today:
https://gist.github.com/kbarber/c6941099bea07096361e ...

Perhaps something in your puppet.conf is doing this, I could imagine
something like:

usecacheonfailure = true

Causing this to happen, but I can't reproduce the exact same conditions
myself.

Here is my basic puppet.conf fwiw:

# cat /etc/puppet/puppet.conf
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter

[agent]
report = true

[master]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb
trusted_node_data = true

Perhaps you can share your so we can see what settings may be causing it.

ken.

On Mon, Jul 21, 2014 at 12:55 PM, Maxim Nikolaev m...@maximnik.com
wrote:
Hi

I'm using Puppet with Dashboard and PuppetDB and Puppetdb board. I can
see
all nodes and rfeports.
Problem is that when puppet fail to run on instance - i get report
unchanged instead of fail.

For ex. I've changed postfix manifest to install package postfix1.
Puppet
failed to run:

Error: Could not retrieve catalog from remote server: Error 400 on
SERVER:
Invalid relationship: File[/etc/postfix/main.cf] { require =
Package[postfix] }, because Package[postfix] doesn't seem to be in the
catalog
But I got unchanged report instead of failed also in dashboard and in
puppetdb.

Puppet: 3.6.2
Facter: 2.1.0
OS: Amazon Linux

https://groups.google.com/d/msgid/puppet-users/29057799-8d81-4272-b02d-8df2f8ba38f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/f9cde2d1-0d85-49ec-a5bd-2582dd01ff16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAF_B3dcBY5T5sEisZrK%3DgyF9woaZiCqdT_hw5zFxDCCUyAr1yQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb export before migration

2014-07-18 Thread Ken Barber

Does this sound like your issue?

https://tickets.puppetlabs.com/browse/PDB-762

We found it recently and have already fixed it in source, but not
shipped a fix yet. We were holding off for someone complaining loud
enough or just shipping it with 2.2.0 (which should be out in a few
weeks or so).

ken.

On Fri, Jul 18, 2014 at 5:42 PM, Fabrice Bacchella
fbacche...@spamcop.net wrote:
 I ran an  puppetdb export yesterday. It ran fine. It was a puppetdb 2.0

 Now, after an export to puppetdb 2.1, I got some strange exception. puppet 
 master is working fine, I didn't see anything special in the release notes. 
 puppetmaster is running fine.

 Did I miss something ?

 $ puppetdb export --outfile ./my-puppetdb-export.tar.gz
 java.lang.AssertionError: Assert failed: %
  at com.puppetlabs.puppetdb.cli.export$events_for_report_hash.invoke 
 (export.clj:114)
 com.puppetlabs.puppetdb.cli.export$reports_for_node$fn__5294.invoke 
 (export.clj:150)
 clojure.core$map$fn__4245.invoke (core.clj:2557)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:484)
 clojure.core$seq.invoke (core.clj:133)
 clojure.core$map$fn__4245.invoke (core.clj:2551)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:484)
 clojure.core$seq.invoke (core.clj:133)
 clojure.core$map$fn__4245.invoke (core.clj:2551)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:484)
 clojure.core$seq.invoke (core.clj:133)
 clojure.core.protocols$seq_reduce.invoke (protocols.clj:30)
 clojure.core.protocols/fn (protocols.clj:54)
 clojure.core.protocols$fn__6031$G__6026__6044.invoke (protocols.clj:13)
 clojure.core$reduce.invoke (core.clj:6289)
 schema.core$eval3377$fn__3395$fn__3412.invoke (core.clj:787)
 schema.core$eval3377$fn__3395$fn__3412.invoke (core.clj:785)
 clojure.core$comp$fn__4192.invoke (core.clj:2403)
 com.puppetlabs.puppetdb.cli.export$eval5305$report__GT_tar__5306.invoke 
 (export.clj:153)
 
 com.puppetlabs.puppetdb.cli.export$eval5329$get_node_data__5330$fn__5334.invoke
  (export.clj:184)
 com.puppetlabs.puppetdb.cli.export$eval5329$get_node_data__5330.invoke 
 (export.clj:170)
 com.puppetlabs.puppetdb.cli.export$_main.doInvoke (export.clj:241)
 clojure.lang.RestFn.invoke (RestFn.java:421)
 clojure.lang.Var.invoke (Var.java:383)
 clojure.lang.AFn.applyToHelper (AFn.java:156)
 clojure.lang.Var.applyTo (Var.java:700)
 clojure.core$apply.invoke (core.clj:624)
 com.puppetlabs.puppetdb.core$run_command.invoke (core.clj:87)
 com.puppetlabs.puppetdb.core$_main.doInvoke (core.clj:95)
 clojure.lang.RestFn.invoke (RestFn.java:436)
 clojure.lang.Var.invoke (Var.java:388)
 clojure.lang.AFn.applyToHelper (AFn.java:160)
 clojure.lang.Var.applyTo (Var.java:700)
 clojure.core$apply.invoke (core.clj:624)
 clojure.main$main_opt.invoke (main.clj:315)
 clojure.main$main.doInvoke (main.clj:420)
 clojure.lang.RestFn.invoke (RestFn.java:482)
 clojure.lang.Var.invoke (Var.java:401)
 clojure.lang.AFn.applyToHelper (AFn.java:171)
 clojure.lang.Var.applyTo (Var.java:700)
 clojure.main.main (main.java:37)
 2014-07-18 18:35:57,650 ERROR [p.t.logging] Uncaught exception
 java.lang.AssertionError: Assert failed: %
 at 
 com.puppetlabs.puppetdb.cli.export$events_for_report_hash.invoke(export.clj:114)
  ~[na:na]
 at 
 com.puppetlabs.puppetdb.cli.export$reports_for_node$fn__5294.invoke(export.clj:150)
  ~[na:na]
 at clojure.core$map$fn__4245.invoke(core.clj:2557) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[puppetdb.jar:na]
 at clojure.lang.RT.seq(RT.java:484) ~[puppetdb.jar:na]
 at clojure.core$seq.invoke(core.clj:133) ~[puppetdb.jar:na]
 at clojure.core$map$fn__4245.invoke(core.clj:2551) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[puppetdb.jar:na]
 at clojure.lang.RT.seq(RT.java:484) ~[puppetdb.jar:na]
 at clojure.core$seq.invoke(core.clj:133) ~[puppetdb.jar:na]
 at clojure.core$map$fn__4245.invoke(core.clj:2551) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[puppetdb.jar:na]
 at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[puppetdb.jar:na]
 at clojure.lang.RT.seq(RT.java:484) ~[puppetdb.jar:na]
 at clojure.core$seq.invoke(core.clj:133) ~[puppetdb.jar:na]
 at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) 
 ~[puppetdb.jar:na]
 at clojure.core.protocols$fn__6078.invoke(protocols.clj:54) 
 ~[puppetdb.jar:na]
 at

Re: [Puppet Users] puppetdb and client certificate

2014-07-17 Thread Ken Barber

Aah well done :-).

Perhaps you can create a ticket with these details in it, that way it
will be preserved for other users and we can later find some time to
extend the current docs:

https://tickets.puppetlabs.com/browse/PDB

Of course you can always raise a pull request with the doc changes
yourself, if thats possible it would be greatly appreciated :-).

https://github.com/puppetlabs/puppetdb/blob/master/documentation/postgres_ssl.markdown

ken.

On Thu, Jul 17, 2014 at 12:03 PM, Fabrice Bacchella
fbacche...@spamcop.net wrote:
I got it !

I dit it using the java way (with a standard jks).

First create a jks with the private key for your account and put in it all
the needed certificates in the chain (both server and user). The cn for user
certificate should match the username used latter.

Add to your JVM args :
-Djavax.net.ssl.trustStore=.../puppetdb.jks
-Djavax.net.ssl.trustStorePassword=JKS password
-Djavax.net.ssl.keyStore=.../puppetdb.jks
-Djavax.net.ssl.keyStorePassword=JKS password

In case of problems, -Djavax.net.debug=ssl,defaultctx might help.
My database.ini is :
[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //localhost:5432/puppetdb?ssl=true
log-slow-statements = 10
username = puppetdb

In pg_hba.conf, I added :
hostssl allall 0.0.0.0/0 cert clientcert=1

And in postgresql.conf :
ssl = on
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_ca_file = 'root.crt'

The file root.crt contains all the needed certificates (both client and
server)
The file server.crt contains only the server certificate
The file server.key contains the private key.

Those 3 files are stored as PEM files.

org.postgresql.ssl.LibPQFactory is used if you want to mimic the psql client
configuration and use PEM files instead of JKS, and it take the same
arguments, some documentation can be found at :
http://www.postgresql.org/docs/8.4/static/libpq-connect.html#LIBPQ-CONNECT-SSLMODE

Le 16 juil. 2014 à 17:05, Ken Barber k...@puppetlabs.com a écrit :

I wrote that document, at the time client based certificates weren't
really supported or something like that.

Specifically not supporting client auth is hinted in the JDBC driver
details here: http://jdbc.postgresql.org/documentation/head/ssl-factory.html

I seem to recall there being a problem with the current default
SslFactory not allowing this kind of thing easily, but there is an
alternate factory one can possibly use:

https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/ssl/jdbc4/LibPQFactory.java

Sorry, the main documentation page doesn't contain details of this
class, its fairly unknown to most people - I had to go digging for it
in source to find it.

This should in theory allow one to provide parameters in the JDBC url
to provide a client certificate, but I've never seen it working yet
(and we may not even ship a version of the jdbc driver that has this
class :-).

The parameters gleaned from the source are:

sslmode,sslcert,sslkey,sslrootcert,sslhostnameverifier,sslpasswordcallback,sslpassword

So you could construct something like:

[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname =
//HOST:PORT/DATABASE?ssl=truesslfactory=org.postgresql.ssl.LibPQFactorysslmode=whateversslcert=whatever
... etc ...
username = USERNAME
password = PASSWORD

But the details are something I'm not across sorry, you'll have to
experiment on your own :-). Respond if you need more help, or if you
make more progress.

ken.

On Wed, Jul 16, 2014 at 3:40 PM, Fabrice Bacchella
fbacche...@spamcop.net wrote:
I'm trying to authenticate a puppetdb on a postgresql server using a client
certificate.

I managed to set up SSL on server side without any problem but
http://docs.puppetlabs.com/puppetdb/2.1/postgres_ssl.html says :
Note: At this point the documentation below only covers server-based SSL,
client certificate support is not documented.

So did someone with a better knowledge of the postgresql jdbc connector did
try this and succeed ?

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/1082A318-AA8C-4C47-BAE9-4A980DD1D4D6%40spamcop.net.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkEqRYWCJQ8JgFBPX8F9A-ZgkD5h5Pxn7_3foHCa5BWgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout

Re: [Puppet Users] puppetdb and client certificate

2014-07-16 Thread Ken Barber

I wrote that document, at the time client based certificates weren't
really supported or something like that.

Specifically not supporting client auth is hinted in the JDBC driver
details here: http://jdbc.postgresql.org/documentation/head/ssl-factory.html

I seem to recall there being a problem with the current default
SslFactory not allowing this kind of thing easily, but there is an
alternate factory one can possibly use:

https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/ssl/jdbc4/LibPQFactory.java

Sorry, the main documentation page doesn't contain details of this
class, its fairly unknown to most people - I had to go digging for it
in source to find it.

The parameters gleaned from the source are:

sslmode,sslcert,sslkey,sslrootcert,sslhostnameverifier,sslpasswordcallback,sslpassword

So you could construct something like:

But the details are something I'm not across sorry, you'll have to
experiment on your own :-). Respond if you need more help, or if you
make more progress.

ken.

On Wed, Jul 16, 2014 at 3:40 PM, Fabrice Bacchella
fbacche...@spamcop.net wrote:
I'm trying to authenticate a puppetdb on a postgresql server using a client
certificate.

So did someone with a better knowledge of the postgresql jdbc connector did
try this and succeed ?

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/1082A318-AA8C-4C47-BAE9-4A980DD1D4D6%40spamcop.net.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkEqRYWCJQ8JgFBPX8F9A-ZgkD5h5Pxn7_3foHCa5BWgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] PuppetDB 2.1.0 final now available

2014-07-08 Thread Ken Barber

---
Chris Price, Eric Timmerman, Ken Barber, Melissa Stone, Niels Abspoel,
Ryan Senior, Wyatt Alt

PuppetDB 2.1.0 Changlog
---

Chris Price (7):
  c7628d2 Allow web app URL prefix to be configured
  0b38aac Support for configurable URL in terminus
  f584224 (PDB-651) Fix dashboard to be compatible with url prefix
  23e0e26 Add deprecation warning to docs for `explicit` metric type strings
  a69bf71 (PDB-651) Add docs for url-prefix setting
  a60b843 Update `with-http-app` fixture to include default for url-prefix
  94ebe0a (PDB-651) Fix spec failure on ruby 1.8.7

Eric Timmerman (2):
  c619cec (RE-1497) Remove quantal from build_defaults
  17af083 (RE-1497) Remove quantal from build_defaults

Ken Barber (24):
  f81ac21 Update release notes with more upgrade warnings
  93817db (PDB-16) Fix terminus to always submit reports and
include the status
  650bc7e (PDB-16) Include warning when environment doesn't exist in report
  4ed1e81 Fix reference to PuppetDB 1.6.x in index.markdown
  871086b (PDB-551) Include a versioning policy
  3bde8ac (PDB-551) Some minor cleanups to the versioning policy
  530ac79 (PDB-551) Reduce wording based on feedback
  98e8315 Unpin the version of beaker
  0f44e0c (PDB-597) Add trusty build default
  3623e45 Fix comparison in dup resources acceptance test
  3f9b81c (PDB-467) Collapse versioned files for http tests
  adb4fcd (PDB-467) Create deftestseq and implement it everywhere
  61764ae (PDB-686) Add warning about PDB-686 to release notes
  388d0c6 Typo in v4/reports.markdown
  edbef54 (PDB-467) Convert endpoints to vecs of vecs
  0957425 (PDB-660) Enable event streaming and provide other
streaming cleanups
  ee011c5 (PDB-660) Throw deprecation warning for
'event-query-limit' and fix a docstring
  5e2449b (PDB-660) Turn on streaming for nodes end-point
  c9df21c (PDB-660) Environments end-point streaming
  ab02b8e (PDB-660) Provide better validation for query params on
environments  singleton end-points
  271dc51 (PDB-660) Report end-point streaming
  fcd629e (PDB-660) Remove extraneous parens
  9f5967b (PDB-660) Event-counts streaming
  03767ad (PDB-660) Final refactor for streaming

Melissa Stone (3):
  466e4cd (PDB-675) Stop pdb process even if pidfile missing
  a6e5a21 (maint) Restrict which process will be stopped
  c90e765 (maint) Add log entry for when process stopped without PIDfile

Niels Abspoel (1):
  1e021db sbin_dir logic to Rakefile

Ryan Senior (17):
  5d8d204 (PDB-309) Update config conversion code for Schema 0.2.1
  128934c Fix a race condition in the import/export round-trip clojure tests
  890f31e (PDB-16) Added status to store report and report query results
  0af169f (PDB-657) Query logic refactor
  910e48c (PDB-658) Move reports endpoint over to the new query engine
  df7eb02 (PDB-658) Move facts endpoint to new query engine
  85d50b9 (PDB-658) Moving events querying to the new query engine
  ee55859 (PDB-658) Moved event-counts endpoints to the query engine
  b4667cf (PDB-658) Move environments endpoint to the query engine
  a7a3ac2 (PDB-697) Test and document new reports features
  0feb553 (PDB-697) Added docs/tests for new queryable nodes
fields and operators
  6455bec (PDB-660) Fix testing for not(regexp) match case
  96d5af9 (PDB-697) Document and test new events and resources features
  a235006 (PDB-162) Add regexp support to resource parameter queries
  66ccddd7 (PDB-720) Fix services test with hard coded Jetty port
  48e1eb5 (PDB-728) Update release notes for 2.1.0
  b82f58f (PDB-729) Updated documentation indexes for PuppetDB 2.1

wkalt (8):
  b9e3459 (PDB-700) Allow changes to maxFrameSize in activemq
  598444f (PDB-577) Lower KahaDB MessageDatabase logging threshold.
  c964d91 (PDB-130) Fixes a nasty traceback exposed when users run
import from command line with an invalid filename. A friendly message
is now printed instead.
  7fb9418 (PDB-337) Remove extraneous _timestamp fact
  0454e09 (PDB-220) Coerce numerical function output in manifests to string
  aefc579 (PDB-240) Replace anonymize.clj read-string with
clojure.edn/read-string
  ab71ba2 (PDB-164) Add documentation for select-nodes subquery operator
  1a90ff7 (PDB-601) Do not require query operator on reports endpoint

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmrF6au3T0yDtQjq%2B%3DiH5ogUBmu%2BxXKXoZTw1cmqfzkjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Puppetlabs Firewall

2014-07-04 Thread Ken Barber

So puppetlabs-firewall is an active provider, whenever it 'runs' in
the catalog it applies the rule straight away. You are probably seeing
this because you're applying a blocking rule (like a DROP or default
DROP for the table) before the SSH allowance rule gets applied.

Take a close look at the pre/post suggestion here:
https://forge.puppetlabs.com/puppetlabs/firewall#beginning-with-firewall

Notice how it suggests creating a course-grained ordering to setup the
DROP rule as the very last thing that runs. Now to be clear this
concept is about puppet resource execution order, not the order which
the rules are setup in iptables (ie. with the number in the title).

ken.

On Fri, Jul 4, 2014 at 1:19 PM, Danny Roberts
dannyroberts.perso...@googlemail.com wrote:
To clarify; we have to use SSH to connect to the servers in this
environment, they are all VMs the hosting provider does not give any means
of accessing a console (not ideal but sadly beyond our control).

Our standard process is after building a new server to have manually run
Puppet once to bring it up to our standard ASAP. Normally Puppet runs
daemonized beyond this point.

This is our first production environmnet that uses the Puppetlabs Firewall
module so our first time encountering this in anger. Oddly the server
remains unreachable via SSH after this for at least 2 hours which is enough
for 3/4 Puppet runs to sort out any issues. This still seems a bit long.

I'm about to try another test by stopped the firewall before doing another
Puppet run on a fresh server to see how that behaves.

On Wednesday, 2 July 2014 14:27:05 UTC+1, jcbollinger wrote:

On Tuesday, July 1, 2014 9:30:57 AM UTC-5, Danny Roberts wrote:

I am using the Puppetlabs firewall module to manage our firewall. All
servers get our core ruleset:
[...]
This worked perfectly when I spun up a server with no role (and therefore
no extra rules. However when I spun up servers with the 'puppet'
'database' roles (and therefore the extra rules) it hung at:

Notice: /Stage[main]/Mycompany/Firewall[9001
fe701ab7ca74bd49f13b9f0ab39f3254]/ensure: removed

My SSH session eventually disconnects with a broken pipe. The puppet
server I spun up yesterday was available when I got into the office this
morning so it seems they do eventually come back but it takes some time. Is
there any reason I am getting cut of like that and is there any way to avoid
it?

I'm a little confused. What does your SSH session have to do with it? I
don't find it especially surprising that an existing SSH connection gets
severed when the destination machine's firewall is manipulated by Puppet, if
that's what you're describing. I would not necessarily have predicted it,
but in retrospect it seems reasonable.

I'm supposing that you were connected remotely via SSH to the machine on
which the agent was running, following the progress of the run in real time.
In that case, are you certain that the run was in fact interrupted at all?
Maybe the output from the remote side was curtailed when your SSH connection
was disrupted, but the run continued. Or if you were running un-daemonized,
then perhaps the run was interrupted when severing the SSH connection
produced a forced logout from the controlling terminal.

Any way around, the fact that the subject systems eventually recover on
their own makes me suspect that the problem lies in how you were monitoring
the run, rather than in your manifests. You could try running puppet in
daemon mode, or otherwise disconnected from a terminal, and checking the log
after the fact to make sure everything went as it should.

John

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/5eb83bdd-c8a5-4e36-956d-ff87eafd7acb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm94EFtp5ZsB9NzgYY0R4CRoEfkE84T1HX2ue4J3%2B8M3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] how to force an error

2014-07-03 Thread Ken Barber

Does exist some way to force a  error while applying the catalog? We need
 to check some facts vs configuration and force an error if doesnt' match, so
 we could have a report from the node in the puppetdb with the failed state.
 (we cannot use a compilation/evaluation error because it doesn't report to
 the puppetdb)

This is fixed in the next release, 2.1.0.

   Currently we use this ugly workaround:

 exec {'dummy exec to force an error':
   path = ['/bin'],
 }

   Is there any better solution?

I know its vapourware until its released, but I'd wait for 2.1.0 and
use the fail function,

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkvx%2BFQKPXVcqp5AURUS8c2s-eOSuVFOLVBYdva_wAc2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] PuppetDB: Problems with PID file missing

2014-07-02 Thread Ken Barber

This is probably a long shot ... but we have this ticket here for PuppetDB:

https://tickets.puppetlabs.com/browse/PDB-675

It describes a scenario where when the PID file is missing, the
service script is unable to stop the running process. Now our solution
is simple, we have a fallback to kill any java process owned by
puppetdb if the PID file is missing, so easily worked around really
... however ...

The issue I wanted to talk about though, is why this is happening.

I'm looking for people who perhaps felt this kind of 'restart' pain
during upgrade. I had heaps of people mentioning this to me on the IRC
#puppet channel a few weeks ago, but it seems to have dried up. I was
foolish not to confirm the issue there and then, as I thought it was
easily replicable but its proven more elusive.

Has someone ran into the missing PID issue? Can you help me try to
track down the issue? The only information I have is that it was
during an upgrade, and only on Debian based boxes.

Thanks.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTk6eFgUpGotu8DNCErOrjJ%2B2870R7kF0iW41ryOufR8mQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb '[replace catalog]' postgresql errors with index 'catalog_resources_pkey'

2014-07-01 Thread Ken Barber

Thanks mate :-).

On Tue, Jul 1, 2014 at 7:57 PM, Mathew Crane mathew.cr...@gmail.com wrote:

 No problem :-).

 Can you raise a bug on the original exec {} issue for me?
 https://tickets.puppetlabs.com/browse/PDB

 ken.


 https://tickets.puppetlabs.com/browse/PDB-742


 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/puppet-users/cffe9c19-f419-47d2-8793-65ba4506db89%40googlegroups.com.

 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm7_hhfD047Ngo9aiXCMbJY0JprOjDAiQOXCgKrz8ktaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetdb not receiving any catalog updates.

2014-06-30 Thread Ken Barber

Well that looks fine. How odd.

If you run the puppet master in debug mode, (something like this):

puppet master --no-daemonize --debug --trace

What do you get?

My results are here:

https://gist.github.com/kbarber/a70eee998ec8ae1acf8e

You can clearly see the replace catalog attempt. I wonder if the debug
log is going to show us more information in your case.

ken.

On Thu, Jun 26, 2014 at 10:59 AM, Jelle B. jelle...@gmail.com wrote:
http://pastebin.com/UnNxw0n6

There you go all in one

Cleaned soem comments out to reduce the size a bit , but I think it is
relativlly vanilla

On Wednesday, June 25, 2014 7:25:57 PM UTC+2, Ken Barber wrote:

Been fighting witht his now for a bit , and IRC didnt seem to have any
answers so thought I would ask here.

I have 2 platforms , one prod one pre-prod the pre-prod runs basiclly
the
latest versions puppetdb 2 and puppet 3.62 for clients and master. This
platform is also where I am having problems.

in a normal run I would expect to see this in my puppetdb log :
2014-06-25 13:00:04,196 INFO [command-proc-52] [puppetdb.command]
[5d54df68-4ccb-439b-affe-5549a2dfb0fa] [replace facts] ostack-dash
2014-06-25 13:00:05,542 INFO [command-proc-52] [puppetdb.command]
[f43fca51-5964-4b00-a092-1afbc3a020f3] [replace catalog] ostack-dash
2014-06-25 13:00:09,303 INFO [command-proc-52] [puppetdb.command]
[27d43a8a-527c-4fc2-8cda-6547069ab0d4] [store report] puppet v3.5.1 -
ostack-dash

But I only see this :
2014-06-25 11:16:45,882 INFO [c.p.p.command]
[118ed328-f50a-47e4-a0a8-caa1cb143a64] [replace facts] lab-vcache1
2014-06-25 11:17:09,274 INFO [c.p.p.command]
[9104aa00-8215-431c-8cda-0f5fc4c24dad] [store report] puppet v3.6.2 -
lab-vcache1

I never get a replace catalog entry so needles to say my resources are
also
empty , this is no major issue , I think but and inconsistancy I would
liek
to resolve before it becomes a problem.

Now I have compared all my configs etc and they are all identical , and
I
sue the same classes in both so there is not differences there either,
the
only difference are versions, so is this no longer functional as off
puppet
3.?? and/or pupeptdb 2.? or am I missing somethign ?

It should work fine.

Can I please see your /etc/puppet/puppet.conf on your puppet master
host, and your routes file (/etc/puppet/routes.yaml) ... and for good
measure your /etc/puppet/puppetdb.conf file as well. Place these in a
gist/pastie for accuracy perhaps, as opposed to pasting them inline
here.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/c500b4d5-b938-4134-9018-86e5ebfd50f0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DJO7n-pgA5CTurKcYbjX%3D%2Bftt-C3djdMCwN%2BRmw_Vusw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppetdb not receiving any catalog updates.

2014-06-25 Thread Ken Barber

 Been fighting witht his now for a bit , and IRC didnt seem to have any
 answers so thought I would ask here.

 I have 2 platforms , one prod one pre-prod the pre-prod runs basiclly the
 latest versions puppetdb 2 and puppet 3.62 for clients and master. This
 platform is also where I am having problems.

 in a normal run I would expect to see this in my puppetdb log :
 2014-06-25 13:00:04,196 INFO  [command-proc-52] [puppetdb.command]
 [5d54df68-4ccb-439b-affe-5549a2dfb0fa] [replace facts] ostack-dash
 2014-06-25 13:00:05,542 INFO  [command-proc-52] [puppetdb.command]
 [f43fca51-5964-4b00-a092-1afbc3a020f3] [replace catalog] ostack-dash
 2014-06-25 13:00:09,303 INFO  [command-proc-52] [puppetdb.command]
 [27d43a8a-527c-4fc2-8cda-6547069ab0d4] [store report] puppet v3.5.1 -
 ostack-dash

 But I only see this :
 2014-06-25 11:16:45,882 INFO  [c.p.p.command]
 [118ed328-f50a-47e4-a0a8-caa1cb143a64] [replace facts] lab-vcache1
 2014-06-25 11:17:09,274 INFO  [c.p.p.command]
 [9104aa00-8215-431c-8cda-0f5fc4c24dad] [store report] puppet v3.6.2 -
 lab-vcache1

 I never get a replace catalog entry so needles to say my resources are also
 empty , this is no major issue , I think but and inconsistancy I would liek
 to resolve before it becomes a problem.

 Now I have compared all my configs etc and they are all identical , and I
 sue the same classes in both so there is not differences there either, the
 only difference are versions, so is this no longer functional as off puppet
 3.?? and/or pupeptdb 2.? or am I missing somethign ?

It should work fine.

Can I please see your /etc/puppet/puppet.conf on your puppet master
host, and your routes file (/etc/puppet/routes.yaml) ... and for good
measure your /etc/puppet/puppetdb.conf file as well. Place these in a
gist/pastie for accuracy perhaps, as opposed to pasting them inline
here.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DLuCYuCygppR3Vino42oq%3DfS7i%2BbJUHec-OM68TfyNOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-25 Thread Ken Barber

 Did you ever fix this. I am having the same problem.

 It is an issue with SSL.  Chrome gives this error: Error code:
 ERR_SSL_PROTOCOL_ERROR.
 Curl gives this error curl: (35) Unknown SSL protocol error in connection

 If I figure more out, I will post the fix.

These two errors are unrelated to the original problem described in
this thread, perhaps start a new thread?

Chrome will potentially throw an error here, because PuppetDB needs
client certificates to work, so nothing abnormal here - using chrome
is just not a good test unless you have the knowledge to load up the
CA and client certs yourself (not recommended really). I'd have to see
the full curl request to confirm, but it looks like you're tapping a
non-SSL port or somehow the openssl client is constraining the
protocols.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DCQb%3D25w%2BWdSBQeAkz%3DBQXWjLqMYm0Me_aWWmpfZ3dQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-25 Thread Ken Barber

Did you figure this out Sans?

My only next logical step would be to have you show me a full git
repository with a working copy of the code/vagrantfile/etc. that is
actually breaking.

ken.

On Wed, Jun 18, 2014 at 7:17 AM, Sans r.santanu@gmail.com wrote:
 Thanks Rakesh!
 But, as you probably can guess, that didn't change a thing; still getting
 the same error. I'm not sure if PuppetDB look in the DNS record (as opposed
 to hosts file), in that case it never gonna work. Best!!

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/puppet-users/9c9cc6c3-1059-4b67-abd8-29f8d77e8527%40googlegroups.com.

 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmeOQ5hr_dHCWye-Vhe34w-XMcZBtPQ6T6GCjNv7bggiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-17 Thread Ken Barber

Just started using PuppetDB (using the Puppetlabs' module) and getting
issues with connection. First it was giving me server Not Found:

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
Not Found
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
Not Found
Notice: Failed to connect to puppetdb within timeout window of 15 seconds;
giving up.
Error: Unable to connect to puppetdb server! (puppet.internal:8081)
Error:
/Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure:
change from absent to present failed: Unable to connect to puppetdb server!
(puppet.internal:8081)

This has retried to connect a few times then given up because of ...

since then, I'm still getting eventually the same error but with different
Notice:

Warning: Puppet::Util::SUIDManager.run_and_capture is deprecated; please
use Puppet::Util::Execution.execute instead.
(at
/vagrant/VagrantConf/modules/postgresql/lib/puppet/provider/postgresql_psql/ruby.rb:57:in
`run_sql_command')
Notice:
/Stage[main]/Puppetdb::Server::Jetty_ini/Ini_setting[puppetdb_sslhost]/value:
value changed '0.0.0.0' to 'puppet.internal'
Info: Class[Puppetdb::Server::Jetty_ini]: Scheduling refresh of
Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server/Service[puppetdb]: Triggered
'refresh' from 1 events
Notice: Unable to connect to puppetdb server (puppet.internal:8081):
#Errno::ECONNREFUSED: Connection refused - connect(2)
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry

Notice: Unable to connect to puppetdb server (puppet.internal:8081):
#Errno::ECONNREFUSED: Connection refused - connect(2)
Notice: Failed to connect to puppetdb within timeout window of 15 seconds;
giving up.
Error: Unable to connect to puppetdb server! (puppet.internal:8081)
Error:
/Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure:
change from absent to present failed: Unable to connect to puppetdb server!
(puppet.internal:8081)

Can anyone explain to me what actually going on/wrong please? Why it was
Not Found before and now Unable to connect? Any help/pointer would be
much appreciated. Best!

... connection refused. This is the key error. Its a common TCP error,
and is something all administrators should know, because it doesn't
just affect PuppetDB.

What it means is that the hostname and port the client is trying to
connect to, in this case puppet.internal and 8081 respectively, is
either not the correct host or port, PuppetDB is not listening
correctly to this host or port because it is down or misconfigured or
some firewall is rejecting the connection (less likely, but worth
noting).

So what you need to check:

* From your puppet master try telnetting into the port and show your
results. telnet puppet.internal 8081
* Check that the PuppetDB instance is really listening on port 8081
and on a public interface on that host, you can usually check this
with netstat -anp | grep 8081 on the PDB host. Provide the results in
the thread if you can.
* Check the settings in your /etc/puppetdb/conf.d/jetty.ini, and
ensure that ssl-host is set to something like 0.0.0.0 to listen on all
interfaces. You can always refine this once you have it working, but
this is the recommend base setting. If you change it, restart
PuppetDB.
* Check that the PuppetDB java process is running also. ps auxww |
grep java is a good start, provide the results if you like in thread.
* Ensure that the hostname 'puppet.internal' resolves to what you
think it does. A large amount of these errors are due to assumptions
about what hostnames resolve to ... it must resolve to the IP that
PuppetDB is listening on. Check /etc/hosts on your puppet master ...

Let me know how you go.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkPLDBb7fNAhQBWvpFsn8%3Db6Z6gi2zKhyGddpZc18pJmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Is PuppetDB environment aware?

2014-06-17 Thread Ken Barber

The support for environments in PDB is for storing the environment
where a catalog/factset/report came from ... and you can certainly
query on it, but currently with ordinary resource collection you
cannot constrain on environment. There is an open ticket in the Puppet
queue to do this in the future:
https://tickets.puppetlabs.com/browse/PUP-2217

For now though one can use something like puppetdbquery, and certainly
filter on environment.

As far as PE inclusion, inclusion of PDB 2.x is slated for PE 3.4 at
this point in time. I'm not sure when the dates for that release will
be confirmed.

ken.

On Tue, Jun 17, 2014 at 8:12 AM, Vadym Chepkov vchep...@gmail.com wrote:

On Jun 17, 2014, at 8:40 AM, Yanis Guenane yguen...@gmail.com wrote:

The latest version of PuppetDB (ie. 2.0.0) does support environments.

The first line of the release note is 'PuppetDB 2.0.0 is a feature release
focusing on environments support'.

Here the full link:
http://docs.puppetlabs.com/puppetdb/2.0/release_notes.html

I guess this means that Puppet Enterprise product currently doesn’t support
environments since it includes PuppetDB 1.5.2
No work arounds? Are there any plans to include PuppetDB 2.0.0 in PE any
time soon?

Thanks,
Vadym

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTm6c%3DO%3D1rPvcW8W2YPy2%3DTma_5eW5xoq8_6iQ7%2B7yrTJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-17 Thread Ken Barber

At first glance this all seems correct. Hrm.

Can you do the telnet test?

telnet puppet.internal 8081

Also, are you destroying and rebuilding these VM's each time and then
its failing? Or are you doing all of this _after_ the vm's are
launched. Its quite possible there is a race condition/ordering issue
in how the provisioning is occuring end-to-end.

ken.

On Tue, Jun 17, 2014 at 10:44 AM, Sans r.santanu@gmail.com wrote:
 Hi Ken,

 Thanks for the heads up!
 First of all, it's a VBox VM, provisioned by Vigrant. PuppetMaster and
 PuppetDB are on the same machine.
 I did go through those basic checks before posting, which appeared fine:


 root@puppet:~# telnet puppet.internal 8081
 Trying 127.0.1.1...
 Connected to puppet.internal.
 Escape character is '^]'.


 root@puppet:~# netstat -ntpl | grep 80
 tcp0  0 0.0.0.0:80  0.0.0.0:*LISTEN
 14345/apache2
 tcp6   0  0 :::8080 :::* LISTEN
 16301/java
 tcp6   0  0 127.0.1.1:8081  :::* LISTEN
 16301/java


 This is my jetty.ini:

 root@puppet:~# awk '!/^($|#)/ {print}' /etc/puppetdb/conf.d/jetty.ini
 [jetty]
 host = 0.0.0.0
 port = 8080
 ssl-host = puppet.internal
 ssl-port = 8081
 ssl-key = /etc/puppetdb/ssl/private.pem
 ssl-cert = /etc/puppetdb/ssl/public.pem
 ssl-ca-cert = /etc/puppetdb/ssl/ca.pem


 Java is also running:

 root@puppet:~# ps auxww | grep java
 puppetdb 16301  1.0 26.8 1558932 135336 ?  Sl   13:47   2:26
 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -XX:OnOutOfMemoryError=kill -9 %p
 -Xmx192m -XX:+HeapDumpOnOutOfMemoryError
 -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof
 -Djava.security.egd=file:/dev/urandom -cp /usr/share/puppetdb/puppetdb.jar
 clojure.main -m com.puppetlabs.puppetdb.core services -c
 /etc/puppetdb/conf.d



 ping  can resolve:

 root@puppet:~# ping -c2 puppet.internal
 PING puppet.internal (127.0.1.1) 56(84) bytes of data.
 64 bytes from puppet.internal (127.0.1.1): icmp_req=1 ttl=64 time=0.023 ms
 64 bytes from puppet.internal (127.0.1.1): icmp_req=2 ttl=64 time=0.032 ms

 --- puppet.internal ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 999ms
 rtt min/avg/max/mdev = 0.023/0.027/0.032/0.006 ms



 but nslookup cannot:

 root@puppet:~# nslookup puppet.internal
 Server:10.0.2.3
 Address:10.0.2.3#53

 ** server can't find puppet.internal: NXDOMAIN

 (nslookup is fine though with localhost)

 This is what my /etc/hosts looks like:

 127.0.0.1localhost
 127.0.1.1puppet.internalpuppet



 It's Ubuntu 12.04 server and I heard that name resolving works differently
 in this version. I'm lost here. Best!!

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/puppet-users/f1f592e5-c15f-407b-bf09-48ee28eb9ab7%40googlegroups.com.

 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DK4WW1%2BtPv0X5Y8Kan9YhwAUa3HHgkSTtFhKoRoxY_kQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-17 Thread Ken Barber

Oh ... and lets see the output of:

iptables -vnL

Perhaps there is a firewall here? Its worth double checking.

On Tue, Jun 17, 2014 at 11:06 AM, Ken Barber k...@puppetlabs.com wrote:
 At first glance this all seems correct. Hrm.

 Can you do the telnet test?

 telnet puppet.internal 8081

 Also, are you destroying and rebuilding these VM's each time and then
 its failing? Or are you doing all of this _after_ the vm's are
 launched. Its quite possible there is a race condition/ordering issue
 in how the provisioning is occuring end-to-end.

 ken.

 On Tue, Jun 17, 2014 at 10:44 AM, Sans r.santanu@gmail.com wrote:
 Hi Ken,

 Thanks for the heads up!
 First of all, it's a VBox VM, provisioned by Vigrant. PuppetMaster and
 PuppetDB are on the same machine.
 I did go through those basic checks before posting, which appeared fine:


 root@puppet:~# telnet puppet.internal 8081
 Trying 127.0.1.1...
 Connected to puppet.internal.
 Escape character is '^]'.


 root@puppet:~# netstat -ntpl | grep 80
 tcp0  0 0.0.0.0:80  0.0.0.0:*LISTEN
 14345/apache2
 tcp6   0  0 :::8080 :::* LISTEN
 16301/java
 tcp6   0  0 127.0.1.1:8081  :::* LISTEN
 16301/java


 This is my jetty.ini:

 root@puppet:~# awk '!/^($|#)/ {print}' /etc/puppetdb/conf.d/jetty.ini
 [jetty]
 host = 0.0.0.0
 port = 8080
 ssl-host = puppet.internal
 ssl-port = 8081
 ssl-key = /etc/puppetdb/ssl/private.pem
 ssl-cert = /etc/puppetdb/ssl/public.pem
 ssl-ca-cert = /etc/puppetdb/ssl/ca.pem


 Java is also running:

 root@puppet:~# ps auxww | grep java
 puppetdb 16301  1.0 26.8 1558932 135336 ?  Sl   13:47   2:26
 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -XX:OnOutOfMemoryError=kill -9 %p
 -Xmx192m -XX:+HeapDumpOnOutOfMemoryError
 -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof
 -Djava.security.egd=file:/dev/urandom -cp /usr/share/puppetdb/puppetdb.jar
 clojure.main -m com.puppetlabs.puppetdb.core services -c
 /etc/puppetdb/conf.d



 ping  can resolve:

 root@puppet:~# ping -c2 puppet.internal
 PING puppet.internal (127.0.1.1) 56(84) bytes of data.
 64 bytes from puppet.internal (127.0.1.1): icmp_req=1 ttl=64 time=0.023 ms
 64 bytes from puppet.internal (127.0.1.1): icmp_req=2 ttl=64 time=0.032 ms

 --- puppet.internal ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 999ms
 rtt min/avg/max/mdev = 0.023/0.027/0.032/0.006 ms



 but nslookup cannot:

 root@puppet:~# nslookup puppet.internal
 Server:10.0.2.3
 Address:10.0.2.3#53

 ** server can't find puppet.internal: NXDOMAIN

 (nslookup is fine though with localhost)

 This is what my /etc/hosts looks like:

 127.0.0.1localhost
 127.0.1.1puppet.internalpuppet



 It's Ubuntu 12.04 server and I heard that name resolving works differently
 in this version. I'm lost here. Best!!

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/puppet-users/f1f592e5-c15f-407b-bf09-48ee28eb9ab7%40googlegroups.com.

 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTn4NfvtzLB8cdDUPCCuY0%2Bv-N3YNy-2SKQpCw-fsdyfvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-17 Thread Ken Barber

Right now I'm creating only one VM, co-locating PuppetMaster and PuppetDB to
make it simple - destroying and rebuilding. But it always fails - during the
provisioning/building and also even after if I login to the machine and run
puppet apply. Telnet works fine:

root@puppet:~# telnet puppet.internal 8081
Trying 127.0.1.1...
Connected to puppet.internal.
Escape character is '^]'.
Connection closed by foreign host.

and iptables is not configured at all but here is the output:

Okay, let me zoom out a bit and rethink this. Going back to a point
you made originally:

Can anyone explain to me what actually going on/wrong please? Why it was Not
Found before and now Unable to connect? Any help/pointer would be much
appreciated. Best!

I think the 404 is common while the application is loading within
Jetty. It might be that the timeout we have for waiting for PuppetDB
to be fully available needs to be adjusted? It could also be a legit
problem also.

So lets try a different tactic.

Lets adjust the setting puppetdb_startup_timeout for the
puppetdb::master::config class, and change it to something high, like
60 seconds or so.

If that still fails, I think we should grab the
/var/log/puppetdb/puppetdb.log output, can you put that in a gist or
something? The log immediately after the attempted provision is best -
so don't try to rerun puppet or anything, we want to see if purely
after the provisioning process fails.

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3Dx5dQEXKt2frzQr6VK0KLYngzWPZr7HRFZU%3DvT%2BSiACQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Historical reporting, BI from puppetdb?

2014-06-17 Thread Ken Barber

I use puppetdb + puppetboard, which are very useful to see the current state
of my environment. Puppetboard also provides a very nice representation of
each agent's most recent reports. However, I want to take it to the next
level and create custom historical reports for business intelligence (eg How
many changes per day over the last month? How many RHEL5 virtual systems as
of $DATE?). I've looked online and not found many solutions on this. Might
Foreman provide this? Even if it doesI'm not sure I'm ready for that
commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool
that supports it (eg Pentah) could be configured to get what I want.
However, is anyone successfully doing this?

So something to note, while this is an interesting idea, you have to
be careful. We make zero commitments about the stability of the
database schema today ... so expect an upgrade to break things,
especially if we have a migration. In short, accessing the data
directly is not a supported stable API. Not saying it won't work, but
just letting you know the risks involved here.

Not sure how useful it is, we do have a lot of complex query
capability just built into PDB via the REST interface which might be
able to provide what you need:

http://docs.puppetlabs.com/puppetdb/2.0/api/query/v4/query.html

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmU6MhRDXN2mQ0_4T%3DOTpz-ZboWfUeMZ7%3Dzd-V8nW%3DGvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] PuppetDB connection issue - Failed to connect!!

2014-06-17 Thread Ken Barber

 It's very strange: Until I run puppetdb ssl-setup -f, I get

 Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
 Not Found

 but after that, I get


 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)



 Since, I increased the timeout to 60, it changed into this:

 Info: Class[Puppetdb::Server::Jetty_ini]: Scheduling refresh of
 Service[puppetdb]
 Notice: /Stage[main]/Puppetdb::Server/Service[puppetdb]: Triggered
 'refresh' from 1 events
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Notice: Unable to connect to puppetdb server (puppet.internal:8081):
 #Errno::ECONNREFUSED: Connection refused - connect(2)
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
 Not Found
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
 Not Found
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 .
 .

 Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
 Not Found
 Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
 Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
 Not Found
 Notice: Failed to connect to puppetdb within timeout window of 60 seconds;
 giving up.

 Error: Unable to connect to puppetdb server! (puppet.internal:8081)
 Error:
 /Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure:
 change from absent to present failed: Unable to connect to puppetdb server!
 (puppet.internal:8081)



 I'm  rebuilding the server now.

The connection refused implies the server hasn't opened the port yet
for listening, the progression to 404 is probably the Jetty server
starting up but not yet being ready for serving the URL we test
against.

In particular, we test the URL similar to this curl request, so its
probably worth seeing if it returns anything on the command line:

curl 'http://puppet.local:8080/v2/metrics/mbean/java.lang:type=Memory'

Perhaps 60 seconds isn't enough? Its unlikely but worth trying to bump
it I guess.

Beyond that we'll need to see the puppetdb.log.

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmaavWMEjo9t57d1w80M_c-7rkD4a%2BKeKwQb1sBoKmN%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Historical reporting, BI from puppetdb?

2014-06-17 Thread Ken Barber

 Thanks, good to know. While the REST API would be the method to get at the
 data, my issue is that I'm not capable of writing a web app + data
 repository that can generate web-based reports, etc. I've actually gotten
 into the habit of running one-off queries using the API with curl to get
 YAML-formatted reports, which in part got me wanting more.

Question, this BI tool Pentah, what formats/apis is it able to ingest
beyond SQL? Are there any ETL capabilities built into this tool?

Would something like CSV work?

ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTkHGudYw9BEUUK2PhemOdB-znzwvaHJ08SU0avBt0yztQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Historical reporting, BI from puppetdb?

2014-06-17 Thread Ken Barber

Sorry, do you mean Pentaho?

On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote:
 Thanks, good to know. While the REST API would be the method to get at the
 data, my issue is that I'm not capable of writing a web app + data
 repository that can generate web-based reports, etc. I've actually gotten
 into the habit of running one-off queries using the API with curl to get
 YAML-formatted reports, which in part got me wanting more.

 Question, this BI tool Pentah, what formats/apis is it able to ingest
 beyond SQL? Are there any ETL capabilities built into this tool?

 Would something like CSV work?

 ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTktZTMegALk8Az6Q6PpoxgSzLUHBmJVL%2BYwBX%3DpF0bVtQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Historical reporting, BI from puppetdb?

2014-06-17 Thread Ken Barber

Ryan,

What about something like this?

http://wiki.pentaho.com/display/EAI/Rest+Client

This page seems to mix in general actions with integration steps, but
there are more integration types available here:

http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps

ken.

On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote:
 Sorry, do you mean Pentaho?

 On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote:
 Thanks, good to know. While the REST API would be the method to get at the
 data, my issue is that I'm not capable of writing a web app + data
 repository that can generate web-based reports, etc. I've actually gotten
 into the habit of running one-off queries using the API with curl to get
 YAML-formatted reports, which in part got me wanting more.

 Question, this BI tool Pentah, what formats/apis is it able to ingest
 beyond SQL? Are there any ETL capabilities built into this tool?

 Would something like CSV work?

 ken.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmkViViz%3D466%3D_rrkiCGF7bRY7GmmcAnq1DYSyX3Ag1Ow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Historical reporting, BI from puppetdb?

2014-06-17 Thread Ken Barber

And others:

http://wiki.pentaho.com/display/EAI/JSON+Input
http://wiki.pentaho.com/display/EAI/HTTP+Client

Perhaps, you could combine the JSON input perhaps on the contents of a
'puppetdb export' tarball if you wanted to analyze the data from a
previous backup :-).

ken.

On Tue, Jun 17, 2014 at 10:01 PM, Ken Barber k...@puppetlabs.com wrote:
Ryan,

What about something like this?

http://wiki.pentaho.com/display/EAI/Rest+Client

This page seems to mix in general actions with integration steps, but
there are more integration types available here:

http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps

ken.

On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote:
Sorry, do you mean Pentaho?

On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote:
Thanks, good to know. While the REST API would be the method to get at the
data, my issue is that I'm not capable of writing a web app + data
repository that can generate web-based reports, etc. I've actually gotten
into the habit of running one-off queries using the API with curl to get
YAML-formatted reports, which in part got me wanting more.

Question, this BI tool Pentah, what formats/apis is it able to ingest
beyond SQL? Are there any ETL capabilities built into this tool?

Would something like CSV work?

ken.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTmG2ugH4KOp-TsjVC67wUKqJghRDrfHQrpVtcvKV0_yCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] can a puppet run tell puppet master to run puppet on other host based on previous events (e.g. new host)?

2014-06-16 Thread Ken Barber

Alex,

The more complete idea would be to trigger when resources have
actually been applied. So I would probably consider a report listener
for this kind of thing, as it shows when a resource has changed rather
than compiled.

I think Chris Spence has a tool for this kind of thing that uses MCO
to trigger the runs:

https://github.com/fiddyspence/puppet-mconotify

ken.

On Sat, Jun 14, 2014 at 1:15 PM, Cristian Falcas
cristi.fal...@gmail.com wrote:
No.

Maybe you can do it if you have a script that monitors a puppetdb? But this
is not what you are asking, so the answer will be no.

The master is involved only on giving the ENC data and doing the catalog
compilation.

Cristi Falcas

On Sat, Jun 14, 2014 at 4:46 PM, Alex Leonhardt aleonhardt...@gmail.com
wrote:

hi,

to ease orchestration and changes to systems with dynamic configurations /
e.g. configure load balancer based on how many webs are known to puppet / I
was wondering if it's possible to make a puppet master trigger puppet runs
on e.g. the load balancers *after* it got told there is a new host that got
class 'web' applied ?

in particular am looking for someting like saltstacks salt-reactor w/
salt-mine system

thanks!
alex

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/5c9348f8-aeb1-4c10-9ccd-a70618f3c761%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an
email to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAMo7R_dXQxkZ31reRGO03O960voihCL7P7mtkqZzbjRXJw5uOg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNTk%2BNJNig6m0dE_UYag8j%3DHKhbzc6eRfiVLb%2B51LwchECA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

1 2 3 4 5 6 >

1 - 100 of 515 matches

Mail list logo