Jira (PDB-2486) PuppetDB crashes during startup migration with namespaced trgm indices
Title: Message Title Nick Moriarty commented on PDB-2486 Re: PuppetDB crashes during startup migration with namespaced trgm indices Hi Russ, I asked for the information from our DBA - here's the reply: We've revoked all privs on public schema from public (as mentioned here http://www.postgresql.org/docs/9.4/static/ddl-schemas.html). Then as we're not using the public schema for public use it made sense to revoke all privs from it entirely and made a new schema called puppetdb for user also called puppetdb who has a search path set to the same. Regards Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-2486) PuppetDB crashes during startup migration with namespaced trgm indices
Title: Message Title Nick Moriarty created an issue PuppetDB / PDB-2486 PuppetDB crashes during startup migration with namespaced trgm indices Issue Type: Bug Affects Versions: PDB 2.3.8 Assignee: Unassigned Created: 2016/03/01 6:43 AM Environment: PuppetDB 2.3.8-1puppetlabs1 on Ubuntu 14.04 Running against PostgreSQL 9.5 Priority: Normal Reporter: Nick Moriarty Our PuppetDB instance runs against a managed PostgreSQL service hosting multiple databases. This means that indices end up being created under the 'puppetdb' namespace rather than the 'public' one. With pg_trgm indices enabled, PuppetDB crashes at startup as it tries to create duplicate indices during the migration operation. This is because it ends up looking in the 'public' namespace by default. In storage-utils, the default namespace for testing whether an index exists is 'public', which is not necessarily correct for a given deployment. The result is a BatchUpdateException, which kills the PuppetDB application straight away.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' The trouble with the above is that, apart from requiring module authors to all add this workaround to their code, you can't guarantee that the dependency can be found at a root starting from ../..(plus any number of levels) - depending on how modulepath is set, it could (potentially validly) be anywhere else in the filesystem - you can only figure out where it's likely to be by knowing modulepath. It would be great to see Puppet fix up LOAD_PATH so that these dependencies "just work". If you want module authors to change their code, it would at least be nice to have a helper function that does a require based on the current modulepath. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' I've done some digging into this. After many backtraces and adding a lot of extra debugging, it turns out that the following situation arises (but is sometimes masked as something gets polluted): When autoloading types and providers, Puppet's autoload mechanism does the heavy lifting to take care of environment paths This extra logic isn't used when puppetx/filemapper is _require_d Since $LOAD_PATH doesn't contain the extra environment directories, the underlying Kernel.load (for filemapper) fails This exception causes the outer load to fail It's possible that where this doesn't fail, LOAD_PATH has been either polluted by something else, or it's been set to include wherever filemapper is installed - maybe if the master has it in its own environment. We explicitly configure our masters to ensure their own environment doesn't pollute clients' environments. I have drafted a basic solution to this issue, which simply involves setting up and restoring $: ($LOAD_PATH) before and after the Kernel.load in autoload.rb. This will mean that any sources using require / load will be able to load from the same paths that Puppet knows about. It shouldn't be difficult to clean this up a bit, so that it can add the environment-specific paths only, and then remove them again afterwards. This is a bit of a hack, as it modifies the global state during catalog compilation; this is only safe if different environments can't compile concurrently under the same LOAD_PATH reference (so multithreading is probably unsafe, but forks should be fine). Is anyone able to comment on this safety issue, and knows what form of concurrency is used in the Puppetmaster? Add Comment
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' Submitted PR #4445 with a draft fix for 3.x - should be portable to 4. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-2836) Incorrect lockfile placement
Title: Message Title Nick Moriarty commented on PUP-2836 Re: Incorrect lockfile placement It's a lockfile. It signals that configuration is currently being applied, and so another run shouldn't be started. I believe it only contains a PID to allow Puppet to attempt to check whether it's actually valid - it checks that there is actually a process with that PID, in case the agent crashed and left the lockfile behind. Unfortunately, I've found in the past (this may or may not still be true) that Puppet doesn't seem to validate the process name, so you can end up in a situation where (especially after a reboot if the lockfile persists) another process ends up using that PID, which causes Puppet never to run again. Using $rundir (which should be defined somewhere that doesn't persist across reboots), as mentioned above, in the path to this lockfile has cured the problem in our case; that should only require changing the default. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty updated an issue Puppet / PUP-4450 Cannot load 'provider/network_route/routes.rb' Change By: Nick Moriarty Affects Version/s: PUP 3.7.5 Affects Version/s: PUP 3.8.4 Environment: RHEL6 + Puppet SC 1 Ubuntu 14.04 + Puppetmaster 3.7.5 / 3.8.4 on Apache2/Passenger Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' To follow up on my earlier statement, I have exactly the same problem after upgrading our test puppetmaster to 3.8.4. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' I've reproduced the same bug _sproradically_against puppet 3.7.5. In my case it's network_config/interfaces that can't load puppetx/filemapper. I added some debug logging to the relevant code. What I can see is that, sometimes, util/autoload.rb ends up missing out the environment modulepath's directories completely when trying to load providers. When it fails, the search path (including gems) only consists of the gempath. In working runs, it includes the lib directories in all modules under each element of modulepath. This seems like it might be a race condition somewhere. I note that $env_module_directories stores the mapping between the current environment and the load path. This is a global variable, but the comment on module_directories says "per-thread cache". I can't see any sort of protection on it; is it possible this is being corrupted somewhere? Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-4450) Cannot load 'provider/network_route/routes.rb'
Title: Message Title Nick Moriarty commented on PUP-4450 Re: Cannot load 'provider/network_route/routes.rb' On further inspection, I can't see anything actually hooking the require 'puppetx/filemapper' in autoload.rb - this almost looks like the fact it's succeeding could be some strange side-effect of it being in the load path at some point, sometimes - but restarting the master does seem to reliably kill it. Does anyone know how the require / autoload bits actually work? Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-1909) Priority setting also given to started processes/services
Title: Message Title Nick Moriarty commented on PUP-1909 Re: Priority setting also given to started processes/services Hi, I'd like to report having seen the above issue as well. I'm running on Ubuntu 14.04 with Puppetlabs-packaged (Open-source) 3.8.3. If I define priority in [agent], it gets ignored; it works on the command-line or in the [main] section. I'd also like to know how masters are affected by this setting, and also why some settings are completely ignored when put in [agent], even though puppet config print --section agent priority shows the correct value. As to services, I also see started services (using the default provider on ubuntu - in this case for traditional init-script-driven services) running using Puppet's nice value; while I understand why this occurs, it seems very much broken behaviour. My reason for looking into priority in the first place is to see whether we can make sure Puppet doesn't affect other services running on a given system, and this makes priority-setting unusable for us as it stands. If Puppet is running as a privileged process, it should be able to change its priority value as necessary e.g. when starting services. Execs are more of a grey area, but are generally more associated with the Puppet run; I'd expect a parameter on that type might be useful in some cases though. Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-3238) puppet reports end of file reached if server closes HTTP connection
Title: Message Title Nick Moriarty commented on PUP-3238 Re: puppet reports end of file reached if server closes HTTP connection For info (in case anyone else runs into this), I found the source of another odd issue causing sporadic disconnects (symptom was sporadic end of file reached on various file resources. Our masters run Apache 2.4 with KeepAlive (with an appropriate timeout with a good margin), and were configured to use mod_reqtimeout with default request timeouts. There seems to be an interesting interaction between KeepAlive and reqtimeout which causes some strangeness and can result in connections dropping. I never saw any substantial delays in the logs (certainly nothing near the configured timeouts), and I did find a reference to this on the Apache bug tracker. https://bz.apache.org/bugzilla/show_bug.cgi?id=56729 We've disabled reqtimeout for now to prevent this cropping up. Testing HTTP connections looks to be moderately difficult as it appears to require examining the internal state of HTTP, and possibly making at least a non-blocking read for at least one byte from the underlying socket - this results in either Errno::EAGAIN (if there's no data), or EOFError (if the socket was closed by the server). Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-2836) Incorrect lockfile placemen
Title: Message Title Nick Moriarty commented on PUP-2836 Re: Incorrect lockfile placemen We're hitting this issue occasionally; at present, we're planning to work around it by changing agent_catalog_run_lockfile to $rundir/agent_catalog_run.lock. At present, the default value of this is $statedir/agent_catalog_run.lock, which persists across reboots - I'd recommend this change to $rundir/agent_catalog_run.lock (which is where the main pidfiles live) so it doesn't persist. This issue crops up occasionally (we're seeing it in 3.7.3) - even with the sanity checking of lockfiles, it's possible to have a catalog run lockfile that persists across a reboot, containing a valid PID which references a completely different process after the reboot. In this case, Puppet may never run (consider the case where it ends up pointing at sshd, for example). Add Comment This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-2836) Incorrect lockfile placement
Title: Message Title Nick Moriarty updated an issue Puppet / PUP-2836 Incorrect lockfile placement Change By: Nick Moriarty Affects Version/s: PUP3.7.3 Summary: Incorrectlockfile placemen placement Add Comment This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-731) Masters cannot reliably distinguish between multiple versions of a type/function/plugin used in different environments
Title: Message Title Nick Moriarty commented on PUP-731 Re: Masters cannot reliably distinguish between multiple versions of a type/function/plugin used in different environments I'd like to ask for some clarification around Puppetmaster's behaviour with this sort of problem, as we want to run with directory environments but wish to know a little more about how and when this issue occurs (with a view to working around it). We run Puppetmaster 3.7 on Apache/Passenger. Currently, all environments are served by one passenger pool, so the Puppetmasters will no doubt be prone to this issue, as systems in different environments will be randomly assigned to free workers. What I'm not sure of, and hoping that someone may be able to clarify - in the instance that there are multiple directory environments present at startup, will a given Puppetmaster process behave correctly as long as it is only serving requests for a single environment (e.g. because of a reverse proxy segregating requests), or will it load (some?) plugins from each environment at startup, and therefore always be in an inconsistent state? If it's only polluting Ruby constants (and hence potentially hitting this problem) when it's actually compiling catalogs for hosts in multiple environments (and not loading things at startup), I believe I could potentially work around this problem by configuring Passenger to use separate worker pools per environment (as the URIs appear to all begin with the environment name, so it should be able to easily separate the requests). Is this likely to work with any form of reliable segregation between environments? A point I'm particularly interested in is how Puppetmaster behaves if an ENC overrides a node's environment - does it stop there and let the node re-request its catalogue for the right environment, or does it 'switch' into the other environment? We may wish to use ENC-supplied environments in the near future, and I realised that the workaround I'm talking about could be prone to issues if puppetmaster processes ever perform a switch like this. Finally, is there an expected behaviour with regards to whether a Puppetmaster process will see one version or another (e.g. whichever was loaded first / most recently), or is it even possible to end up with neither (e.g. if the versions interfere or end up merged in some manner)? I guess answers to the above might be helpful to others in deciding how to work with environments. Many thanks. Add Comment
Jira (PDB-1115) PuppetDB Forge Module has incorrect name in metadata.json
Title: Message Title Nick Moriarty commented on PDB-1115 Re: PuppetDB Forge Module has incorrect name in metadata.json Pull request submitted as #164. Add Comment This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-3238) puppet reports end of file reached if server closes HTTP connection
Title: Message Title Nick Moriarty commented on PUP-3238 Re: puppet reports end of file reached if server closes HTTP connection I've finally tracked down the cause of the client-initiated connection closures causing our issue. These were caused by bad file descriptor usage in a custom fact; in Ruby 1.8, IO objects seem to be very low-level, and the Open3 library can return illegal but valid-looking IO objects from a call to popen3 even if the underlying command fails to run. It looks like these objects can end up relating to other, existing IOs; in our case, this was sometimes the HTTP socket. Closing of the connection appeared to be a side-effect of these invalid IOs. I've fixed the associated fact, and the problem seems to have subsided. I'd still like to see a retry feature and/or a test of HTTP connections to see if they're actually still open before using them. Add Comment This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-3238) puppet reports end of file reached if server closes HTTP connection
Title: Message Title Nick Moriarty commented on PUP-3238 Re: puppet reports end of file reached if server closes HTTP connection Updates on my previous comment. We're not seeing the above problem on all of our environments; it appears to be constrained to only some. I added some extra debugging and have discovered the following: The client-issued Encrypted Alert described above is indeed a Close Notify (determined by changing our test server to use a non-DH cipher and packet sniffing). This appears to be the first sign of any difference between normal runs and those where the connection is dropped. Having added extra debugging around fact retrieval, I can confirm that the Close Notify is sent after all facts have been retrieved, but before the debug line Failed to load library 'msgpack' for feature 'msgpack'. The connection close definitely seems to be client-initiated. Add Comment This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-3238) puppet reports end of file reached if server closes HTTP connection
Title: Message Title Nick Moriarty commented on PUP-3238 Re: puppet reports end of file reached if server closes HTTP connection We're also seeing end of file sometimes with KeepAlive. We're using 4 seconds / 10 seconds on client / server respectively. Clients are running 3.7.2 and .3, masters are running 3.7.3. We're seeing bursts of failed runs on the affected hosts, around seven hours apart but not correlated with each other. I've added local debugging and I'm running tcpdump on a client and test master. What I believe I'm seeing on the failed runs (waiting on more data) is: Everything seems normal, and the connection is cached after downloading facts, but before Loading facts. The client sends an SSL Encrypted Alert, which I believe contains a request to close the connection (at present I've not decrypted this as we're using DH - turned this off to get more data now). The server acknowledges and sends a FIN/ACK to close the connection, which the client acknowledges immediately. The client picks up the connection (which hasn't reached its keepalive timeout) and tries to use it (it sends successfully). The web server never sees / ignores the request as the connection is already half-closed. An exception is thrown from /usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread' when the client tries to read from the (closed) incoming stream - it does this without timing out. I'm still trying to find out why this is happening reliably in bursts (but I can't seem to force it to happen), as it looks like the client is deciding to close the connection, but it's definitely not being done via the Puppet connection pool class. The issue would be alleviated if the connection pool checked that connections were actually still fully open before selecting them - it seems valid that the server should be allowed to close the connection of its own accord, and the client shouldn't then try to re-use the connection. This might be something that can be checked on the HTTP object when determining which connections are active. Add Comment
Jira (PUP-3781) Probably race condition in HTTP session pool for KeepAlive
Title: Message Title Nick Moriarty created an issue Puppet / PUP-3781 Probably race condition in HTTP session pool for KeepAlive Issue Type: Bug Affects Versions: PUP 3.7.3, PUP 3.7.2 Assignee: Unassigned Components: Puppet Server Created: 2014/12/18 3:16 AM Environment: Ubuntu 12.04.5 / 14.04.1 Labels: puppet-agent Priority: Normal Reporter: Nick Moriarty There may be a race condition when retrieving cached connections in network/http/pool.rb, which causes sporadic failures of Puppet when using KeepAlive. We have keepalive set to 4 seconds as default on the client, and have increased the server keepalive from 5 to 10 seconds with no improvement.
Jira (PUP-3781) Probable race condition in HTTP session pool for KeepAlive
Title: Message Title Nick Moriarty updated an issue Puppet / PUP-3781 Probable race condition in HTTP session pool for KeepAlive Change By: Nick Moriarty Summary: Probably Probable raceconditioninHTTPsessionpoolforKeepAlive Add Comment This message was sent by Atlassian JIRA (v6.3.7#6337-sha1:2ed701e) -- You received this message because you are subscribed to the Google Groups Puppet Bugs group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
