Title: Message Title
| |
|
|
|
|
PuppetDB "commands" are operations to update the data associated with puppet nodes. These commands are identified by certname, and odd things can happen if an empty string or other null value is provided as the certname. One consequence is a {{java.lang.NullPointerException}} during startup. This
When PE DR is enabled, this exception prevents the PuppetDB service from exiting maintenance mode and responding to read and write requests.
h2. Reproduction Case
- Install PE 2019.8.8 on a CentOS 7 node and add a DR replica
- Deactivate an empty certname: {{puppet node deactivate ''}}
- Re-start the {{pe-puppetdb}} service
h3. Outcome
The deactivate command results in the following error message being recorded to the PuppetDB log:
{noformat}
2021-09- 21T22 21T23 : 20 21 : 47 07 . 271Z 566Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread- 13 2 to stderr and log
java.lang.NullPointerException: null
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
This also leaves behind a command int the PuppetDB processing queue with an empty certname:
{noformat}
# find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -print -exec cat {} \;
/opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q/ 50 20 - 1632262847254 1632266467556 - 115_rm 103_rm -node_3_.json
{"certname":"","producer_timestamp":"2021-09- 21T22 21T23 : 20 21 : 47 07 . 139 453 +00:00"}
{noformat}
Upon restart, the PuppetDB services fails to initialize the command processing pool and never exits maintenance mode:
{noformat}
2021-09- 21T22 21T23 : 33 22 : 58 20 . 072Z 499Z ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-1 to stderr and log
java.lang.NullPointerException: null
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:76)
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
at metrics.meters$mark_BANG_.invokeStatic(meters.clj:74)
at metrics.meters$mark_BANG_.invoke(meters.clj:72)
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invokeStatic(command.clj:244)
at puppetlabs.puppetdb.command$mark_both_metrics_BANG_.invoke(command.clj:240)
at puppetlabs.puppetdb.command$process_message$retry__37262.invoke(command.clj:755)
at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:810)
at puppetlabs.puppetdb.command$process_message.invoke(command.clj:742)
at puppetlabs.puppetdb.command$message_handler$fn__37274.invoke(command.clj:820)
at puppetlabs.puppetdb.threadpool$dochan$fn__36851$fn__36852.invoke(threadpool.clj:116)
at puppetlabs.puppetdb.threadpool$gated_execute$fn__36813.invoke(threadpool.clj:69)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-09-21T23:22:20.715Z INFO [p.p.c.services] Finished sweep of stale nodes (threshold: 7 days)
2021-09-21T23:22:20.717Z INFO [p.p.c.services] Starting purge deactivated and expired nodes (threshold: 14 days)
2021-09-21T23:22:20.760Z INFO [p.p.c.services] Finished purge deactivated and expired nodes (threshold: 14 days)
2021-09-21T23:22:20.761Z INFO [p.p.c.services] Starting sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
2021-09-21T23:22:20.849Z INFO [p.p.c.services] Finished sweep of stale reports (threshold: 14 days) and resource events (threshold: 14 days)
2021-09-21T23:22:20.849Z INFO [p.p.c.services] Starting gc packages
2021-09-21T23:22:20.852Z INFO [p.p.c.services] Finished gc packages
2021-09-21T23:22:20.853Z INFO [p.p.c.services] Starting database garbage collection
2021-09-21T23:22:20.909Z INFO [p.p.s.storage] Starting sweep of unused fact paths
2021-09-21T23:22:20.932Z INFO [p.p.s.storage] Finished sweep of unused fact paths
2021-09-21T23:22:20.940Z INFO [p.p.c.services] Finished database garbage collection
{noformat}
There should be a "PuppetDB finished starting, disabling maintenance mode" message in the above.
h3. Expected Outcome
An empty certname in a command submission does not block PuppetDB startup
h3. Recommended Workaround
Clear the PuppetDB command queue and re-start the service:
{code:bash}
find /opt/puppetlabs/server/data/puppetdb/stockpile/cmd/q -type f -delete
systemctl restart pe-puppetdb
{code} |
|
|
|
| |
|
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.416719.1632261204000.136024.1632266700043%40Atlassian.JIRA.
|
