Jira (PDB-1019) respond with 503 when database is unavailable

Charlie Sharpsteen (JIRA) Wed, 29 Apr 2015 09:57:18 -0700

Title: Message Title

PDB-1019

Kenneth Barber: I think the issue here isn't with the retry behavior, but with the HTTP error code that is returned while PuppetDB is trying to establish a connection during startup.

Generally, PuppetDB uses the 404 error code to indicate that a query did not match any data:

#curl -i http://localhost:8080/v4/nodes/foo.bar.baz

HTTP/1.1 404 Not Found

Date: Wed, 29 Apr 2015 16:13:40 GMT

Warning: v4 query API is experimental and may change without warning. For stability use the v3 api.

Content-Type: application/json; charset=utf-8

Content-Length: 59

Server: Jetty(9.2.z-SNAPSHOT)

  "error" : "No information is known about foo.bar.baz"

If PostgreSQL dies after PuppetDB has started, a 500 error is returned that clearly calls out a database connection as the source of the problem

# service pe-postgresql stop

# curl -i http://localhost:8080/v4/nodes/$(hostname -f)

HTTP/1.1 500 org.postgresql.util.PSQLException: This connection has been closed.

Date: Wed, 29 Apr 2015 16:14:30 GMT

Cache-Control: must-revalidate,no-cache,no-store

Content-Type: text/html; charset=ISO-8859-1

Content-Length: 375

Connection: close

Server: Jetty(9.2.z-SNAPSHOT)

<html>

<head>

<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>

<title>Error 500 </title>

</head>

<body>

<h2>HTTP ERROR: 500</h2>

<p>Problem accessing /v4/nodes/pe-380-master.puppetdebug.vlan. Reason:

<pre>    org.postgresql.util.PSQLException: This connection has been closed.</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

However, if PuppetDB is in the middle of startup and can't establish a database connection, an ambiguous 404 error is returned instead:

# service pe-postgresql stop

# service pe-puppetdb restart

# curl -i http://localhost:8080/v4/nodes/$(hostname -f)

HTTP/1.1 404 Not Found

Date: Wed, 29 Apr 2015 16:27:33 GMT

Cache-Control: must-revalidate,no-cache,no-store

Content-Type: text/html; charset=ISO-8859-1

Content-Length: 317

Server: Jetty(9.2.z-SNAPSHOT)

<html>

<head>

<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>

<title>Error 404 </title>

</head>

<body>

<h2>HTTP ERROR: 404</h2>

<p>Problem accessing /v4/nodes/pe-380-master.puppetdebug.vlan. Reason:

<pre>    Not Found</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

It seems as though the PuppetDB app hasn't been fully mounted to the Jetty webserver, so Jetty its self is responding with a 404.

The problem with this is that it makes it difficult for downstream services to programmatically distinguish a 404 query result that found no data and a 404 query result that failed due to a DB connection, and present a meaningful error message to the user. Often, a user has to manually log into the PuppetDB node and examine the logs to resolve the ambiguity and correct the PostgreSQL issue.

For example, is the following run failing due to missing data in PuppetDB, or due to a database that is down?

# /opt/puppet/bin/puppet agent -t

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 400 on SERVER: Failed to find facts from PuppetDB at pe-380-master.puppetdebug.vlan:8081: [404 ] <html><head><meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/><title>Error 404 </title></head><body><h2>HTTP ERROR: 404</h2><p>Problem accessing /v3/nodes/pe-380-master.puppetdebug.vlan/facts. Reason:<pre>    Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i></body></html>

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for pe-380-master.puppetdebug.vlan to PuppetDB at pe-380-master.puppetdebug.vlan:8081: [404 ] <html><head><meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/><title>Error 404 </title></head><body><h2>HTTP ERROR: 404</h2><p>Problem accessing /v3/commands. Reason:<pre>    Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i></body></html>

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Add Comment

This message was sent by Atlassian JIRA

--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PDB-1019) respond with 503 when database is unavailable

Reply via email to