We have 700 servers and recently puppetserver stopped responce catalog to
agents. The first few minutes it works fine, and then throws exceptions for
all thread:
2017-12-16 21:42:19,963 WARN [qtp1380493193-555] [o.e.j.s.HttpChannel]
/puppet/v3/catalog/db03st-ru.star-conflict.com?environment=production
java.nio.channels.WritePendingException: null
at
org.eclipse.jetty.server.HttpConnection$SendCallback.reset(HttpConnection.java:624)
at
org.eclipse.jetty.server.HttpConnection$SendCallback.access$100(HttpConnection.java:594)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:479)
at
org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:135)
at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:166)
at org.eclipse.jetty.server.HttpWriter.close(HttpWriter.java:49)
at java.io.PrintWriter.close(PrintWriter.java:339)
at ring.util.servlet$set_body.invokeStatic(servlet.clj:89)
at ring.util.servlet$set_body.invoke(servlet.clj:84)
at
ring.util.servlet$update_servlet_response.invokeStatic(servlet.clj:115)
at ring.util.servlet$update_servlet_response.invoke(servlet.clj:107)
at
puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__30506.invoke(jetty9_core.clj:426)
at
puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown
Source)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at
puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers$eval30081$normalize_uri_handler__30086$fn__30087$fn__30088.invoke(normalized_uri_helpers.clj:80)
at
puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown
Source)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
2017-12-17 00:10:17,030 WARN [qtp1943211232-575] [o.e.j.s.HttpChannel]
/puppet/v3/catalog/pl-01.local.net?environment=production
Agent Log:
puppet-agent [19302]: Could not retrieve catalog from remote server: end of
file reached
It works again only after the restart.The puppetserver are Intel Xeon E312
machines, so they have 32 cores and 20GB of memory, and Intel SSDs. Server
does not have high LA.
How can I identify the source of the problem?
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/d2d8467b-ad20-4037-bf15-d835ff86f5c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
