Unfortunately setting max-threads to 200 did not help.
# java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
# dpkg -l |grep -i java
ii ca-certificates-java 20110912ubuntu6 Common
CA certificates (JKS keystore)
ii java-common 0.43ubuntu2 Base of
all Java packages
ii libjs-jquery 1.7.1-1ubuntu1
JavaScript library for dynamic web applications
ii openjdk-7-jre-headless 7u55-2.4.7-1ubuntu1~0.12.04.2 OpenJDK
Java runtime, using Hotspot JIT (headless)
ii tzdata-java 2014c-0ubuntu0.12.04 time
zone and daylight-saving time data for use by java runtimes
This is a fresh output in logs:
2014-05-20 15:21:52,331 INFO [p.t.s.w.jetty9-service] Initializing web server.
2014-05-20 15:21:52,403 INFO [p.t.s.w.jetty9-service] Starting web server.
2014-05-20 15:21:52,406 INFO [o.e.j.s.Server] jetty-9.1.z-SNAPSHOT
2014-05-20 15:21:52,436 INFO [o.e.j.s.ServerConnector] Started
ServerConnector@2557ce03{HTTP/1.1}{localhost:8080}
2014-05-20 15:21:52,504 INFO [o.e.j.s.ServerConnector] Started
ServerConnector@253e73c6{SSL-HTTP/1.1}{puppet.site:8081}
2014-05-20 15:21:52,563 INFO [c.p.p.c.services] PuppetDB version 2.0.0
2014-05-20 15:21:52,704 INFO [c.p.p.c.services] Starting broker
2014-05-20 15:21:53,581 WARN [o.a.a.b.BrokerService] Store limit is 100000 mb,
whilst the data directory: /var/lib/puppetdb/mq/localhost/KahaDB only has 63697
mb of usable space
2014-05-20 15:21:53,582 INFO [c.p.p.c.services] Starting 4 command processor
threads
2014-05-20 15:21:53,602 INFO [c.p.p.c.services] Starting query server
2014-05-20 15:21:53,605 WARN [o.e.j.s.h.ContextHandler] Empty contextPath
2014-05-20 15:21:53,617 INFO [o.e.j.s.h.ContextHandler] Started
o.e.j.s.h.ContextHandler@31771aa1{/,null,AVAILABLE}
2014-05-20 15:21:53,643 INFO [c.p.p.c.services] Starting sweep of stale
reports (threshold: 14 days)
2014-05-20 15:21:53,667 INFO [c.p.p.c.services] Finished sweep of stale
reports (threshold: 14 days)
2014-05-20 15:21:53,667 INFO [c.p.p.c.services] Starting database garbage
collection
2014-05-20 15:21:53,698 INFO [c.p.p.c.services] Finished database garbage
collection
2014-05-20 15:22:05,832 INFO [c.p.p.command]
[ce2e1e6e-5bc8-4f0a-bb20-e1edd153038d] [replace facts] prod1.site
2014-05-20 15:22:36,859 WARN [o.e.j.s.HttpChannel]
/v3/commands?checksum=7b886250b6f4ff03643ca1a82ff38476b015ec02
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 30000/30000 ms
at
org.eclipse.jetty.util.BlockingCallback.block(BlockingCallback.java:101)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:62)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:392)
~[puppetdb.jar:na]
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:161)
~[puppetdb.jar:na]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
~[na:1.7.0_55]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
~[na:1.7.0_55]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_55]
at java.io.InputStreamReader.read(InputStreamReader.java:184)
~[na:1.7.0_55]
at java.io.BufferedReader.fill(BufferedReader.java:154) ~[na:1.7.0_55]
at java.io.BufferedReader.read(BufferedReader.java:175) ~[na:1.7.0_55]
at clojure.core$slurp.doInvoke(core.clj:6396) ~[puppetdb.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[puppetdb.jar:na]
at ring.util.request$eval10113$fn__10114.invoke(request.clj:30) ~[na:na]
at clojure.lang.MultiFn.invoke(MultiFn.java:227) ~[puppetdb.jar:na]
at
com.puppetlabs.middleware$payload_to_body_string$fn__10429.invoke(middleware.clj:273)
~[na:na]
at
com.puppetlabs.middleware$verify_content_type$fn__10379.invoke(middleware.clj:128)
~[na:na]
at com.puppetlabs.puppetdb.http.v3$v3_app$fn__11827.invoke(v3.clj:23)
~[na:na]
at com.puppetlabs.puppetdb.http.v3$v3_app.invoke(v3.clj:21) ~[na:na]
at
com.puppetlabs.puppetdb.http.server$routes$fn__12370.invoke(server.clj:47)
~[na:na]
at
net.cgrand.moustache$alter_request$fn__10771.invoke(moustache.clj:54) ~[na:na]
at com.puppetlabs.puppetdb.http.server$routes.invoke(server.clj:47)
~[na:na]
at
ring.middleware.resource$wrap_resource$fn__12339.invoke(resource.clj:24)
~[na:na]
at ring.middleware.params$wrap_params$fn__10714.invoke(params.clj:58)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_authorization$fn__10346.invoke(middleware.clj:40)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_certificate_cn$fn__10350.invoke(middleware.clj:55)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_default_body$fn__10354.invoke(middleware.clj:62)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__10401$fn__10410.invoke(middleware.clj:207)
~[na:na]
at
com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069$fn__10070.invoke(metrics.clj:14)
~[na:na]
at
com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown
Source) ~[na:na]
at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
at
com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069.invoke(metrics.clj:14)
~[na:na]
at
com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069$fn__10070.invoke(metrics.clj:14)
~[na:na]
at
com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown
Source) ~[na:na]
at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
at
com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069.invoke(metrics.clj:14)
~[na:na]
at
com.puppetlabs.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:17)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__10401.invoke(middleware.clj:206)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_globals$fn__10358.invoke(middleware.clj:73)
~[na:na]
at
com.puppetlabs.middleware$wrap_with_debug_logging$fn__10342.invoke(middleware.clj:27)
~[na:na]
at
puppetlabs.trapperkeeper.services.webserver.jetty9_core$proxy_handler$fn__16765.invoke(jetty9_core.clj:66)
~[na:na]
at
puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown
Source) ~[na:na]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1048)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
~[puppetdb.jar:na]
at
org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:325)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
~[puppetdb.jar:na]
at org.eclipse.jetty.server.Server.handle(Server.java:459)
~[puppetdb.jar:na]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280)
~[puppetdb.jar:na]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
[puppetdb.jar:na]
at
org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
[puppetdb.jar:na]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
[puppetdb.jar:na]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
[puppetdb.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
30000/30000 ms
at
org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:153)
~[puppetdb.jar:na]
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
~[puppetdb.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_55]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_55]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
~[na:1.7.0_55]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
... 1 common frames omitted
Also I noticed that it happens only to some particular hosts. Some of our
puppet agents run without any problems. So may be this is a client related
issue?
On May 20, 2014, at 5:19 PM, Ken Barber <[email protected]> wrote:
> Also ... I'd be curious to know what exact version of the JVM you are running.
>
> I'm cautious this is a jetty bug of some kind, perhaps we can ship a
> new Jar with a later version of Jetty to you and see if that helps.
>
> ken.
>
> On Tue, May 20, 2014 at 2:03 PM, Ken Barber <[email protected]> wrote:
>> We don't expose the idle timeout parameter for users to modify.
>>
>> I am curious though, what happens when you increase max-threads in
>> your jetty.ini to something like 200?
>>
>> ken.
>>
>> On Tue, May 20, 2014 at 12:41 PM, Sergey Arlashin
>> <[email protected]> wrote:
>>> Here is described
>>> http://www.eclipse.org/jetty/documentation/current/configuring-connectors.html
>>> how idle timeout for jetty cat be configured . But I don't know how to apply
>>> this for puppetdb :(
>>>
>>> Does anybody have any ideas ?
>>>
>>>
>>>
>>> On May 20, 2014, at 9:43 AM, Sergey Arlashin <[email protected]>
>>> wrote:
>>>
>>> 2014-05-20 05:34:36,684 INFO [c.p.p.command]
>>> [e806b0a2-7703-4a4a-8107-65cd2c0db9a8] [replace facts] prod1.site
>>> 2014-05-20 05:34:44,112 INFO [c.p.p.command]
>>> [e8cb3511-9734-4ba7-b040-96f663404c00] [replace facts] redmine.site
>>> 2014-05-20 05:34:45,387 INFO [c.p.p.command]
>>> [9026d779-3a43-4c06-a23d-aeefd4eb6e1f] [replace catalog] redmine.site
>>> 2014-05-20 05:34:47,487 INFO [c.p.p.command]
>>> [1234e95c-e254-4b1d-b46f-8ef5b07e9158] [replace facts] www-node1.site
>>> 2014-05-20 05:34:48,350 INFO [c.p.p.command]
>>> [56be0dc9-03b9-45b8-8b8a-527f1c25e057] [replace facts] dev01.site
>>> 2014-05-20 05:34:48,404 INFO [c.p.p.command]
>>> [17474841-6043-46cf-98a1-122f116d5cd6] [replace facts] parser1.site
>>> 2014-05-20 05:34:48,634 INFO [c.p.p.command]
>>> [44da2889-811f-4e80-8190-ad7313f0ca81] [replace facts] dev02.site
>>> 2014-05-20 05:34:48,942 INFO [c.p.p.command]
>>> [4080aa24-23c7-4f16-9011-2104a457e201] [replace catalog] www-node1.site
>>> 2014-05-20 05:34:51,429 INFO [c.p.p.command]
>>> [6b62c283-f306-44e7-8101-4948110e3d78] [replace catalog] dev01.site
>>> 2014-05-20 05:34:51,437 INFO [c.p.p.command]
>>> [55fb7348-3b36-4c5a-9265-61ca3c1f55c3] [replace catalog] dev02.site
>>> 2014-05-20 05:34:52,925 INFO [c.p.p.command]
>>> [9ace137e-f708-4656-829a-92de7722b388] [replace facts] lb-node1.site
>>> 2014-05-20 05:34:55,017 INFO [c.p.p.command]
>>> [0b6e6592-97f2-4b8a-8cc7-6d321793d2ed] [replace catalog] lb-node1.site
>>> 2014-05-20 05:34:58,285 INFO [c.p.p.command]
>>> [971285dd-b141-4f44-a6d0-375aa3af830a] [replace facts] lb-node2.site
>>> 2014-05-20 05:34:59,333 INFO [c.p.p.command]
>>> [3977fc34-aa41-43e7-8eb9-ef5398dc3567] [replace facts] www-node2.site
>>> 2014-05-20 05:35:01,625 INFO [c.p.p.command]
>>> [f095ef50-4b71-41d6-ba83-6dd4a0030fe2] [replace facts] parser2.site
>>> 2014-05-20 05:35:02,107 INFO [c.p.p.command]
>>> [539a78de-1743-4135-adeb-c5ed3b43cc24] [replace catalog] lb-node2.site
>>> 2014-05-20 05:35:02,379 INFO [c.p.p.command]
>>> [f4b7fc4d-cccd-4fb9-881e-da4007d57f6e] [replace facts] monitor.site
>>> 2014-05-20 05:35:03,945 INFO [c.p.p.command]
>>> [f49a6fcd-ca63-49eb-bbec-7712ba590675] [replace catalog] www-node2.site
>>> 2014-05-20 05:35:05,350 INFO [c.p.p.command]
>>> [6913aceb-0d60-4701-9d31-d84af8f92374] [replace facts] db-node1.site
>>> 2014-05-20 05:35:05,587 INFO [c.p.p.command]
>>> [bd9796df-e989-4788-addc-d31f2d2dda3a] [replace catalog] monitor.site
>>> 2014-05-20 05:35:05,848 INFO [c.p.p.command]
>>> [f9215b15-f646-403a-b4d3-21fd532e1c06] [replace facts] puppet.site
>>> 2014-05-20 05:35:05,871 INFO [c.p.p.command]
>>> [7820d545-1c5f-4b0f-a2e8-0dcbc12d4571] [replace facts] db-node2.site
>>> 2014-05-20 05:35:07,642 WARN [o.e.j.s.HttpChannel]
>>> /v3/commands?checksum=e51aa526e2236306cff22ef33c8cb4467166f9c0
>>> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
>>> expired: 30002/30000 ms
>>> at org.eclipse.jetty.util.BlockingCallback.block(BlockingCallback.java:101)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:62)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:392)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:161)
>>> ~[puppetdb.jar:na]
>>> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[na:1.7.0_55]
>>> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[na:1.7.0_55]
>>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[na:1.7.0_55]
>>> at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.7.0_55]
>>> at java.io.BufferedReader.fill(BufferedReader.java:154) ~[na:1.7.0_55]
>>> at java.io.BufferedReader.read(BufferedReader.java:175) ~[na:1.7.0_55]
>>> at clojure.core$slurp.doInvoke(core.clj:6396) ~[puppetdb.jar:na]
>>> at clojure.lang.RestFn.invoke(RestFn.java:410) ~[puppetdb.jar:na]
>>> at ring.util.request$eval10113$fn__10114.invoke(request.clj:30) ~[na:na]
>>> at clojure.lang.MultiFn.invoke(MultiFn.java:227) ~[puppetdb.jar:na]
>>> at
>>> com.puppetlabs.middleware$payload_to_body_string$fn__10429.invoke(middleware.clj:273)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$verify_content_type$fn__10379.invoke(middleware.clj:128)
>>> ~[na:na]
>>> at com.puppetlabs.puppetdb.http.v3$v3_app$fn__11827.invoke(v3.clj:23)
>>> ~[na:na]
>>> at com.puppetlabs.puppetdb.http.v3$v3_app.invoke(v3.clj:21) ~[na:na]
>>> at
>>> com.puppetlabs.puppetdb.http.server$routes$fn__12370.invoke(server.clj:47)
>>> ~[na:na]
>>> at net.cgrand.moustache$alter_request$fn__10771.invoke(moustache.clj:54)
>>> ~[na:na]
>>> at com.puppetlabs.puppetdb.http.server$routes.invoke(server.clj:47) ~[na:na]
>>> at ring.middleware.resource$wrap_resource$fn__12339.invoke(resource.clj:24)
>>> ~[na:na]
>>> at ring.middleware.params$wrap_params$fn__10714.invoke(params.clj:58)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_authorization$fn__10346.invoke(middleware.clj:40)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_certificate_cn$fn__10350.invoke(middleware.clj:55)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_default_body$fn__10354.invoke(middleware.clj:62)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__10401$fn__10410.invoke(middleware.clj:207)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069$fn__10070.invoke(metrics.clj:14)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown
>>> Source) ~[na:na]
>>> at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
>>> at
>>> com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069.invoke(metrics.clj:14)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069$fn__10070.invoke(metrics.clj:14)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown
>>> Source) ~[na:na]
>>> at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
>>> at
>>> com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__10068$fn__10069.invoke(metrics.clj:14)
>>> ~[na:na]
>>> at com.puppetlabs.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:17)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__10401.invoke(middleware.clj:206)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_globals$fn__10358.invoke(middleware.clj:73)
>>> ~[na:na]
>>> at
>>> com.puppetlabs.middleware$wrap_with_debug_logging$fn__10342.invoke(middleware.clj:27)
>>> ~[na:na]
>>> at
>>> puppetlabs.trapperkeeper.services.webserver.jetty9_core$proxy_handler$fn__16765.invoke(jetty9_core.clj:66)
>>> ~[na:na]
>>> at
>>> puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown
>>> Source) ~[na:na]
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1048)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:325)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.server.Server.handle(Server.java:459)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280)
>>> ~[puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
>>> [puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
>>> [puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>>> [puppetdb.jar:na]
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>>> [puppetdb.jar:na]
>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
>>> Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
>>> 30002/30000 ms
>>> at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:153)
>>> ~[puppetdb.jar:na]
>>> at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
>>> ~[puppetdb.jar:na]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> ~[na:1.7.0_55]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>> ~[na:1.7.0_55]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>> ~[na:1.7.0_55]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> ~[na:1.7.0_55]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> ~[na:1.7.0_55]
>>> ... 1 common frames omitted
>>>
>>>
>>>
>>> On May 20, 2014, at 9:38 AM, Spencer Krum <[email protected]> wrote:
>>>
>>> This looks like a puppetdb error. Can you check the PuppetDB logs and post
>>> any relevant errors?
>>>
>>> Thanks,
>>> Spencer
>>>
>>>
>>> On Mon, May 19, 2014 at 10:02 PM, Sergey Arlashin
>>> <[email protected]> wrote:
>>>>
>>>> Hi!
>>>>
>>>> Every now and then I get the following error while running puppet agent.
>>>>
>>>> Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
>>>> Failed to submit 'replace catalog' command for prod2.site to PuppetDB at
>>>> puppet.site:8081: [500 java.util.concurrent.TimeoutException: Idle timeout
>>>> expired: 30000/30000 ms] <html><head><meta http-equiv="Content-Type"
>>>> content="text/html;charset=ISO-8859-1"/><title>Error 500
>>>> </title></head><body><h2>HTTP ERROR: 500</h2><p>Problem accessing
>>>> /v3/commands. Reason:<pre> java.util.concurrent.TimeoutException: Idle
>>>> timeout expired: 30000/30000 ms</pre></p><hr /><i><small>Powered by
>>>> Jetty://</small></i></body></html>
>>>>
>>>> Does anybody know how I can get rid of this error?
>>>>
>>>> Thanks in advance!
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Sergey Arlashin
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "Puppet Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/puppet-users/8A2FC518-6B42-42D5-8524-12D39B94DBD4%40gmail.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> --
>>> Spencer Krum
>>> (619)-980-7820
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/puppet-users/CADt6FWNbXASsOh6_hzcSQAeJjzu8cZYTPG4qkYxTBoe%2B6heLtw%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Puppet Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/puppet-users/FFF9F6EE-8B3A-48E4-A78B-42932EF41A3B%40gmail.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3D0SGVVbO%2Bevm6f3HYBx0RZSZMdmW1UYALHc-6Xk1isqA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/0CD567ED-2E73-4C0B-968C-057BA0FB0396%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
