Restlet thread pool exhaustion
Hello Restlet folks. I'm the lead developer on the WattDepot system, which collects and stores energy data, and we use Restlet on both server and client sides. We recently switched from Restlet 1.1.8 to 2.0.11 and now have some pretty severe stability problems. Our sensor client spawns multiple threads (one per power meter being polled), each of which does PUTs to our server. The problem we have run into is that after a period of time (about 2-3 days), the server stops accepting connections. The ending log file entries on the server look something like this: Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution WARNING: Unable to run the following server-side task: Read connection messages: true Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service state: Full Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service tasks: 0 queued, 10 active, 71196 completed, 71206 scheduled. Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service thread pool: 1 core size, 10 largest size, 10 maximum size, 10 current size Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.Controller run INFO: Stop accepting new connections and transactions. Consider increasing the maximum number of threads. At this point, using netstat, I can see many established TCP connections to the server, and the server does not respond to further connections. Terminating the client causes the TCP connections to go away (as one would expect), but the server still does not respond to new connections. I've created a client that sends bogus data very quickly so that I can more easily reproduce the problem (and test fixes). The amount of time before the problem occurs seems to vary from as little as 5 minutes, to 35 minutes (or more) so the time to failure is not deterministic. I've searched this mailing list, and found helpful threads like this one that explain how to increase the maximum number of threads for HTTP listening: http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447viewType=browseAlldsMessageId=2625612 However, increasing maxThreads from the default of 10 seems like it would just postpone the thread exhaustion, rather than solve the underlying problem. Another classic cause of thread exhaustion appears to be a failure to release ClientResources after use, as discussed in this very helpful thread: http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447viewType=browseAlldsMessageId=2885182#messagefocus The SimpleServerPut example code given in that thread produces log messages very similar to the ones we see, and when a call to clientResource.release() is added to SimpleServerPut the problems go away. I've read over our relevant client code, and I think we are calling clientResource.release() after each use, but it is possible there is some path where it isn't getting called. So here are my questions: 1) What's going on when the server gets wedged? If a client was holding open too many connections, I could see how the server would stop accepting new ones, but when I stop the client, the server remains wedged. Is Restlet leaking threads somehow? It seems like this should never happen. 2) Is there a good way to tell if we are failing to release a ClientResource somewhere? 3) Is it legal to call ClientResource.release() more than once? I might try wrapping the client methods in a big try/catch/finally to ensure that release() is always being called. 4) Having to ensure that each ClientResource is explicitly released seems like a pain. Is there a way to avoid this? This stability problem is preventing us starting a beta test of a project that relies on WattDepot, so I'm eager to resolve this quickly. I'm open to any suggestions on how to debug, fix, or workaround this problem. Would upgrading to Restlet 2.1 help? Thank you! -- Robert Brewer http://excitedcuriosity.wordpress.com/ https://www.facebook.com/kukuicup -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2976364
Re: Restlet thread pool exhaustion
Hi Robert, neat project you have going on there. I will answer a tiny part of your question, I've load tested 2.1-RC5 heavily and after careful tweaking I've yet to reach the worker limit scenario you detail here. My setup is: 150 maxThreads, 10 minThreads, 145 lowThreads, maxQueued 20 but ofc YMMV. On 6 Jul 2012, at 04:40, Robert Brewer rbre...@lava.net wrote: Hello Restlet folks. I'm the lead developer on the WattDepot system, which collects and stores energy data, and we use Restlet on both server and client sides. We recently switched from Restlet 1.1.8 to 2.0.11 and now have some pretty severe stability problems. Our sensor client spawns multiple threads (one per power meter being polled), each of which does PUTs to our server. The problem we have run into is that after a period of time (about 2-3 days), the server stops accepting connections. The ending log file entries on the server look something like this: Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution WARNING: Unable to run the following server-side task: Read connection messages: true Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service state: Full Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service tasks: 0 queued, 10 active, 71196 completed, 71206 scheduled. Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.BaseHelper$1 rejectedExecution INFO: Worker service thread pool: 1 core size, 10 largest size, 10 maximum size, 10 current size Jul 5, 2012 3:48:33 PM org.restlet.engine.http.connector.Controller run INFO: Stop accepting new connections and transactions. Consider increasing the maximum number of threads. At this point, using netstat, I can see many established TCP connections to the server, and the server does not respond to further connections. Terminating the client causes the TCP connections to go away (as one would expect), but the server still does not respond to new connections. I've created a client that sends bogus data very quickly so that I can more easily reproduce the problem (and test fixes). The amount of time before the problem occurs seems to vary from as little as 5 minutes, to 35 minutes (or more) so the time to failure is not deterministic. I've searched this mailing list, and found helpful threads like this one that explain how to increase the maximum number of threads for HTTP listening: http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447viewType=browseAlldsMessageId=2625612 However, increasing maxThreads from the default of 10 seems like it would just postpone the thread exhaustion, rather than solve the underlying problem. Another classic cause of thread exhaustion appears to be a failure to release ClientResources after use, as discussed in this very helpful thread: http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447viewType=browseAlldsMessageId=2885182#messagefocus The SimpleServerPut example code given in that thread produces log messages very similar to the ones we see, and when a call to clientResource.release() is added to SimpleServerPut the problems go away. I've read over our relevant client code, and I think we are calling clientResource.release() after each use, but it is possible there is some path where it isn't getting called. So here are my questions: 1) What's going on when the server gets wedged? If a client was holding open too many connections, I could see how the server would stop accepting new ones, but when I stop the client, the server remains wedged. Is Restlet leaking threads somehow? It seems like this should never happen. 2) Is there a good way to tell if we are failing to release a ClientResource somewhere? 3) Is it legal to call ClientResource.release() more than once? I might try wrapping the client methods in a big try/catch/finally to ensure that release() is always being called. 4) Having to ensure that each ClientResource is explicitly released seems like a pain. Is there a way to avoid this? This stability problem is preventing us starting a beta test of a project that relies on WattDepot, so I'm eager to resolve this quickly. I'm open to any suggestions on how to debug, fix, or workaround this problem. Would upgrading to Restlet 2.1 help? Thank you! -- Robert Brewer http://excitedcuriosity.wordpress.com/ https://www.facebook.com/kukuicup -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2976364 -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2976752
Re: Restlet java beans pojo gwt
Thank you Thierry for the clear mail. Best regards, On Thu, Jul 5, 2012 at 1:23 PM, Thierry Boileau thierry.boil...@restlet.com wrote: Hello Danilo, I work on a similar kind of project and we decided to split the source code into three eclipse projects : client, common, server. Common mainly contains the Java beans and is not based on Restlet. Client is the GWT project and depends on Common. Server depends on Common also. A build script (ant+jenkins) allows us to generate a common.jar archive that can be shared and used, for example, by the Web application (packaged as a war file, since the customer uses Weblogic). As a side note, the build script generates also the static GWT files, distinct jars for the server code (one for the dao, one for the server code except the dao). As the server is based on annotated interfaces, we could imagine to export these annotated interfaces in a separate jar in order to help client projects based on Restlet to wrap the interfaces and automatically cope with serialization/deserialization. One can also add Maven repository in order to store the distinct releases of the generated archives. Regarding the third party client, you're right. Either your code reuse annotated interfaces, or not. In the first case, keep attention to the version of the beans on both sides (server and client) otherwise the automatic serialization may fail. In the second case, I think I will use one of the textual formats you mention. I hope this will help you. Best regards, Thierry Boileau Hi all, Sorry for the long text but I have to provide the such information in order to solve my doubt. I have the following scenario: - Server-side: - Java SE - Jetty - Restlet framework - My system API - Client-side (mandatory): - GWT consuming My system API - Client-side (planned): - 3rd part applications consuming My system API I'm concerned about the project organization and the team development over it. I don't know if I am in the correct way to solve that. I've tried creating an project to keep the java POJOs/Beans and adding its reference to the Java and GWT project. But I got some reference errors in GWT (I still trying to solve it right now ... LOL) So, what is the best way to share the java POJO/Beans code to the both projects (Java project and GWT)? I would like to share this source between both projects and keep track of updates. I mentioned the client-side (planned) / 3rd part applications because I read the wiki ( http://wiki.restlet.org/docs_2.0/13-restlet/21-restlet/318-restlet/303-restlet.html). In this case, how to keep my server loose coupled? For example, using JSON / XML / ? instead of java serialization. Best regards, -- Danilo Rosetto Muñoz munozdan...@gmail.com http://br.linkedin.com/in/danilomunoz -- Danilo Rosetto Muñoz munozdan...@gmail.com http://br.linkedin.com/in/danilomunoz -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2976759
RE: Enable only selected cipher suites
Andreas, I am trying to run the code you provided, I am unable to find the class org.restlet.ext.ssl.DefaultSslContextFactory in the jar file org.restlet.ext.ssl.jar; I am only able to find org.restlet.ext.ssl.PkixSslContextFactory and JsslutilsSslContextFactory I am also having issues with enabling specific Cipher Suites using restlet Client, any pointers on this will also help. Thanks, Enoch -- View this message in context: http://restlet-discuss.1400322.n2.nabble.com/Enable-only-selected-cipher-suites-tp7578116p7578181.html Sent from the Restlet Discuss mailing list archive at Nabble.com. -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2977136
Re: Restlet thread pool exhaustion
On 6 Jul 2012, at 06:06:38, Ioannis Mavroukakis imavrouka...@gmail.com wrote: Hi Robert, neat project you have going on there. I will answer a tiny part of your question, I've load tested 2.1-RC5 heavily and after careful tweaking I've yet to reach the worker limit scenario you detail here. My setup is: 150 maxThreads, 10 minThreads, 145 lowThreads, maxQueued 20 but ofc YMMV. Thanks for the reply Ioannis. So did the careful tweaking you did come down to just those parameters above? -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2977064
