On Thu, Sep 6, 2012 at 7:29 PM, Claudio Corsi <[email protected]> wrote: > Claus, > > Why did you not consider using the keep alive option of the thread pool? > This would reclaim threads that have been idle after a given certain time and > those resources would of been reclaimed. The only consideration is that you > would need to reduce the number of active threads to a small value since this > is useful for created threads that exceed the minimum thread pool size. >
The thread pools in AMQ do have keep alive times, usually set to 30 sec, and some have lower values such as 10 sec etc. And any of the patches/commits do not change how these thread pools is configured out of the box. Its the fact the old code uses a shared static thread pool (for many of its async tasks related to stopping/closing), and that thread pool is newer shutdown. Which leaves the thread pool around as well its threads. And as well some thread pools wasn't shutdown either (which could potential leave threads around, until they terminate). And btw some thread pools such as the scheduled thread pool (AFAIR this is how they have been implemented in the JDK) never terminate, so its important to shutdown these manually. Also the goal when shutting down is to give all active and running threads a chance to complete. And if not, then become more aggressive and shutdown them using interrupts etc. In the end when your app with embedded AMQ Client (or broker) is shutdown and being undeployed, the goal is to not have any threads or leaks running around. As you have looked in the JIRA ticket about that "Tomcat report" you have seen that it helps fixing this by shutting down that shared thread pool. So we are on a path for a solution. Just taking a bit more time, to have a common API to shutdown thread pools in AMQ. Then we can better "control" this. > --Claudio > > >>________________________________ >> From: Claus Ibsen <[email protected]> >>To: [email protected] >>Sent: Thursday, September 6, 2012 11:20 AM >>Subject: Re: [HEADS UP] - Graceful shutdown of AMQ and fixing some leaks when >>doing hot-deployment >> >>Hi >> >>Okay I have been running a bunch of unit tests, and I can see that the >>tests from activemq-core, is kinda dependent on the old behavior >>of shutting down thread pools aggressively using shutdownNow or no >>attempt to wait for in progress tasks to complete gracefully. >> >>So the changes being committed is currently to keep same behavior of >>shutting down. Then we can gently switch over to >>graceful shutdown as we fix the tests and identify the ones to switch over. >> >>So ThreadPoolUtils have 3 set of shutdown methods >>- shutdown >>- shutdownNow >>- shutdownGraceful >> >>The first 2 is just like on the JDK executor service. >>The last one is the graceful one. >> >> >> >>On Thu, Sep 6, 2012 at 2:47 PM, Claus Ibsen <[email protected]> wrote: >>> Hi >>> >>> Recently I have been diving into fixing some leaks in ActiveMQ. It >>> started with the activemq-pool component which caused a Camel JMS >>> client to become slower, due memory leaks not being released. >>> >>> The activemq-pool has been fixed now, and runs flawless again. >>> On a side note Tim Bish is working on improving the activemq-pool to >>> be a better pool by leveraging more logic from commons-pool on the >>> connection side as well. Beforehand it was only the sessions that was >>> leveraged. >>> >>> Anyway continuing with the Camel JMS client, we have reports, about >>> leaks in Tomcat, when you redeploy an application. So I have been >>> tracking those down, and got most of it fixed. Have some pending >>> commits for the JMS client. >>> >>> However on embedded a full blow ActiveMQ broker in a web container as >>> Tomcat, and then supporting hot deployment, surfaced some issues with >>> some thread pools not being closed graceful. Instead the pools is >>> closed aggresively, and as some of them have idle threads timeout of >>> 30 seconds, then those threads dont timeout before the application is >>> redeployed (assuming you re-deploy the app; to remedy this you would >>> have to stop the app, and wait for > 30 sec, and then deploy again). >>> >>> As we have some good graceful shutdown code in Camel, I have ported >>> part of that to AMQ, in the ThreadPoolUtils class in the util package. >>> There is a number of shutdown methods, to use for graceful shutdown. >>> When a graceful shutdown of a thread pool is in the works, and if it >>> takes some times, then we now log every 5th second about the shutdown >>> in progress. >>> >>> >>> There is a number of tickets in JIRA about this recent work (all the >>> stuff above, etc.) >>> https://issues.apache.org/jira/browse/AMQ-4026 >>> https://issues.apache.org/jira/browse/AMQ-4019 >>> https://issues.apache.org/jira/browse/AMQ-3451 >>> https://issues.apache.org/jira/browse/AMQ-4016 >>> https://issues.apache.org/jira/browse/AMQ-3959 >>> https://issues.apache.org/jira/browse/AMQ-3797 >>> https://issues.apache.org/jira/browse/AMQ-4008 >>> https://issues.apache.org/jira/browse/AMQ-4011 >>> https://issues.apache.org/jira/browse/AMQ-3997 >>> >>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> FuseSource >>> Email: [email protected] >>> Web: http://fusesource.com >>> Twitter: davsclaus, fusenews >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >> >> >> >>-- >>Claus Ibsen >>----------------- >>FuseSource >>Email: [email protected] >>Web: http://fusesource.com >>Twitter: davsclaus, fusenews >>Blog: http://davsclaus.com >>Author of Camel in Action: http://www.manning.com/ibsen >> >> >> -- Claus Ibsen ----------------- FuseSource Email: [email protected] Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
