We just rolled this change into our prod systems today. Things are working great, with about 600 active sessions tomcat has a thread pool of 110 threads HTTP handling threads and uPortal has a pool of 33 portlet execution threads. We'll be keeping an eye on it over the week to see if we have any problems but it is working great so far and only having enough threads live to handle the current load is nice as well.

On 04/05/2012 02:56 PM, Eric Dalquist wrote:
I'd like to consider removing the option for queuing portlet execution workers. Looking at the more web-server centric software side (apache, tomcat) they have a dynamic pool of workers (portlet execution threads in our case) that grows and shrinks with demand. These applications are configured with a max number of workers and when that limit is hit requests hang for a specified period of time then fail.

In uPortal (2, 3 and 4) we've had this two part setup, there is a pool of worker threads for executing portlets and a queue which fills up when all the worker threads are busy. While the intention here is good (queue up work to do once the pool isn't so busy) the result is bad (use more memory while the server is already under load).

I'd like to propose that in uPortal 4, where we are using an easily configured ExecutorService instead of a custom thread pool we do away with the portlet worker queue. As portal requests happen and portlet executions are requested these are handed off directly to worker threads. If we're at the max threads limit the request to queue the worker waits for some configurable timeout (say 2 seconds by default) and then fails with a good message explaining why that portlet couldn't render and logging a message telling admins what to do and look for.

Especially with the work that Drew has done around preventing execution of miss-behaved portlets deployers shouldn't have as many cases of bad portlets eating up all the threads and if you do start running out of portlet threads it means you're actually running out of server capacity and no queue will help that problem.


-Eric


Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

Reply via email to