I've thought of a significant simplification, based on the idea that I only need samples of the task count.
I was going to try to use a concurrent queue and deal with observations getting out of order. I just realized I can make it much cleaner and simpler by using a ReentrantLock. The manager thread would use lockInterruptibly, and wait indefinitely to get the lock if another thread holds it. A thread that has just made a change to the task count would use tryLock(). If it gets the lock, it can add its report to a simple LinkedList. If it fails to get the lock it would just carry on without collecting a sample. Either the management thread is processing collected data, or another thread is logging the task count at about the same time. Either way it does not need to collect a sample. This strategy automatically throttles the number of observations that get collected for the manager to consider, though I may also want a way to further limit the number of messages. However, that can be done more simply, and with better information, inside code controlled by the lock. Patricia On 8/9/2011 12:36 PM, Patricia Shanahan wrote:
I'm investigating extended ThreadPoolExecutor to get an Executor that bases the number of threads on the mean number of tasks over an interval. If successful, that would flatten out sudden peaks in the task load, using the queue as a buffer, but would make the number of threads match a sustained load, so that there would be no unnecessary bottleneck. The current TaskManager allows specification of e.g. one thread per three tasks, which does result in a lower rate of change but can leave the number of threads a fraction of the number of tasks in a sustained workload. Here is a general outline of a design: Methods that change the number of tasks would log the new number of tasks and a timestamp on a concurrent queue. As a refinement, there may be an option to use a ThreadLocal Random to log only with a specified probability to bound the number of messages on the queue. A thread (more about which thread later) would periodically drain the queue into a local TreeSet, getting them sorted, and then calculate the mean number of tasks and adjust the thread count accordingly. The big design problem is which thread does this work. Here are the options I'm considering: 1. Add it to the tasks, so that a ThreadPoolExecutor worker thread does it. The downside is that all the existing worker threads could get busy on long running tasks and tasks are building up in the queue. Nothing would be done about it until a worker thread finishes its current task. 2. Do it from the execute method, in whatever thread is adding the task. Again, we could get into situations in which nothing is done about a situation that could be improved by adding threads. The effect would be to make the execute method randomly slow from the point of view of its caller. 3. Use a dedicated thread that sleeps until it is time to reconsider the number of tasks, processes the queue of log messages, does any adjustment, and then goes back to sleep. This is the most reliable in terms of getting adjustments made when they are needed, but costs the overhead of an extra thread that is just managing the thread pool. Any views, alternatives? Thanks, Patricia
