> The current problem is that the mis-use of the Scheduler
I'd prefer to simply say that it is being applied for a purpose that does
not match it's design point. :-)
> Basically, with only five concurrent connections,
> you can easily kill the Scheduler implementation
> with consistent load.
I've done that several times today with Harmeet, testing various changes to
his idea of how to make use of the Scheduler or TimerTask. Eiter way the
memory usage kills the server. Here is a set of tests:
time,messages,data(K),errors,connections,SSL connections
15:16,705,606,32,490,0
15:17,700,587,23,461,0
15:18,464,374,38,308,0
15:19,4,2,223,0,0
time,messages,data(K),errors,connections,SSL connections
15:39,687,584,29,476,0
15:40,629,536,24,425,0
15:41,523,420,36,339,0
time,messages,data(K),errors,connections,SSL connections
16:19,868,730,39,614,0
16:20,858,738,27,572,0
time,messages,data(K),errors,connections,SSL connections
16:33,1044,872,21,709,0
16:34,872,740,48,570,0
16:35,0,0,192,0,0
Those are 4 separate tests, each one stopping when the server died. Each
test shows James dying within 2-3 minutes. The final test was with
Harmeet's latest code, and with sendMail commented out.
Here is my current log (still running) for Peter's server with the sendMail
commented out:
time,messages,data(K),errors,connections,SSL connections
18:27,579,479,45,410,0
18:28,682,575,3,465,0
18:29,692,554,4,450,0
18:30,679,572,16,453,0
18:31,682,560,5,453,0
18:32,691,568,4,455,0
18:33,690,565,5,454,0
18:34,678,564,6,459,0
18:35,677,565,2,452,0
18:36,692,566,7,448,0
18:37,684,577,10,451,0
18:38,681,552,10,463,0
18:39,674,568,16,460,0
18:40,691,577,9,459,0
18:41,688,565,6,455,0
18:42,679,555,4,464,0
18:43,688,570,4,465,0
18:44,687,551,8,457,0
18:45,681,558,4,450,0
18:46,680,564,12,461,0
18:47,695,598,6,446,0
18:48,680,577,3,456,0
18:49,676,562,3,460,0
18:50,684,586,11,457,0
18:51,686,554,7,465,0
18:52,694,578,7,448,0
18:53,689,563,5,462,0
18:54,683,568,2,454,0
18:55,681,572,5,453,0
18:56,683,563,6,460,0
18:57,687,571,4,463,0
18:58,688,571,12,454,0
18:59,690,573,7,458,0
19:00,679,557,8,445,0
19:01,693,576,2,449,0
19:02,637,532,2,428,0
19:03,684,572,2,451,0
That's 36 minutes. Performance appears to be consistent, CPU and memory
appear to be consistent according to Peter on his end. We're leaving this
running. Last night I was not able to achieve this result, but he has since
made changes to the watchdog implementation, and commented out sendMail
(turns out that there may be some issues in the spool manager, too).
> The priority queue will hold on to the events, causing out
> of memory errors. This is one reason why I believe the
> scheduler is the wrong approach.
Harmeet believes that he can come up with a new data structure that will
provide for rapid change. I've pointed out to him that it needs to support
1000s of resets per second (or more). He believes that using two threads
per connection doesn't scale. That is his specific issue. I suggest that
as the number of connections goes up, a priority queue problem has scaling
problems, and for James' purposes, I don't believe that the two thread
solution doesn't scale suitably. However, the interface to the watchdog
mechanism doesn't the two thread solution, so it can be changed. The
interface to the Scheduler, however, isn't designed to be a watchdog.
The Watchdog interface is simple:
public interface Watchdog {
void startWatchdog();
void resetWatchdog();
void stopWatchdog();
}
The current implementation(s) have a constructor like:
public WatchdogImpl(long timeout, WatchdogTarget target);
That's it. There is no lookup in the interface, neither is there a mandate
for either a queued or multi-threaded implementation. If we needed to
handled so many watchdogs that we didn't want to spawn a thread for each, a
new Watchdog implementation could decide how to handle it. One thread,
multiple threads balancing the length of a priority queue, or whatever. I
think that this is a simple, reusable interface that does what we need, and
the current implementation is demonstrating that it can handle the load
we're putting on it.
--- Noel
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
