Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-19 Thread 'Jesus' Jeff Rogers

dhogaza@PACIFIER.COM wrote:

I have three servers running identical installations of
AOLserver/3.3.1+ad13.  On two (development and production, very low and
relatively low traffic volumes respectively) all scheduled procs have
stopped firing.



My God, it sounds to me like you're all being bit by the Y2.006K problem! :)
  
That answer is closer than you think (at least if everyone is having the 
same problem I was) ... actually it's Y2.038K


-J


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-19 Thread dhogaza
 dhogaza@PACIFIER.COM wrote:

 My God, it sounds to me like you're all being bit by the Y2.006K
 problem! :)

 That answer is closer than you think (at least if everyone is having the
 same problem I was) ... actually it's Y2.038K

Yes, indeed, I suspect everyone here on the list is aware of the linux
32-bit time issue.


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-17 Thread Stan Kaufman

Zachary Shaw wrote:

before may 13th nanosleep was in the form

[pid   614] nanosleep({0, 34478},  unfinished ...

after the 12th there were nanosleeps in the form
[pid   614] nanosleep({9, 934211000},  unfinished ...
  


This looks like a blatant clue -- to someone who understands this. 
nanosleep is a *nix lib command 
(http://linux.about.com/library/cmd/blcmdl2_nanosleep.htm), which makes 
me think the problem may not be with AOLServer. Simple-mindedly grepping 
for nanosleep through the ad33.13 source and the aolserver includes etc 
returns nothing, so the code isn't calling nanosleep directly. I wonder 
if this is a TCL issue, though grepping through tcl source also returns 
nothing. Hmm...



one additional piece of information.  It appears that our logs stopped rolling
on may 13th.
  


My logs are rolling just fine -- both the error and access logs. Weirder 
and weirder.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Janine Sisk
In addition to the below, there are at least two of us over at  
openacs.org reporting that scheduled procs stopped firing over the  
weekend.  I have one site left on 3.3+ad13 and the other guy says he  
has 3.2+ad12.


This site hasn't been upgraded because the client didn't want to  
upgrade, so any pointers on fixing this would be greatly appreciated.


janine

On May 15, 2006, at 9:55 AM, Guan Yang wrote:

I've experienced this problem over the weekend on half a dozen  
servers in three different locations:


After starting up, AOLserver (mostly 3.4.2, also a 3.5.11) begins  
allocating a lot of virtual memory. When I follow the process in  
top, the virtual memory counter (but not real memory) rapidly  
increases over the course of 10 to 20 seconds, until it hits the  
limit of 3 GB virtual memory in user space on x86 Linux boxes. The  
process then crashes with this message:


nsthread(3094) error: pthread_create failed in NsThreadCreate:  
Cannot allocate memory


(The process is then restarted by supervise/svscan.)

I couldn't find a fix for the problem and had to migrate to  
AOLserver 4.0.10 (we had already planned this for next month)  
immediately. But I find it a bit weird that practically every  
AOLserver 3 installation that I know of encounters this problem  
over the same weekend. Just bad luck? Date sensitive code?


Guan


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the  
Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Janine Sisk
By the way, one more thing:  I have three sites running on this  
server - staging, live and an old keepalive instance.  The live site  
is the only one exhibiting the scheduled proc problem; it's also the  
only one that gets any significant traffic.  So I'm thinking it's  
some kind of overflow where it only causes problems when a busy site  
is using memory and ends up overwriting something.


janine

On May 16, 2006, at 12:10 AM, Janine Sisk wrote:

In addition to the below, there are at least two of us over at  
openacs.org reporting that scheduled procs stopped firing over the  
weekend.  I have one site left on 3.3+ad13 and the other guy says  
he has 3.2+ad12.


This site hasn't been upgraded because the client didn't want to  
upgrade, so any pointers on fixing this would be greatly appreciated.


janine

On May 15, 2006, at 9:55 AM, Guan Yang wrote:

I've experienced this problem over the weekend on half a dozen  
servers in three different locations:


After starting up, AOLserver (mostly 3.4.2, also a 3.5.11) begins  
allocating a lot of virtual memory. When I follow the process in  
top, the virtual memory counter (but not real memory) rapidly  
increases over the course of 10 to 20 seconds, until it hits the  
limit of 3 GB virtual memory in user space on x86 Linux boxes. The  
process then crashes with this message:


nsthread(3094) error: pthread_create failed in NsThreadCreate:  
Cannot allocate memory


(The process is then restarted by supervise/svscan.)

I couldn't find a fix for the problem and had to migrate to  
AOLserver 4.0.10 (we had already planned this for next month)  
immediately. But I find it a bit weird that practically every  
AOLserver 3 installation that I know of encounters this problem  
over the same weekend. Just bad luck? Date sensitive code?


Guan


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave  
the Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the  
Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Tom Jackson
Janine,

Are these completely unrelated installs wrt Guan's sites?

I seem to have forgotten exactly why it is difficult or time consuming to 
upgrade from 3.x to 4.x. Does anyone have a summary of the problems?

tom jackson

On Tuesday 16 May 2006 00:10, Janine Sisk wrote:
 In addition to the below, there are at least two of us over at
 openacs.org reporting that scheduled procs stopped firing over the
 weekend.  I have one site left on 3.3+ad13 and the other guy says he
 has 3.2+ad12.

 This site hasn't been upgraded because the client didn't want to
 upgrade, so any pointers on fixing this would be greatly appreciated.


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Janine Sisk
I realized after I sent this last night that the other two sites are  
probably still working because they haven't been restarted in quite a  
while.  The live site gets restarted fairly frequently, so it started  
exhibiting the problem right away.


janine

On May 16, 2006, at 12:18 AM, Janine Sisk wrote:

By the way, one more thing:  I have three sites running on this  
server - staging, live and an old keepalive instance.  The live  
site is the only one exhibiting the scheduled proc problem; it's  
also the only one that gets any significant traffic.  So I'm  
thinking it's some kind of overflow where it only causes problems  
when a busy site is using memory and ends up overwriting something.


janine

On May 16, 2006, at 12:10 AM, Janine Sisk wrote:

In addition to the below, there are at least two of us over at  
openacs.org reporting that scheduled procs stopped firing over the  
weekend.  I have one site left on 3.3+ad13 and the other guy says  
he has 3.2+ad12.


This site hasn't been upgraded because the client didn't want to  
upgrade, so any pointers on fixing this would be greatly appreciated.


janine

On May 15, 2006, at 9:55 AM, Guan Yang wrote:

I've experienced this problem over the weekend on half a dozen  
servers in three different locations:


After starting up, AOLserver (mostly 3.4.2, also a 3.5.11) begins  
allocating a lot of virtual memory. When I follow the process in  
top, the virtual memory counter (but not real memory) rapidly  
increases over the course of 10 to 20 seconds, until it hits the  
limit of 3 GB virtual memory in user space on x86 Linux boxes.  
The process then crashes with this message:


nsthread(3094) error: pthread_create failed in NsThreadCreate:  
Cannot allocate memory


(The process is then restarted by supervise/svscan.)

I couldn't find a fix for the problem and had to migrate to  
AOLserver 4.0.10 (we had already planned this for next month)  
immediately. But I find it a bit weird that practically every  
AOLserver 3 installation that I know of encounters this problem  
over the same weekend. Just bad luck? Date sensitive code?


Guan


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave  
the Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave  
the Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the  
Subject: field of your email blank.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Guan Yang

Tom,

Janine's sites are completely unrelated with mine. And I've seen this  
problem on three different unrelated sets of servers.


Our upgrade from 3.x to 4.x was not that painful, the most time- 
consuming part was rewriting approximately 35 configuration files for  
the tons of services we have running: adding nsdb, removing tdom from  
the config file, converting .ini config files, and the changes in  
nsopenssl configuration.


Guan

On 16 May  2006, at 15:57 , Tom Jackson wrote:

Janine,

Are these completely unrelated installs wrt Guan's sites?

I seem to have forgotten exactly why it is difficult or time  
consuming to

upgrade from 3.x to 4.x. Does anyone have a summary of the problems?

tom jackson

On Tuesday 16 May 2006 00:10, Janine Sisk wrote:

In addition to the below, there are at least two of us over at
openacs.org reporting that scheduled procs stopped firing over the
weekend.  I have one site left on 3.3+ad13 and the other guy says he
has 3.2+ad12.

This site hasn't been upgraded because the client didn't want to
upgrade, so any pointers on fixing this would be greatly appreciated.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Dave Siktberg
Janine Sisk wrote:
 In addition to the below, there are at least two of us over at
 openacs.org reporting that scheduled procs stopped firing over the
 weekend.  I have one site left on 3.3+ad13 and the other guy says he
 has 3.2+ad12.

I have three servers running identical installations of
AOLserver/3.3.1+ad13.  On two (development and production, very low and
relatively low traffic volumes respectively) all scheduled procs have
stopped firing.  On both of these, the logs show this behavior began after
restarting AOL Server last night around 11PM EDT.  Scheduled procs on both
were working fine up until the restart -- then none.  Scheduled procs are
still running fine on the third server (throw-away test, infinitesimal
volume), where AOL Server has not been restarted for several weeks.  I would
love to help someone who knows the internals figure this out.  I'm leaving
the third server untouched but am happy to use it to help someone figure out
what's going on.

I can also fiddle with my development server.  Could this be a date-related
problem?  I think I'll set the system date back several days, restart, and
see what happens.  Any other ideas?

Any other information I can provide or things I can do to help debug this?
As with Janine, upgrading AOLServer is not a feasible option in the short
term.

Dave


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Nathan Folkman

[EMAIL PROTECTED] wrote:

Janine,

Are these completely unrelated installs wrt Guan's sites?

I seem to have forgotten exactly why it is difficult or time consuming to 
upgrade from 3.x to 4.x. Does anyone have a summary of the problems?


tom jackson

On Tuesday 16 May 2006 00:10, Janine Sisk wrote:
  

In addition to the below, there are at least two of us over at
openacs.org reporting that scheduled procs stopped firing over the
weekend.  I have one site left on 3.3+ad13 and the other guy says he
has 3.2+ad12.

This site hasn't been upgraded because the client didn't want to
upgrade, so any pointers on fixing this would be greatly appreciated.




--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.
  
It's been a while, and I'm still on my first cup of coffee, but I think 
your biggest hurdle is probably going to be the loss of ns_share in 
favor of nsv. Also 4.x requires Tcl 8.x. Hope that helps!


- n


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Janine Sisk

On May 16, 2006, at 8:04 AM, Nathan Folkman wrote:

It's been a while, and I'm still on my first cup of coffee, but I  
think your biggest hurdle is probably going to be the loss of  
ns_share in favor of nsv. Also 4.x requires Tcl 8.x. Hope that  
helps!


Ugh.  You're absolutely right - this site is *full* of ns_shares.   
That's probably another reason why we decided not to tackle this a  
while back.  I know the client didn't want to spend any time on the  
old site if it could possibly be avoided.


janine


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Guan Yang

On 16 May  2006, at 17:04 , Janine Sisk wrote:


On May 16, 2006, at 6:57 AM, Tom Jackson wrote:


Are these completely unrelated installs wrt Guan's sites?


Yes.  I also am not seeing the memory problems he is;  it seems  
that 3.2/3.3 and 3.4 are exhibiting different weirdnesses.


I've seen the memory problem on the Linux 2.6 servers. On the few  
servers that run Linux 2.4 I've not encountered the memory problem,  
but the scheduled procs did stop running there.


Guan


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Nathan Folkman

[EMAIL PROTECTED] wrote:

On May 16, 2006, at 8:04 AM, Nathan Folkman wrote:

It's been a while, and I'm still on my first cup of coffee, but I 
think your biggest hurdle is probably going to be the loss of 
ns_share in favor of nsv. Also 4.x requires Tcl 8.x. Hope that 
helps!


Ugh.  You're absolutely right - this site is *full* of ns_shares.  
That's probably another reason why we decided not to tackle this a 
while back.  I know the client didn't want to spend any time on the 
old site if it could possibly be avoided.


janine


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to 
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the 
Subject: field of your email blank.
Yeah, it's unfortunately one of those things that's not backwards 
compatible due to the way 'ns_share variables were created. I think at 
one point we ended up writing a new version of ns_share that was 
compatible with Tcl 8.x, using Tcl traces or something... Seem to 
remember there being a few caveats, but also that it worked well enough 
to provide a temporary migration path until our code was able to be 
updated to use nsv. I don't remember where that code lives, however... 
;-) It might have been checked in as part of the core server, or we 
might have done it as a stand alone module. I'll try to locate for you...


PS - SourceForge finally upgraded their CVS servers, so be sure to check 
the site for information - the CVS server host names have changed.


- n


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread dhogaza
 I have three servers running identical installations of
 AOLserver/3.3.1+ad13.  On two (development and production, very low and
 relatively low traffic volumes respectively) all scheduled procs have
 stopped firing.

My God, it sounds to me like you're all being bit by the Y2.006K problem! :)


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Stan Kaufman
Dave Siktberg david.siktberg at WEBILITY.MD writes:

 restarting AOL Server last night around 11PM EDT.  Scheduled procs on both
 were working fine up until the restart -- then none.  Scheduled procs are

On a development box, I just started up clones of my
AOLserver/3.3.1+ad13/OpenACS 3.2.5 sites, and they appear to run fine --
including scheduled procs. I'm not seeing any VM problems either.

 I can also fiddle with my development server.  Could this be a date-related
 problem?  I think I'll set the system date back several days, restart, and
 see what happens.  Any other ideas?

Using the current date caused no problems for me; what was the effect of
back-dating?

 Any other information I can provide or things I can do to help debug this?
 As with Janine, upgrading AOLServer is not a feasible option in the short
 term.

Same here.

Could this be an OS problem?

Stan


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Nathan Folkman
Ha! ns_share continues to live on, although I wouldn't recommend using 
it, especially in high-load environments... nsv is where it's at! ;-)


$TOP/aolserver/nsd/tclshare.c

AOLserver/4.5.0a (aolserver4_5) for osx built on May  2 2006 at 10:57:19
CVS Tag: $Name:  $
server1:nscp 1 ns_share
wrong # args: should be ns_share ?-init script? varName ?varName ...?

So you should be able to migrate code that contains ns_share up to the 
latest AOLserver versions (4.0.x and upcoming 4.5).


- n


[EMAIL PROTECTED] wrote:

On May 16, 2006, at 8:04 AM, Nathan Folkman wrote:

It's been a while, and I'm still on my first cup of coffee, but I 
think your biggest hurdle is probably going to be the loss of 
ns_share in favor of nsv. Also 4.x requires Tcl 8.x. Hope that 
helps!


Ugh.  You're absolutely right - this site is *full* of ns_shares.  
That's probably another reason why we decided not to tackle this a 
while back.  I know the client didn't want to spend any time on the 
old site if it could possibly be avoided.


janine


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to 
[EMAIL PROTECTED] with the
body of SIGNOFF AOLSERVER in the email message. You can leave the 
Subject: field of your email blank.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Dave Siktberg
 Could this be an OS problem?

In case this provides a clue, the dmesg entries for my two servers that have
the problem are ...

Linux version 2.4.7-10 ([EMAIL PROTECTED]) (gcc version
2.96 2731 (Red Hat Linux 7.1 2.96-98)) #1 Thu Sep 6 17:27:27 EDT 2001

Linux version 2.4.20-18.7bigmem ([EMAIL PROTECTED]) (gcc
version 2.96 2731 (Red Hat Linux 7.3 2.96-113)) #1 SMP Thu May 29
06:59:28 EDT 2003

... and for the server without the problem ...

Linux version 2.4.7-10 ([EMAIL PROTECTED]) (gcc version
2.96 2731 (Red Hat Linux 7.1 2.96-98)) #1 Thu Sep 6 17:27:27 EDT 2001

I have not yet tried backdating the development server and rebooting - will
report results shortly.

Dave


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.


Re: [AOLSERVER] Weird memory leak problem in AOLserver 3.4.2/3.x

2006-05-16 Thread Dave Siktberg
[listserv etiquette question: the topic of this thread no longer matches the
title - should we change it to scheduled proc failures?]

Progress report - Smoking gun found

When I set my development server's system clock via the date command to
2006-05-12 21:28 and restart AOLServer, scheduled procs do not run (the box
runs on Eastern time).  The following Warning message appears in the log and
seems to be the distinctive evidence that scheduled procs will not be run.
(I tried a lot of different times to zero in on this window.)

[12/May/2006:21:28:05][27264.2051][-sched-] Notice: Done running scheduled
proc ug_init_serve_group_pages.
[12/May/2006:21:28:25][27264.1024][-main-] Warning: sched: timeout waiting
for sched idle!
[12/May/2006:21:28:25][27264.1024][-main-] Notice: binder:
listen(192.168.1.72,443) = 14

Whe I set the system clock to 2006-05-12 21:25 or earlier and restart the
server, scheduled procs DO run and the following log entries appear in the
same place as the ones above:

[12/May/2006:21:25:06][27288.2051][-sched-] Notice: Done running scheduled
proc ug_init_serve_group_pages.
[12/May/2006:21:25:06][27288.1024][-main-] Notice: sched: idle
[12/May/2006:21:25:06][27288.1024][-main-] Notice: binder:
listen(192.168.1.72,443) = 14


Further information: my site was built using ACS (precursor ro OpenACS) from
July 2000 or so as a starting point.  We use very little of the ACS
functionality, but the code has remained due to inertia.

Any ideas where to hunt next?

Dave


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] 
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: 
field of your email blank.