On Fri, Feb 03, 2006 at 11:53:01AM -0500, Nima wrote:

> What basically happens is that once I have more than 150 users
> logged in the response time for a page takes a minute or more which
> is very frustrating for users.  Today we received an email saying
> that the system is simply "shit". I don't know what to do next. All
> I know is that from semester to semester the load is getting higher
> but the frustration as well.

Nima, rrom your description this did it NOT happen recently all at
once, it has been gradually getting worse as you add more users,

It sounds like you're fairly lost and that you have an urgent
performance problem affecting your users.  I don't know just what
dotLRN installation this is, but if you really have 150 "concurrent"
users (whatever that means precisely in this case), then it's probably
one of the big dotLRN installations at a large university somewhere
(ah, from the config file below, "uni-mannheim.de").

I hope that means you also have some sort of support contract with one
of the OpenACS / dotLRN gurus.  If it's not something you can
immediately (like, today) identify and fix, my advice is get them
involved ASAP.  It might be just a simple misconfiguration somewhere,
or it might be something deeper and trickier to fix.  Either way,
having all your users actively angry at you makes it plenty urgent
enough to call in the big guns...

> We have three linux boxes. One for an aolserver with database connection, 
> one for a static aolserver and one for the database.
> The database box never goes above 5-10%. The static server is also not 

That's only cpu load.  You also want to check it's I/O activity.
(Solaris top also shows "I/O wait" percentages but Linux unfortunately
does not.)  On older Linux boxes "iostat 5" was the way to do that.
Newer Linux systems may have different/better ways to do that.

> very busy but the dynamic server can go upt to 99% and a load of 10 and 
> more.

Well, that VERY strongly suggests that the rate limiter is simply
executing all that Tcl code in your AOLserver.  If so, add more
AOLserver boxes, and set up Pound or the like as a front-end server to
split the load between them.  And/or upgrade to a much faster server.

In addition, try to find out what pages are eating up most of the
processing time, and speed them up.  A lot of that processing may be
redundant and/or innefficient.  Some judicious cacheing and/or code
tuning could make a huge difference.

Oh, and a silly question:  What version of Tcl are you using, and did
you compile it with optimization?  You definitely want to be using the
latest Tcl 8.4.x version compiled with either "-g -O2" (my preferecne)
or "-O2".  I don't know how much slower Tcl is if you leave compiler
optimization turned off, but it's probably enough to be very noticable
in your case.  (Make sure AOLserver was also compiled with optization
of course; it re-uses the Tcl build flags.)

Finally, this is more of a research project, but your site is large
and busy enough to benefit from figuring out just what the current
status is of this patch:

  Cache compiled Tcl page bytecode

> Currently:
> 41.1  0.0 7448 27147 unima2    25   0 1770m 1.6g S   
> 0:36.66 /opt/aolserver4/bin/nsd -u unima2 -t /www/unima2/etc/config.tcl 
> with 44 users logged into the system.

That says that your AOLserver is using 1.7 GB total memory, almost all
of it resident.  Which is huge for most people, but probably quite
reasonable for you, since you have 4 GB in that box.  That at least
probably means that the box isn't thrashing between RAM and disk,

> dotlrn (dynamic server)
> AOLServer 4.0.10 (connected to the database)
> Pound 1.8.2 (as reverse proxy for ssl and load balancing)
> Apache 2.0.53 (only redirect from 80 to 443 where pound is)

Oh, you're already using Pound as the front-end.  So, shouldn't it be
easy to stick in additional AOLservers behind it for dynamic content?

The CATCH is, is your site and all its code, both stock and custom,
already set up to work nicely with multiple AOLservers?  Or does it
rashly ASSUME only 1 AOLserver process in some places, such that you
are going to see bugs or inconsistencies when using multiple
AOLservers?  I dunno.  For that, I definitely recommend talking to the
other folks running multiple AOLservers with OpenACS and dotLRN.

It sounds like you're running Pound on the same box as AOLserver.
You'll definitely need to change that in order to add dynamic content
servers.  (I don't understand why you're using Apache to redirect
client browsers from port 80 to 443 either, that seems odd.)

> SuSE 9.2
> Linux Linux version 2.6.8-24.18-smp (gcc version 3.3.4 (pre 3.3.5 
> 20040809)) #1 SMP Fri Aug 19 11:56:28 UTC 2005
> 4 CPU Intel(R) Xeon(TM) CPU 3.06GHz , L2 cache: 512K
> 4 GByte RAM - Memory: 4070968k/4111296k available (2339k kernel code, 
> 39528k reserved, 824k data, 252k init, 3193792k highmem)
> 2 Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet Cards

Hm, do you REALLY have 4 Xeon CPUs in that box, or is that the Intel
hyper-threading feature turned on?  I suspect you have 2 Xeon
single-core sockets, with hyper-threading turned on.  Back with Linux
2.4.x the ule of was to always turn hyper-threading OFF, as Linux
didn't know how to use it properly and hyper-threading would slow
things down, not speed them up.  I don't know whether that has changed
with the newer 2.6.x kernels.

I was going to suggest that if your single AOLserver box is a few
years old, then immediately replacing it with a (much faster) brand
spanking new one may be the easiest and most cost effective way to
alleviate the problem.  Then you can take more time to get multiple
boxes set up, make sure that your code works correctly in that
configuration, etc.  However, from the specs above you're already
using a fairly high end machine, so that might not make sense.
Keeping your existing box and adding more is probably the way to go.

For comparison though, for about $5900 US, right now you could order a
PowerEdge 1850 1U box from Dell with 2 sockets each with a dual-core
2.8 GHz Xeon (2x2 MB L2 cache), 8 GB of RAM (expandable to 16 GB),
RAID-1 w/ 2 15k rpm SCSI drives.  That should give you roughly 2x the
performance of your current box.  Or the same Dell box with 2
single-core 3.8 GHz Xeons for about $5600.

I wouldn't necessarily pick either that machine or Dell, but that's a
useful price point for comparison.  The Dell box is mildly gold-plated
anyway, since this is effecively a compute box, it is running only
AOLserver no RDBMS or anything else disk intensive, I would probably
go with hardware RAID-1 but with SATA or even plain old IDE drives, no
need to pay extra for SCSI.  If I had many of these identical boxes
and was set up to easily automatically install them, I might even skip
the RAID card.

Andrew Piskorski <[EMAIL PROTECTED]>

AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Reply via email to