Re: Performance limits for a linux box

[Brian Blakeley]

Thanks Allen!  More than enough information to keep me digging for a while.

Will update as/if I make progress.


Brian


Allen Gilliland wrote:

Brian Blakeley wrote:

Hi Allen,

Thanks for your detailed response!

The server has 3 gigabytes of total memory. Yes more would be better but I am presently spending around $300.00 dollars Canadian on a website that has generated about $4.00 in google ad revenue for me. LOL I cannot finance the revolution to any greater amount from my humble pockets! :-)

Understandable. But if your installation happens to be getting enough traffic then more/better hardware may be the only option. 3gb is pretty good, but since you are running many things on one box you definitely could benefit from more. For example, increasing your mysql buffer sizes is likely cutting into your memory quite a bit.

Also, it seems to be that Roller should be able to be configured to sever reasonable traffic with this system.

I have tried to get jvmstats going but have not been able to manage that goal.

If I try to to start the JVM with any more than 570m I get a failure message about the JVM not being able to get the needed memory.

It's tough to say if you need more than 570m without better stats, but that's definitely on the low side if you are really getting good amounts of traffic. You may want to try and get some better profiling on the way your system is using memory.

The errors that I have seen included:

JAVA out of memory
APACHE MaxClient setting reached
MySQL open file limit reached (that was with 275 max connections)

The configuration below ran for about 12-14 hours before crashing with a max client reached error in the Apache error_log

Here is my catalina.out log for since midnight last night:

INFO: Response already commited
Oct 23, 2006 12:10:07 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 12:10:13 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 12:10:16 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 12:10:21 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 1:39:15 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 1:59:21 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited
Oct 23, 2006 2:09:09 PM org.apache.jk.server.JkCoyoteHandler action
INFO: Response already commited

Oct 23, 2006 4:32:45 PM org.apache.tomcat.util.threads.ThreadPool logFull SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status

Oct 23, 2006 4:53:08 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Oct 23, 2006 4:53:08 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2463 ms
Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.0.30
Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled

Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHost getDeployer

INFO: Create Host deployer for direct deployment ( non-jmx )

Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/balancer.xm

l

Oct 23, 2006 4:53:10 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/roller.xml

Oct 23, 2006 4:53:10 PM java.util.jar.Attributes read
WARNING: Duplicate name in Manifest: Class-Path

AbandonedObjectPool is used ([EMAIL PROTECTED])

  LogAbandoned: false
  RemoveAbandoned: true
  RemoveAbandonedTimeout: 300

Oct 23, 2006 4:53:25 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/manager.xml Oct 23, 2006 4:53:25 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/admin.xml Oct 23, 2006 4:53:29 PM org.apache.catalina.core.StandardHostDeployer install INFO: Installing web application at context path from URL file:/usr/java/jakarta-tomcat-5.0.30/webapps/ROOT

Oct 23, 2006 4:53:30 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Oct 23, 2006 4:53:30 PM org.apache.jk.common.ChannelSocket init
INFO: JK2: ajp13 listening on /0.0.0.0:8009
Oct 23, 2006 4:53:30 PM org.apache.jk.server.JkMain start

INFO: Jk running ID=0 time=16/54 config=/usr/java/jakarta-tomcat-5.0.30/conf/jk2.properties

Oct 23, 2006 4:53:30 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 21673 ms
(END)

There is a severe error when the max threads count was reached this afternoon.

Yes, Roller can be a bit liberal with it's db connections usage. Having 200 connections open is troublesome though, since I very highly doubt your are getting enough traffic to warrant that. We get millions of hits a day on BSC and don't need nearly that many connections most of the time.

Also remember that when your db connections are high that mysql is consuming lots of extra memory with per connections buffers, that you have tomcat working harder than usual so it's memory usage is high, and you also have apache httpd running, so that's going to be consuming memory as well. That's a lot to run on a single machine.

In any case, I would try and focus on figuring out why so many connections are being needed.

On real odd thing is that using my javaenv.sh script below everything starts up and runs OK (excepting the lock up ;-) ) , but when I do a ps auxx | grep java there are references to my jdk1.4 directories in the classpath.

[ [EMAIL PROTECTED] logs]# ps auxx | grep java

user 8932 9.8 6.6 417692 210128 ? S< 16:53 8:00 /usr/java/j2sdk1.4.2_06/bin/java -server -Xms50m -Xmx150m -Djava.endorsed.dirs=/usr/java/jakarta-tomcat-5.0.30/common/endorsed -classpath /usr/java/j2sdk1.4.2_06/lib/tools.jar:/ usr/java/jakarta-tomcat-5.0.30/bin/bootstrap.jar:/usr/java/jakarta-tomcat-5.0.30/bin/commons-logging-api.jar -Dcatalina. base=/usr/java/jakarta-tomcat-5.0.30 -Dcatalina.home=/usr/java/jakarta-tomcat-5.0.30 -Djava.io.tmpdir=/usr/java/jakarta-tomcat-5.0.30/temp org.apache.catalina.startup.Bootstrap start

Hey, it is not even starting with the correct JAVA startup command! Why is that?

I don't know the details about the 'ps' command on linux, but it's entirely possible that you don't have your environment configured properly.

Any idea where this might be coming from?

When I do which java I get the right response in my mind. Similarly, when I do java -version I get:

[EMAIL PROTECTED] logs]# which java
/usr/java/jdk1.5.0_09/bin/java

[EMAIL PROTECTED] logs]# java -Xmx50m -version
java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b01)
Java HotSpot(TM) Server VM (build 1.5.0_09-b01, mixed mode)

Just because it says that when *you* type in those commands doesn't mean that's what actually gets run. Do you start tomcat as a different user using a script? Does the script launch other scripts which may be clobbering your environment settings?

I will look into switching to innoDB on mysql.

As to metrics at the time of the lockup of the server this pm these are the numbers that I uncovered.

400 tasks
java        485132kb
apache       25592kb
mysql         60112kb

nothing too surprising here, but only using 60mb for mysql may be a bit slim. tough to say without better metrics on how the server is being used.

results from free:

total     3145728
used       789712
free      2356016

Mysql  show status variables:

max_used_connetions  127
open_files               81
threads_connect   126
threads_created    127
threads_running     126

126 running threads seems like a lot to me. can you check on what the db is doing during that time? are there any mysql errors in the logs? can you check the output of 'show status'? do you have a slow query log configured? that could help you determine if there were queries running on the db which were performing very slowly.

you may also want to check out what kind of comment spam you are getting. we ran into times in the past when our comment spam got way out of control and we would get 100k comment spams in a single day, which can cause some performance issues. if that's the case you may want to try and find the offending ips and ban them somehow.

-- Allen

That's about all I know, Allen.

Thanks again for your time and consideration!


Brian



Allen Gilliland wrote:

Brian,

there is certainly a whole assortment of things that I think you can do to get your site running more efficiently, but before I talk about any of those I've got to ask what kind of analysis you have done on your actual traffic numbers. The more metrics you have about your system the easier it will be to pinpoint your bottlenecks and see if they can be improved. So if you can offer information about traffic, like hits per day, etc. Even breaking it down by hits to the page servlet, feed servlets, static resources, etc, will be helpful. Same goes for things like caching efficiency numbers, metrics on db usage, what kind of errors are there in your logs, and on. We need info about how the system is running now to know what wrong with it.

Here's a few simple things off the top of my head though ...

1. You should be using InnoDB as your mysql storage engine. It looks like you are using myisam tables and there are various problems with that. Check out InnoDB.

2. Only starting your jvm with min/max heap sizes of 256m/570m is *way* too low if you are really getting a lot of traffic. When you say you have 3gb what does that mean? 3gb of memory for everything you do in your virtual server? or 3gb for just that one jvm? I would definitely suggest you bump that up as high as you can.

3. Upgrade to new versions of Roller, things have only gotten better since 2.1 =)

4. You may need to tweak your cache sizes, but that will be dependent on #2 since you don't want to start running out of memory. If you add the "-verbose:gc" jvm option then you should start seeing info in your logs about memory usage so that you can have something to use to monitor that. Since you are using jdk 5 it's even better if you can use jconsole and get metrics on your jmv from that. Anways, back to caching ...

You should probably set your 'defaultFactory' to not use the Expiring cache. That means that the app will automatically expire content that is older than a certain time and that's not very efficient. We have used the non-Expiring cache on BSC for a long time now and it works fine, although I can't exactly recall what the cache code looked like in 2.1.

Remember that cache sizes are for the entire site, so in general you want those as large as possible. I would suggest having the feed cache be *at least* as big as your # of weblogs, but 2x # of weblogs would be better. The page cache takes up a lot more room, so you probably want at least 4x the # of weblogs for that.

5. I'm not sure that you should need 1024 open file descriptors just for Roller, but if your db is on the same box then that wouldn't surprise me in the slightest. Handling a decent amount of web traffic requires descriptors, so I would jack that number up to something high, like 16K.

Well, that's a few things to try.  g/l.

-- Allen



Brian Blakeley wrote:

I have combined the various configurations that I have set in this listing to provide more background for anyone/everyone with better knowledge than myself.

Also, Richard, I missed a couple of mysql configurations that I set in my.cnf

Thanks for any suggestions that you may be able to provide to me.


Che Blogs Configurations:

+++++++++++++++++++++++++++++++++++++++++++
JAVA Configuration:

# define the various homes homes

BASEDIR=/usr/java/jakarta-tomcat-5.0.30
ANT_HOME=/usr/java/jakarta-ant-1.5.1
JAVA_HOME=/usr/java/jdk1.5.0_09
JAVA_OPTS='-server -Xms256m -Xmx570m'
CATALINA_HOME=/usr/java/jakarta-tomcat-5.0.30
TOMCAT_HOME=/usr/java/jakarta-tomcat-5.0.30/webapps
ROLLER_HOME=/usr/java/jakarta-tomcat-5.0.30/webapps/roller
JVMSTAT_HOME=/usr/java/jvmstat

# add java to path

PATH=$PATH:$BASEDIR
PATH=$PATH:$JAVA_HOME/bin
PATH=$PATH:$CATALINA_HOME/bin
PATH=$PATH:$ANT_HOME/bin
PATH=$PATH:$JVMSTAT_HOME/bin


# exports

export BASEDIR
export JAVA_OPTS
export JAVA_HOME CATALINA_HOME ANT_HOME
export TOMCAT_HOME ROLLER_HOME JVMSTAT_HOME
javaenv.sh (END)


+++++++++++++++++++++++++++++++++++++++++++
Apache configuration:

#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow

# during a persistent connection. Set to 0 to allow an unlimited amount.

# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#

# KeepAliveTimeout: Number of seconds to wait for the next request from the

# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start

# MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare

# MaxClients: maximum number of server processes allowed to start

# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>
StartServers       75
MinSpareServers    75
MaxSpareServers   150
MaxClients        250
MaxRequestsPerChild  50000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule worker.c>
StartServers        25
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     50
ThreadsPerChild     25
MaxRequestsPerChild 10000
</IfModule>

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
:

+++++++++++++++++++++++++++++++++++++++++++
MySQL set-up:


[mysqld]
#default-character-set=utf8
set-variable = max_connections=150
set-variable = max_user_connections=125
set-variable = wait_timeout=60
set-variable = thread_cache_size=40

my.cnf (END)


+++++++++++++++++++++++++++++++++++++++++++
Roller properties:

#----------------------------------
# Cache settings.
# Remember ... times are in seconds
# Default settings suitable for 100 user system

# Cache properties all follow the given format ...
#     cache.<cache_id>.<prop>=<value>

# we then pass all <prop>=<value> pairs into the cache manager when the cache # is being constructed. this makes it easy to add cache properties that can

# be used by the specified CacheFactory you are using.
#

# NOTE: it is expected that property validation happens in the CacheFactory

#-----------------------------------

# The default cache implementation we want to use

cache.defaultFactory=org.roller.presentation.cache.ExpiringLRUCacheFactoryImpl

cache.customHandlers=

# Main/Planet page cache (this is low on purpose)
cache.mainpage.size=10
cache.mainpage.timeout=1800
# set "true" to NOT cache the custom pages for users who are logged in
cache.mainpage.excludeOwnerEditPages=true

# Weblog page cache (all the weblog content)
cache.weblogpage.size=400
cache.weblogpage.timeout=3600
# set "true" to NOT cache the custom pages for users who are logged in
cache.weblogpage.excludeOwnerEditPages=true

# Weblog page last-modified-date cache
# you want this fairly high, like weblogs * 10, with long timeouts
cache.ifmodified.weblogpage.size=1000
cache.ifmodified.weblogpage.timeout=14400

# Feed cache (xml feeds like rss, atom, etc)
cache.feed.size=200
cache.feed.timeout=3600

# Feed last-modified-date cache
# you want a reasonable size, like weblogs * 2, with long timeouts
cache.ifmodified.feed.size=200
cache.ifmodified.feed.timeout=14400

# Planet cache (planet page and rss feed)
cache.planet.size=10
cache.planet.timeout=1800
# set "true" to NOT cache the custom pages for users who are logged in
cache.planet.excludeOwnerEditPages=false

#----------------------------------
# Secure login configs

+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Roller Database connector:

<Context path="/roller" docBase="roller" debug="99" workDir="work/Catalina/localhost/roller">

<Realm className="org.apache.catalina.realm.JDBCRealm"
   driverName="com.mysql.jdbc.Driver"
   connectionURL=
       "jdbc:mysql://localhost:3306/roller?autoReconnect=true&amp;

useUnicode=true&amp;characterEncoding=utf-8&amp;mysqlEncoding=utf8"

   connectionName="******"
   connectionPassword="*******"
   userTable="rolleruser"
   userNameCol="username"
   userCredCol="passphrase"
   userRoleTable="userrole"
   roleNameCol="rolename" debug="99" />

<Resource name="jdbc/rollerdb" auth="Container" type="javax.sql.DataSource" />

<ResourceParams name="jdbc/rollerdb">
   <parameter>
      <name>factory</name>
      <value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
   </parameter>
   <parameter>
      <name>driverClassName</name>
      <value>com.mysql.jdbc.Driver</value>
   </parameter>
   <parameter>
     <name>url</name>
     <value>jdbc:mysql://localhost:3306/roller?autoReconnect=true&amp;

useUnicode=true&amp;characterEncoding=utf-8&amp;mysqlEncoding=utf8

     </value>
   </parameter>
   <parameter><name>username</name><value>*******</value></parameter>
   <parameter><name>password</name><value>*******</value></parameter>
   <parameter><name>maxActive</name><value>250</value></parameter>
   <parameter><name>maxIdle</name><value>150</value></parameter>

<parameter><name>removeAbandoned</name><value>true</value></parameter>

   <parameter><name>maxWait</name><value>3000</value></parameter>
</ResourceParams>

<!--
To enable email notification of comments: uncomment the resouce below,
set your mailhost, and make sure you have mail.jar and activation.jar
in <tomcat>/common/lib.
-->
<!--

<Resource name="mail/Session" auth="Container" type="javax.mail.Session"/>

   <ResourceParams name="mail/Session">
       <parameter>
           <name>mail.smtp.host</name>
           <value>mailhost.example.com</value>
       </parameter>
   </ResourceParams>
-->
roller.xml


:



Brian Blakeley wrote:

Hi Richard,

Thanks Richard for your reply.

I only set my.cnf with the following:

max_connections= 150
max_user_connections=125

I had previously set max/user to 275/250 but I hit the max open files limit of 1024 on my server and it lock up a couple of times in the same day. So I back the limits off to 150/125 as above.

Your example had 500 max connections I can not see how I could run some many with the 1024 open files limit on my server.

Brian

Richard Jones wrote:

Hi Brian,

I had a few probs a while back with Roller using-up all the MySQL

connections; changing the MySQL configuration solved this for me. I'm

not running as many blogs as you, but I've copied my conf below.

Cheers,

Richard


[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-innodb
max_connections = 500
key_buffer = 16M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 64
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

[mysqldump]
quick
max_allowed_packet = 16M




On 22/10/06, webmaster <[EMAIL PROTECTED]> wrote:

Hi Everyone,

Che Blogs has been cracking up these past couple of weeks under the

load. What I am wondering is if anyone has any sense of the maximum load that I should be able to achieve with Roller running on a hosted system

under Linux?

My setup is:

Fedora Core 2
JAVA Hotspot Server 1.5.0_09-b01
Apache 2.0.51
MySQL 4.1.21
mod_jk

Roller 2.1 incubating

The Server has 3.0G of Ram installed and 800 processes/threads are

available to it. It is a "virtual dedicated server" in the language of

my hosting provider.

After much hair pulling I set the my configurations to attempt to allow

 250 client connections for MySQL, Apache and Roller.

The jvm is booted with the following directive:
-server -Xms256m -Xmx570m

This setup seemed to be working ok, but it has problems after a bit of time.

Today it has crashed twice because of hitting the 1024 limit on open files.

Right now it is running but I cannot access Che Blogs but I can access

two other little planetplanet sites that are on the same server.

One other problem is that I cannot start the jvm with any more than

Xmx570m even thought I have 3G's of ram. Once the JVM for Roller is running I cannot start another JVM with anything above a trivial amount

on memory allocated to it.

I am sorry that is this so long and I realize that this is a really

opened question, but I have been reading, googling, testing and tweaking for over a week now and I am at my wits end as to what more I can do.

By the way, Che Blogs has probably less than 500 blogs on it total. I am trying to confirm that number but the MySQL response is unbelievably slow. I think that is a clue ;-) Lately, it is being hit by all the

search engines and several RSS basis services very heavily.

Any tuning suggestions for MySQL?

So, I will post this and please let me know if I am missing something

simple so that I can get on with the other areas of my life!

Thanks for your kind consideration of this plea.


Brian