Brian Blakeley wrote:
Hi Allen, Thanks for your detailed response!The server has 3 gigabytes of total memory. Yes more would be better but I am presently spending around $300.00 dollars Canadian on a website that has generated about $4.00 in google ad revenue for me. LOL I cannot finance the revolution to any greater amount from my humble pockets! :-)
Understandable. But if your installation happens to be getting enough traffic then more/better hardware may be the only option. 3gb is pretty good, but since you are running many things on one box you definitely could benefit from more. For example, increasing your mysql buffer sizes is likely cutting into your memory quite a bit.
Also, it seems to be that Roller should be able to be configured to sever reasonable traffic with this system.I have tried to get jvmstats going but have not been able to manage that goal.If I try to to start the JVM with any more than 570m I get a failure message about the JVM not being able to get the needed memory.
It's tough to say if you need more than 570m without better stats, but that's definitely on the low side if you are really getting good amounts of traffic. You may want to try and get some better profiling on the way your system is using memory.
The errors that I have seen included: JAVA out of memory APACHE MaxClient setting reached MySQL open file limit reached (that was with 275 max connections)The configuration below ran for about 12-14 hours before crashing with a max client reached error in the Apache error_logHere is my catalina.out log for since midnight last night: INFO: Response already commited Oct 23, 2006 12:10:07 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 12:10:13 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 12:10:16 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 12:10:21 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 1:39:15 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 1:59:21 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 2:09:09 PM org.apache.jk.server.JkCoyoteHandler action INFO: Response already commited Oct 23, 2006 4:32:45 PM org.apache.tomcat.util.threads.ThreadPool logFullSEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet statusOct 23, 2006 4:53:08 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-8080 Oct 23, 2006 4:53:08 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2463 ms Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/5.0.30 Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHost start INFO: XML validation disabled Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHost getDeployer INFO: Create Host deployer for direct deployment ( non-jmx )Oct 23, 2006 4:53:09 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/balancer.xmlOct 23, 2006 4:53:10 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/roller.xmlOct 23, 2006 4:53:10 PM java.util.jar.Attributes read WARNING: Duplicate name in Manifest: Class-PathAbandonedObjectPool is used ([EMAIL PROTECTED])LogAbandoned: false RemoveAbandoned: true RemoveAbandonedTimeout: 300Oct 23, 2006 4:53:25 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/manager.xml Oct 23, 2006 4:53:25 PM org.apache.catalina.core.StandardHostDeployer install INFO: Processing Context configuration file URL file:/usr/java/jakarta-tomcat-5.0.30/conf/Catalina/localhost/admin.xml Oct 23, 2006 4:53:29 PM org.apache.catalina.core.StandardHostDeployer install INFO: Installing web application at context path from URL file:/usr/java/jakarta-tomcat-5.0.30/webapps/ROOTOct 23, 2006 4:53:30 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8080 Oct 23, 2006 4:53:30 PM org.apache.jk.common.ChannelSocket init INFO: JK2: ajp13 listening on /0.0.0.0:8009 Oct 23, 2006 4:53:30 PM org.apache.jk.server.JkMain startINFO: Jk running ID=0 time=16/54 config=/usr/java/jakarta-tomcat-5.0.30/conf/jk2.propertiesOct 23, 2006 4:53:30 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 21673 ms (END)There is a severe error when the max threads count was reached this afternoon.
Yes, Roller can be a bit liberal with it's db connections usage. Having 200 connections open is troublesome though, since I very highly doubt your are getting enough traffic to warrant that. We get millions of hits a day on BSC and don't need nearly that many connections most of the time.
Also remember that when your db connections are high that mysql is consuming lots of extra memory with per connections buffers, that you have tomcat working harder than usual so it's memory usage is high, and you also have apache httpd running, so that's going to be consuming memory as well. That's a lot to run on a single machine.
In any case, I would try and focus on figuring out why so many connections are being needed.
On real odd thing is that using my javaenv.sh script below everything starts up and runs OK (excepting the lock up ;-) ) , but when I do a ps auxx | grep java there are references to my jdk1.4 directories in the classpath.[ [EMAIL PROTECTED] logs]# ps auxx | grep javauser 8932 9.8 6.6 417692 210128 ? S< 16:53 8:00 /usr/java/j2sdk1.4.2_06/bin/java -server -Xms50m -Xmx150m -Djava.endorsed.dirs=/usr/java/jakarta-tomcat-5.0.30/common/endorsed -classpath /usr/java/j2sdk1.4.2_06/lib/tools.jar:/ usr/java/jakarta-tomcat-5.0.30/bin/bootstrap.jar:/usr/java/jakarta-tomcat-5.0.30/bin/commons-logging-api.jar -Dcatalina. base=/usr/java/jakarta-tomcat-5.0.30 -Dcatalina.home=/usr/java/jakarta-tomcat-5.0.30 -Djava.io.tmpdir=/usr/java/jakarta-tomcat-5.0.30/temp org.apache.catalina.startup.Bootstrap startHey, it is not even starting with the correct JAVA startup command! Why is that?
I don't know the details about the 'ps' command on linux, but it's entirely possible that you don't have your environment configured properly.
Any idea where this might be coming from?When I do which java I get the right response in my mind. Similarly, when I do java -version I get:[EMAIL PROTECTED] logs]# which java /usr/java/jdk1.5.0_09/bin/java [EMAIL PROTECTED] logs]# java -Xmx50m -version java version "1.5.0_09" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b01) Java HotSpot(TM) Server VM (build 1.5.0_09-b01, mixed mode)
Just because it says that when *you* type in those commands doesn't mean that's what actually gets run. Do you start tomcat as a different user using a script? Does the script launch other scripts which may be clobbering your environment settings?
I will look into switching to innoDB on mysql.As to metrics at the time of the lockup of the server this pm these are the numbers that I uncovered.400 tasks java 485132kb apache 25592kb mysql 60112kb
nothing too surprising here, but only using 60mb for mysql may be a bit slim. tough to say without better metrics on how the server is being used.
results from free: total 3145728 used 789712 free 2356016 Mysql show status variables: max_used_connetions 127 open_files 81 threads_connect 126 threads_created 127 threads_running 126
126 running threads seems like a lot to me. can you check on what the db is doing during that time? are there any mysql errors in the logs? can you check the output of 'show status'? do you have a slow query log configured? that could help you determine if there were queries running on the db which were performing very slowly.
you may also want to check out what kind of comment spam you are getting. we ran into times in the past when our comment spam got way out of control and we would get 100k comment spams in a single day, which can cause some performance issues. if that's the case you may want to try and find the offending ips and ban them somehow.
-- Allen
That's about all I know, Allen. Thanks again for your time and consideration! Brian Allen Gilliland wrote:Brian,there is certainly a whole assortment of things that I think you can do to get your site running more efficiently, but before I talk about any of those I've got to ask what kind of analysis you have done on your actual traffic numbers. The more metrics you have about your system the easier it will be to pinpoint your bottlenecks and see if they can be improved. So if you can offer information about traffic, like hits per day, etc. Even breaking it down by hits to the page servlet, feed servlets, static resources, etc, will be helpful. Same goes for things like caching efficiency numbers, metrics on db usage, what kind of errors are there in your logs, and on. We need info about how the system is running now to know what wrong with it.Here's a few simple things off the top of my head though ...1. You should be using InnoDB as your mysql storage engine. It looks like you are using myisam tables and there are various problems with that. Check out InnoDB.2. Only starting your jvm with min/max heap sizes of 256m/570m is *way* too low if you are really getting a lot of traffic. When you say you have 3gb what does that mean? 3gb of memory for everything you do in your virtual server? or 3gb for just that one jvm? I would definitely suggest you bump that up as high as you can.3. Upgrade to new versions of Roller, things have only gotten better since 2.1 =)4. You may need to tweak your cache sizes, but that will be dependent on #2 since you don't want to start running out of memory. If you add the "-verbose:gc" jvm option then you should start seeing info in your logs about memory usage so that you can have something to use to monitor that. Since you are using jdk 5 it's even better if you can use jconsole and get metrics on your jmv from that. Anways, back to caching ...You should probably set your 'defaultFactory' to not use the Expiring cache. That means that the app will automatically expire content that is older than a certain time and that's not very efficient. We have used the non-Expiring cache on BSC for a long time now and it works fine, although I can't exactly recall what the cache code looked like in 2.1.Remember that cache sizes are for the entire site, so in general you want those as large as possible. I would suggest having the feed cache be *at least* as big as your # of weblogs, but 2x # of weblogs would be better. The page cache takes up a lot more room, so you probably want at least 4x the # of weblogs for that.5. I'm not sure that you should need 1024 open file descriptors just for Roller, but if your db is on the same box then that wouldn't surprise me in the slightest. Handling a decent amount of web traffic requires descriptors, so I would jack that number up to something high, like 16K.Well, that's a few things to try. g/l. -- Allen Brian Blakeley wrote:I have combined the various configurations that I have set in this listing to provide more background for anyone/everyone with better knowledge than myself.Also, Richard, I missed a couple of mysql configurations that I set in my.cnfThanks for any suggestions that you may be able to provide to me. Che Blogs Configurations: +++++++++++++++++++++++++++++++++++++++++++ JAVA Configuration: # define the various homes homes BASEDIR=/usr/java/jakarta-tomcat-5.0.30 ANT_HOME=/usr/java/jakarta-ant-1.5.1 JAVA_HOME=/usr/java/jdk1.5.0_09 JAVA_OPTS='-server -Xms256m -Xmx570m' CATALINA_HOME=/usr/java/jakarta-tomcat-5.0.30 TOMCAT_HOME=/usr/java/jakarta-tomcat-5.0.30/webapps ROLLER_HOME=/usr/java/jakarta-tomcat-5.0.30/webapps/roller JVMSTAT_HOME=/usr/java/jvmstat # add java to path PATH=$PATH:$BASEDIR PATH=$PATH:$JAVA_HOME/bin PATH=$PATH:$CATALINA_HOME/bin PATH=$PATH:$ANT_HOME/bin PATH=$PATH:$JVMSTAT_HOME/bin # exports export BASEDIR export JAVA_OPTS export JAVA_HOME CATALINA_HOME ANT_HOME export TOMCAT_HOME ROLLER_HOME JVMSTAT_HOME javaenv.sh (END) +++++++++++++++++++++++++++++++++++++++++++ Apache configuration: # KeepAlive On # # MaxKeepAliveRequests: The maximum number of requests to allow # during a persistent connection. Set to 0 to allow an unlimited amount. # We recommend you leave this number high, for maximum performance. # MaxKeepAliveRequests 100 ## KeepAliveTimeout: Number of seconds to wait for the next request from the# same client on the same connection. # KeepAliveTimeout 15 ## ## Server-Pool Size Regulation (MPM specific) ## # prefork MPM # StartServers: number of server processes to start# MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare# MaxClients: maximum number of server processes allowed to start# MaxRequestsPerChild: maximum number of requests a server process serves<IfModule prefork.c> StartServers 75 MinSpareServers 75 MaxSpareServers 150 MaxClients 250 MaxRequestsPerChild 50000 </IfModule> # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare# ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves<IfModule worker.c> StartServers 25 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 50 ThreadsPerChild 25 MaxRequestsPerChild 10000 </IfModule> # # Listen: Allows you to bind Apache to specific IP addresses and/or # ports, in addition to the default. See also the <VirtualHost> # directive. : +++++++++++++++++++++++++++++++++++++++++++ MySQL set-up: [mysqld] #default-character-set=utf8 set-variable = max_connections=150 set-variable = max_user_connections=125 set-variable = wait_timeout=60 set-variable = thread_cache_size=40 my.cnf (END) +++++++++++++++++++++++++++++++++++++++++++ Roller properties: #---------------------------------- # Cache settings. # Remember ... times are in seconds # Default settings suitable for 100 user system # Cache properties all follow the given format ... # cache.<cache_id>.<prop>=<value># we then pass all <prop>=<value> pairs into the cache manager when the cache # is being constructed. this makes it easy to add cache properties that can# be used by the specified CacheFactory you are using. ## NOTE: it is expected that property validation happens in the CacheFactory#----------------------------------- # The default cache implementation we want to usecache.defaultFactory=org.roller.presentation.cache.ExpiringLRUCacheFactoryImplcache.customHandlers= # Main/Planet page cache (this is low on purpose) cache.mainpage.size=10 cache.mainpage.timeout=1800 # set "true" to NOT cache the custom pages for users who are logged in cache.mainpage.excludeOwnerEditPages=true # Weblog page cache (all the weblog content) cache.weblogpage.size=400 cache.weblogpage.timeout=3600 # set "true" to NOT cache the custom pages for users who are logged in cache.weblogpage.excludeOwnerEditPages=true # Weblog page last-modified-date cache # you want this fairly high, like weblogs * 10, with long timeouts cache.ifmodified.weblogpage.size=1000 cache.ifmodified.weblogpage.timeout=14400 # Feed cache (xml feeds like rss, atom, etc) cache.feed.size=200 cache.feed.timeout=3600 # Feed last-modified-date cache # you want a reasonable size, like weblogs * 2, with long timeouts cache.ifmodified.feed.size=200 cache.ifmodified.feed.timeout=14400 # Planet cache (planet page and rss feed) cache.planet.size=10 cache.planet.timeout=1800 # set "true" to NOT cache the custom pages for users who are logged in cache.planet.excludeOwnerEditPages=false #---------------------------------- # Secure login configs +++++++++++++++++++++++++++++++++++++++++++++++++++++++ Roller Database connector:<Context path="/roller" docBase="roller" debug="99" workDir="work/Catalina/localhost/roller"><Realm className="org.apache.catalina.realm.JDBCRealm" driverName="com.mysql.jdbc.Driver" connectionURL= "jdbc:mysql://localhost:3306/roller?autoReconnect=true&useUnicode=true&characterEncoding=utf-8&mysqlEncoding=utf8"connectionName="******" connectionPassword="*******" userTable="rolleruser" userNameCol="username" userCredCol="passphrase" userRoleTable="userrole" roleNameCol="rolename" debug="99" /><Resource name="jdbc/rollerdb" auth="Container" type="javax.sql.DataSource" /><ResourceParams name="jdbc/rollerdb"> <parameter> <name>factory</name> <value>org.apache.commons.dbcp.BasicDataSourceFactory</value> </parameter> <parameter> <name>driverClassName</name> <value>com.mysql.jdbc.Driver</value> </parameter> <parameter> <name>url</name> <value>jdbc:mysql://localhost:3306/roller?autoReconnect=true&useUnicode=true&characterEncoding=utf-8&mysqlEncoding=utf8</value> </parameter> <parameter><name>username</name><value>*******</value></parameter> <parameter><name>password</name><value>*******</value></parameter> <parameter><name>maxActive</name><value>250</value></parameter> <parameter><name>maxIdle</name><value>150</value></parameter><parameter><name>removeAbandoned</name><value>true</value></parameter><parameter><name>maxWait</name><value>3000</value></parameter> </ResourceParams> <!-- To enable email notification of comments: uncomment the resouce below, set your mailhost, and make sure you have mail.jar and activation.jar in <tomcat>/common/lib. --> <!--<Resource name="mail/Session" auth="Container" type="javax.mail.Session"/><ResourceParams name="mail/Session"> <parameter> <name>mail.smtp.host</name> <value>mailhost.example.com</value> </parameter> </ResourceParams> --> roller.xml : Brian Blakeley wrote:Hi Richard, Thanks Richard for your reply. I only set my.cnf with the following: max_connections= 150 max_user_connections=125I had previously set max/user to 275/250 but I hit the max open files limit of 1024 on my server and it lock up a couple of times in the same day. So I back the limits off to 150/125 as above.Your example had 500 max connections I can not see how I could run some many with the 1024 open files limit on my server.Brian Richard Jones wrote:Hi Brian, I had a few probs a while back with Roller using-up all the MySQL connections; changing the MySQL configuration solved this for me. I'm not running as many blogs as you, but I've copied my conf below. Cheers, Richard [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock skip-innodb max_connections = 500 key_buffer = 16M myisam_sort_buffer_size = 64M join_buffer_size = 1M read_buffer_size = 1M sort_buffer_size = 2M table_cache = 1024 thread_cache_size = 64 wait_timeout = 1800 connect_timeout = 10 max_allowed_packet = 16M max_connect_errors = 10 query_cache_limit = 1M query_cache_size = 32M query_cache_type = 1 [mysql.server] user=mysql basedir=/var/lib [safe_mysqld] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [myisamchk] key_buffer = 64M sort_buffer = 64M read_buffer = 16M write_buffer = 16M [mysqldump] quick max_allowed_packet = 16M On 22/10/06, webmaster <[EMAIL PROTECTED]> wrote:Hi Everyone, Che Blogs has been cracking up these past couple of weeks under theload. What I am wondering is if anyone has any sense of the maximum load that I should be able to achieve with Roller running on a hosted systemunder Linux? My setup is: Fedora Core 2 JAVA Hotspot Server 1.5.0_09-b01 Apache 2.0.51 MySQL 4.1.21 mod_jk Roller 2.1 incubating The Server has 3.0G of Ram installed and 800 processes/threads areavailable to it. It is a "virtual dedicated server" in the language ofmy hosting provider.After much hair pulling I set the my configurations to attempt to allow250 client connections for MySQL, Apache and Roller. The jvm is booted with the following directive: -server -Xms256m -Xmx570mThis setup seemed to be working ok, but it has problems after a bit of time.Today it has crashed twice because of hitting the 1024 limit on open files.Right now it is running but I cannot access Che Blogs but I can accesstwo other little planetplanet sites that are on the same server. One other problem is that I cannot start the jvm with any more than Xmx570m even thought I have 3G's of ram. Once the JVM for Roller isrunning I cannot start another JVM with anything above a trivial amounton memory allocated to it. I am sorry that is this so long and I realize that this is a reallyopened question, but I have been reading, googling, testing and tweakingfor over a week now and I am at my wits end as to what more I can do.By the way, Che Blogs has probably less than 500 blogs on it total. I am trying to confirm that number but the MySQL response is unbelievablyslow. I think that is a clue ;-) Lately, it is being hit by all the search engines and several RSS basis services very heavily. Any tuning suggestions for MySQL? So, I will post this and please let me know if I am missing something simple so that I can get on with the other areas of my life! Thanks for your kind consideration of this plea. Brian
