Re: [Radiant] Radiant 0.9 performance

[Anna Billstrom]

This is the future publishing issue- sadly the '/' check on status  
will not work- apologies Marshall


Sent from my iPhone

On Jun 26, 2010, at 10:32 AM, Marshal Linfoot   
wrote:



Another thing that seems odd to me... from the production log:

Processing SiteController#show_page (for 99.232.46.125 at 2010-06-26  
13:26:34) [GET]
  Parameters: {"action"=>"show_page", "url"=>"/",  
"controller"=>"site"}
Completed in 1923ms (DB: 259) | 200 [http:// 
www.octopusgardenyoga.com/]


and almost immediately after, a request for the same page from  
another location:


Processing SiteController#show_page (for 208.76.245.135 at  
2010-06-26 13:26:50) [GET]
  Parameters: {"action"=>"show_page", "url"=>"/",  
"controller"=>"site"}

Completed in 1837ms (DB: 280) | 200 [http://octopusgardenyoga.com/]

Shouldn't the second request have been significantly faster? Yet the  
DB access is higher and the overall time about the same. Like  
nothing is being cached!!!


--
marshal

Re: [Radiant] Radiant 0.9 performance

[Marshal Linfoot]

Another thing that seems odd to me... from the production log:

Processing SiteController#show_page (for 99.232.46.125 at 2010-06-26
13:26:34) [GET]
  Parameters: {"action"=>"show_page", "url"=>"/", "controller"=>"site"}
Completed in 1923ms (DB: 259) | 200 [http://www.octopusgardenyoga.com/]

and almost immediately after, a request for the same page from another
location:

Processing SiteController#show_page (for 208.76.245.135 at 2010-06-26
13:26:50) [GET]
  Parameters: {"action"=>"show_page", "url"=>"/", "controller"=>"site"}
Completed in 1837ms (DB: 280) | 200 [http://octopusgardenyoga.com/]

Shouldn't the second request have been significantly faster? Yet the DB
access is higher and the overall time about the same. Like nothing is being
cached!!!

-- 
marshal

Re: [Radiant] Radiant 0.9 performance

[Marshal Linfoot]

John, thanks again for taking the time to think about this. I'm starting to
think it's a slicehost problem.

Over the past couple of days I've tried:

 1. increased memory from 256 to 512MB on the slice
 - lots of free memory, minor performance boost
  2. ran sqlite3 in production
  - no noticeable difference in DB access times. not so surprising,
lightly used system
  3. tuned mysql
  - followed settings from an example config for system with 512MB,
mysql + webserver
  - some improvement
  4. ran nginx+passenger with 2 worker processes
  -  access times were about the same as litespeed
   5. verified that all html was valid
   6. searched system logs for any signs of problems
   - no indication of issues
   7. updated all gems to latest release (backing off rack 1.2.2 to 1.1.0)
   - this included the latest radiant 0.9.0
   - mysql gem is at 2.8.1
8. rake radiant:unfreeze -- running from the gem instead of edge
9. made sure all extensions were up to date (git pull'ed them all)
- database migrations, extension migrations all okay

None of this has made any difference in the total access times -- still very
high.

(sample mysql+litespeed)
Completed in 442ms (DB: 5) | 200 OK [
http://octopusgardenyoga.com/css/octopus]
Completed in 216ms (DB: 5) | 200 OK [
http://octopusgardenyoga.com/css/gallery]
Completed in 114ms (DB: 6) | 200 OK [http://octopusgardenyoga.com/css/print]
Completed in 3163ms (DB: 187) | 200 [
http://www.octopusgardenyoga.com/events/]
Completed in 1687ms (DB: 78) | 200 [http://www.octopusgardenyoga.com/]
Completed in 392ms (DB: 10) | 200 OK [
http://www.octopusgardenyoga.com/css/gallery]
Completed in 875ms (DB: 47) | 200 OK [
http://www.octopusgardenyoga.com/css/octopus]
Completed in 260ms (DB: 18) | 200 OK [
http://www.octopusgardenyoga.com/css/print]
Completed in 2467ms (DB: 83) | 200 [http://octopusgardenyoga.com/]
Completed in 195ms (DB: 10) | 200 OK [http://octopusgardenyoga.com/css/print
]
Completed in 374ms (DB: 11) | 200 OK [
http://octopusgardenyoga.com/css/gallery]
Completed in 634ms (DB: 130) | 200 OK [
http://octopusgardenyoga.com/css/octopus]
Completed in 1494ms (DB: 81) | 200 [http://octopusgardenyoga.com/fees/]
Completed in 1525ms (DB: 169) | 200 [
http://octopusgardenyoga.com/guidelines/]
Completed in 1216ms (DB: 37) | 200 [http://octopusgardenyoga.com/schedule/]

(sample mysql+nginx/passenger)
Completed in 1328ms (DB: 153) | 200 [
http://www.octopusgardenyoga.com/classes/]
Completed in 1050ms (DB: 60) | 200 [
http://www.octopusgardenyoga.com/FAQ/how-do-i-get-started/]
Completed in 1259ms (DB: 64) | 200 [http://octopusgardenyoga.com/teachers/]
Completed in 263ms (DB: 69) | 200 OK [http://octopusgardenyoga.com/css/print
]
Completed in 364ms (DB: 10) | 200 OK [
http://octopusgardenyoga.com/css/gallery]
Completed in 609ms (DB: 8) | 200 OK [
http://octopusgardenyoga.com/css/octopus]
Completed in 2378ms (DB: 107) | 200 [http://octopusgardenyoga.com/events]
Completed in 1164ms (DB: 34) | 200 [http://octopusgardenyoga.com/news]
Completed in 1571ms (DB: 214) | 200 [http://octopusgardenyoga.com/photos]
Completed in 2441ms (DB: 360) | 200 [
http://octopusgardenyoga.com/photos/mexico-2006]
Completed in 1328ms (DB: 56) | 200 [http://octopusgardenyoga.com/]
Completed in 1143ms (DB: 50) | 200 [
http://www.octopusgardenyoga.com/schedule/]
Completed in 248ms (DB: 5) | 200 OK [
http://www.octopusgardenyoga.com/css/gallery]
Completed in 281ms (DB: 131) | 200 OK [
http://www.octopusgardenyoga.com/css/print]
Completed in 572ms (DB: 11) | 200 OK [
http://www.octopusgardenyoga.com/css/octopus]
Completed in 2112ms (DB: 119) | 200 [
http://octopusgardenyoga.com/photos/mexico-2006]
Completed in 3140ms (DB: 418) | 200 [
http://octopusgardenyoga.com/photos/mexico-2008]

This morning I could not ssh into the slice. Used the Slicehost console and
found this message being repeated continuously:

Out of Memory: Killed process  (ruby)

Had to do a hard reboot to get the system back. I've just started searching
the slicehost forums, found a few others reporting similar problems, so
maybe I'll find something here.

Is it unusual to have 4 ruby processes using about 20% of available memory
each? This is what I'm seeing on top: (BTW, the 0 swap seems to be a result
of resizing the slice. This has been reported on the forums and I'm about to
submit a ticket, but with lots of available memory, it shouldn't be an
immediate problem).

top - 13:06:37 up  2:05,  1 user,  load average: 0.00, 0.02, 0.00
Tasks:  65 total,   1 running,  64 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.2%id,  0.8%wa,  0.0%hi,  0.0%si,
 0.0%st
Mem:524460k total,   370640k used,   153820k free,28396k buffers
Swap:0k total,0k used,0k free,65716k cached

 PID  USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND


 2630 marshal   15   0  218m 108m 4416 S0 21.2   0:05.27 ruby


 3388 marshal   15   0  219m 106m 211

Re: [Radiant] Radiant 0.9 performance

[john muhl]

one other thing that comes to mind is maybe you're missing the mysql
driver (i don't have any evidence or references but i seem to recall
reading that not having it will result in degraded mysql performance).
does `gem list mysql` show something like "mysql (2.8.1)"?

On Sat, Jun 26, 2010 at 11:36 AM, john muhl  wrote:
> On Thu, Jun 24, 2010 at 12:19 PM, Marshal Linfoot  wrote:
>> John, I decided to give nginx+passenger a try. Installed the passenger gem
>> and then "passenger-install-nginx-module" to compile nginx with the
>> passenger module. Little bit of configuring and started it up.
>
> i'm pretty much out of ideas about what the problem may be. i've never
> seen numbers like that ever; not even running in development with
> webrick.
>
> perhaps there is a problem with the slice itself. you could always try
> taking a snapshot of your setup and boot a new slice using that
> snapshot and compare performance on the new slice.
>
>> and 10 minutes later the load avg is back up to 7 and the site is
>> unresponsive. Taking nginx down and restarting litespeed.
>> What the heck is going on??? Desperate for a fix!!!
>
> this sounds like you're using the passenger default which is to run up
> to six application servers. here is an except from an nginx
> configuration file that i use on a 32bit 256mb slicehost server that
> is dedicated to running a single site .
> that server has been running for over a month now with no runaway
> resource usage.
>
>> The mysql database was loaded using taps to copy from my desktop
>> development environment to the slicehost server. Could the database be
>> corrupted? How can I check?
>
> try running in production using your sqlite database.
>

Re: [Radiant] Radiant 0.9 performance

[john muhl]

On Thu, Jun 24, 2010 at 12:19 PM, Marshal Linfoot  wrote:
> John, I decided to give nginx+passenger a try. Installed the passenger gem
> and then "passenger-install-nginx-module" to compile nginx with the
> passenger module. Little bit of configuring and started it up.

i'm pretty much out of ideas about what the problem may be. i've never
seen numbers like that ever; not even running in development with
webrick.

perhaps there is a problem with the slice itself. you could always try
taking a snapshot of your setup and boot a new slice using that
snapshot and compare performance on the new slice.

> and 10 minutes later the load avg is back up to 7 and the site is
> unresponsive. Taking nginx down and restarting litespeed.
> What the heck is going on??? Desperate for a fix!!!

this sounds like you're using the passenger default which is to run up
to six application servers. here is an except from an nginx
configuration file that i use on a 32bit 256mb slicehost server that
is dedicated to running a single site .
that server has been running for over a month now with no runaway
resource usage.

> The mysql database was loaded using taps to copy from my desktop
> development environment to the slicehost server. Could the database be
> corrupted? How can I check?

try running in production using your sqlite database.

Re: [Radiant] Radiant 0.9 performance

[Marshal Linfoot]

John, I decided to give nginx+passenger a try. Installed the passenger gem
and then "passenger-install-nginx-module" to compile nginx with the
passenger module. Little bit of configuring and started it up.

Performance started out okay, but now is MUCH worse. Example timings:

  Parameters: {"action"=>"show_page", "url"=>["schedule"],
"controller"=>"site"}
Completed in 1198120ms (DB: 18562) | 200 [
http://octopusgardenyoga.com/schedule/]

and then same page again 4 minutes later

Parameters: {"action"=>"show_page", "url"=>["schedule"],
"controller"=>"site"}
  Completed in 11021ms (DB: 896) | 200 [
http://www.octopusgardenyoga.com/schedule/]

and 10 minutes later the load avg is back up to 7 and the site is
unresponsive. Taking nginx down and restarting litespeed.

What the heck is going on??? Desperate for a fix!!!

The mysql database was loaded using taps to copy from my desktop
development environment to the slicehost server. Could the database be
corrupted? How can I check?
-- 
marshal

Re: [Radiant] Radiant 0.9 performance

[Marshal Linfoot]

It was an old article (been running this for some years now):
http://www.usefuljaja.com/litespeed

On Wed, Jun 23, 2010 at 8:12 PM, john muhl  wrote:
>
> (just out of curiosity) which article? i did a quick search of their
> articles site but didn't see anything that "had" to be the one you're
> talking about.
>

It was an old article (been running this for some years now):
http://www.usefuljaja.com/litespeed


> since you're on slicehost i'd also look at their 32bit os options (if
> it's not too much of a hassle to switch at this point) which will
> greatly reduce the amount of ram dedicated to all the stuff that isn't
> your rails/radiant app. i recently setup a 32bit debian 5 there and if
> i recall the base system used about 30mb of ram instead of the roughly
> 100mb for the 64bit version; the difference is probably enough to
> support one extra (passenger or whatever) process which is nice. so
> far i haven't noticed any difference in performance.
>

Good suggestion. I've been thinking about upgrading from Hardy. Maybe do
both, 32bit and newer version.

PS. Litespeed uses the ruby-lsapi gem.
-- 
marshal

Re: [Radiant] Radiant 0.9 performance

[john muhl]

On Wed, Jun 23, 2010 at 6:22 PM, Marshal Linfoot  wrote:
> Thanks William and John.
> The http server is Litespeed. No mongrel, thin, etc. One of the reasons I've
> been using it is because it is dead simple to setup/configure and run. It
> was recommended in some Slicehost tutorials as the easiest way to get up and
> running with Rails apps, no need for backend servers or load balancers. It

(just out of curiosity) which article? i did a quick search of their
articles site but didn't see anything that "had" to be the one you're
talking about.

> supports Rack and Rails 2.3.2. BUT, maybe it's time to look at something
> else. What do you suggest, nginx and thin, mongrel, ...? I'm game to try
> something new.

lately i've been using nginx+passenger. if you go with apache use the
mpm worker variant (if at all possible) since it will use less memory
than the prefork version.

since you're on slicehost i'd also look at their 32bit os options (if
it's not too much of a hassle to switch at this point) which will
greatly reduce the amount of ram dedicated to all the stuff that isn't
your rails/radiant app. i recently setup a 32bit debian 5 there and if
i recall the base system used about 30mb of ram instead of the roughly
100mb for the 64bit version; the difference is probably enough to
support one extra (passenger or whatever) process which is nice. so
far i haven't noticed any difference in performance.

Re: [Radiant] Radiant 0.9 performance

[Marshal Linfoot]

Thanks William and John.

The http server is Litespeed. No mongrel, thin, etc. One of the reasons I've
been using it is because it is dead simple to setup/configure and run. It
was recommended in some Slicehost tutorials as the easiest way to get up and
running with Rails apps, no need for backend servers or load balancers. It
supports Rack and Rails 2.3.2. BUT, maybe it's time to look at something
else. What do you suggest, nginx and thin, mongrel, ...? I'm game to try
something new.

I've tried changing several mysql config settings over the past few days
trying to optimize memory and speed. Made some minor improvements today, DB
times are now under 200ms but the total times are still very high. John,
thanks for the link to the msql tuning blog.

As far as I know, I am running in Production mode. Sqlite3 is what I have
set in database.yml for the dev and test environments, mysql for production.
Litespeed has a "Rails environment" setting and it is set to Production for
my vhost. Tweaking mysql settings has an effect, so I'm fairly certain it's
the production mysql database that's being used.

Thanks again for your help.
-- 
marshal

Re: [Radiant] Radiant 0.9 performance

[john muhl]

On Wed, Jun 23, 2010 at 1:56 PM, Marshal Linfoot  wrote:
> The http sever is litespeed.

what is behind there? fastcgi, mongrel, thin, unicorn etc.

> Database is mysql.

are you using the default mysql configuration? if i recall (it's been
a while since i used mysql) the default config isn't really optimized
for speed.


may be a useful read.

> Under these circumstances, I'm seeing these kinds of numbers in the
> production.log when accessing pages on the site:
>   Completed in 1445ms (DB: 127) ...
>   Completed in 1838ms (DB: 317) ...
>   Completed in 4037ms (DB: 998) ...
>   Completed in 4865ms (DB: 488) ...

yikes. are you accidentally running in development mode?

> These are "good" numbers; sometimes the total completed is in the
> 2-4ms range! I don't understand where the bottleneck is. The db
> times are consistently less than 1000ms and usually under 500ms.

1000ms to 500ms db responses sound way slow to me. i just looked at
one of the production servers (running ubuntu 9.10) i have access to
and db responses were all under 100ms using sqlite.

> What's worse though is over time, no idea why, the ruby processes will start
> to runaway and I'll end up with 8-10 of them each vying for 20-30% of
> memory. Machine load avg spikes up to 14 and the system is essentially
> unusable and needs a reboot.

sounds like fcgi. i just had a look at a different production server
that runs 10 thins and it has been up for 135 days with stable memory
usage over that time; no steady increase.

> I'm considering bumping the slice memory up to 512MB, but am concerned that
> all it will do is delay the inevitable runaway ruby processes.

i think you have other issues to track down but as a solution in the
meantime that may be the way to go.

Re: [Radiant] Radiant 0.9 performance

[William Ross]

On 23 Jun 2010, at 19:56, Marshal Linfoot wrote:

> I've setup a radiant 0.9 (edge) instance running on a 256MB Slicehost slice 
> (running Ubuntu Hardy) and am noticing some significant performance issues. 
> Wonder if anyone might shed some light.

I've never seen issues or figures like those you describe. Are you definitely 
running in production mode, and with the cache enabled?

I would suspect litespeed: I've never used it but it doesn't seem to read or 
write etags, probably doesn't support rack::cache, and generally seems to be 
aimed at apache/php users. If your concern is to keep the footprint and process 
count low, can you try nginx and passenger?

will