> -----Original Message-----
> From: Cameron Childress [mailto:[EMAIL PROTECTED]
> Sent: Monday, January 03, 2005 1:13 PM
> To: CF-Talk
> Subject: How'd www.firstnight.org do over new years?
>
> Jim,
>
> Just following up on the thread from last week. I'd be interested in
> knowing how your site did over new years and to learn anything helpful
> you might have to share about your experience with the new years eve
> crush on the firstnight site.
Well - that's a sordid, sordid tale full of highs and lows, villains and
heroes and a deeply fulfilling message that will leave your family closer
than ever.
Anyway... since you asked.
The site started getting high traffic late morning on the 29th. It was,
however sucking up resources and causing all the other sites on the shared
host to slow down so our host (CrystalTech.com) turned it off.
The primary reason, then, for the slow down was that the available database
connections were getting overrun. The site has a "planner" feature for
visitors that allows them to select events that they've like to go see. As
built the planner saves selections to the database whenever it's updated.
With so many people using it the application was squeezing out other
applications on the server.
So I switched the app to only save plans for those (relatively few) people
that logged in and cut down on most of the my end-of-session logging (which
sped up the session manager from taking dozens of seconds every 20 minutes
to taking less than 200ms).
This improved the site performance a lot, but did mean we were losing data.
Ah well.
The traffic died down in the evening. I did some more optimization. I
cached static versions of all navigation, for example. This meant that
pages could no longer present navigation consistent with security rules, but
on this site that's only used for administration so no big loss.
I also found a really dumb problem that (I'm nearly positive) caused serious
problems on server start under load. See the thread called "Well - I feel
like an idiot" posted a few days ago for details on that.
Then I got a call the next morning (the 30th). The traffic was back up,
much higher than before. To protect the other customers CrystalTech shut of
the server again. We attempted to throttle the connection, but that was as
good as having the site disabled (throttling meant that only about a tenth
of the requests were getting through, so even if you DID get the page you'd
miss most of the style sheets, graphics and scripts).
Finally around 2pm the client bit the bullet and let me order a dedicated
server. CrystalTech graciously pushed our build to the top of the queue.
We went with the second option they offer: a P4 3.06GHz with 1 Gig of RAM
and a 120 Gig disk.
The server wasn't released until after 5pm and I went to work building it.
I had to install CFMX, stats, the site, etc. One REALLY nice thing was that
I could still connect to the old database - that saved me more than a little
time.
Around 7pm I was done and we (myself and a couple of volunteers) ran through
a regression test. We were done around 9pm and had some discussion about
how best to transfer traffic.
In the end we decided that a simple redirection on the old server was best
and quickest. Traffic would originate at the old server and get immediately
located over to the raw IP address. Not the best solution, but it beat
waiting for DNS updates to server we most likely couldn't afford to keep.
I turned back on all the logging, added some cache-management headers,
turned on "trusted cache" and attempted to tweak the simultaneous threads
and database threads.
I then went to bed (I'd had less than 6 hours sleep in the previous three
days).
However I SHOULD have remembered to install MX 6.1 updater. That would come
back to bite me.
Anyway I didn't. The new server ran very well - until about 11:45 when JRUN
pegged the CPU and froze up solid. This is a pretty classic symptom of the
older database drivers in 6.1 and would have probably been solved by
installing the updater.
I ended up turning off the plan updates again in an attempt to get things
back on track (there was no time to install the updater then).
But I forgot. Restarting the service got us back on track until about 2:30
when the same thing happened. After that the server ran great for the rest
of the day (under much less traffic). I feel pretty confident that had I
installed the updater the server could have quite happily handled the
traffic.
Also SmarterStats indexing service was running every 15 minutes which could
have contributed to the problems (I turned it off as soon as I noticed it).
The morning of the 1st traffic was back to nothing and I moved the site back
to the old server. We'll most likely be canceling the dedicated server
until next year (but hopefully we'll get it EARLIER next year.)
After that I played lots of "Prince of Persia" and tried Half-Life
deathmatch for the first time (it was great!) ;^)
The end result for the 31st was 5,393,594 Requests ("hits") which translate
to 231,286 page views and 35,048 visits (as reported by SmarterStats). We
transferred about 15Gig of data. Over 50% of that traffic was from
9:30am-1:30pm.
This is actually pretty similar to last year's traffic (which just squeaked
by on the shared server). There were several application changes that could
have produced more load with the same traffic however.
SmarterStats is still rather stupid when it comes to browser calling pretty
much any non-IE brower "Netscape". However it's clear that IE is loosing
its foothold: it had 84.3% of the traffic while "Netscape" (which consists
almost entirely of Firefox and Mozilla) had 14.6%. That's a marked decline
for IE from last year.
Google is still the king of the search engine with over 62% of the
search-engine visits from them. Yahoo was next with 14% and the all the
others trailed far behind.
Interestingly the most common search phrase was "First Night Boston" (over
50%). The next two most common phrases ("Boston First Night" and "First
Night in Boston") took a collective 38%. This indicates that 88% of search
engine visitors new exactly what they're looking for, but didn't know the
URL.
(This also touches on a peeve of mine: we have several media partners her in
Boston that NEVER mention the website. NEVER NEVER NEVER. No wonder nobody
knows were the hell the site is.)
I've not dug into my stats yet (browser details and such) but will soon.
Anything else you'd like to know?
Jim Davis
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Special thanks to the CF Community Suite Silver Sponsor - New Atlanta
http://www.newatlanta.com
Message: http://www.houseoffusion.com/lists.cfm/link=i:4:189214
Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4
Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4
Unsubscribe:
http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Donations & Support: http://www.houseoffusion.com/tiny.cfm/54
