Yep log rolling was the problem. I setup my staging box and reproduced it just fine by rolling the date. Taking out the log roller command from my environment.rb setup worked just fine when I rolled days.
-Michael http://javathehutt.blogspot.com On Jan 2, 2007, at 2:55 PM, Ezra Zygmuntowicz wrote: > Hey Michael- > > You should definitely now use the ruby logger to do rotation. That > has caused tons of issues for me and others. Use syslog or logrotate > to rotate the logs as the Logger rotator is borken trust me. > > As for why you are having opne process stay alive and the others > die. I think what causes that is that all four of your mongrels will > try to rotate the logs. And its a free for all. So the one who does > it last 'wins' and stays alive. Or if it is a mysql timeout issue > then what is probably happening is that only one of your mongrels is > getting 'exercised' . By this I mean the load balancer is only > sending requests to one mongrel because there is not enough traffic > to make it drive all four mongrels. This means that the one mongrel > is getting requests which keep its db connection alive. While the > others are not getting requests so their db connectiosn die off. > > I think it's one of these two things. Sop don't use Logger to rotate > logs for starters and see if that fixes it for you. > > Cheers > -Ezra > > On Jan 2, 2007, at 2:32 PM, Michael Kovacs wrote: > >> Hmm... thanks for the suggestion I'll give that a look as well as >> search the archives here. >> >> I do have the logger rotating daily right now: >> config.logger = Logger.new(config.log_path, 'daily') >> >> I think I'll remove that and see if this occurs again. The thing that >> still bothers me though >> is that if one mongrel is broken from this shouldn't they all be? I >> can always count on 1 of 4 mongrels >> working so that every 3rd request is successful :-) In order to >> monitor that I'd have to hit the site twice to >> ensure that I'm not just getting the good mongrel on the first hit. >> Not very fresh. >> >> -Michael >> http://javathehutt.blogspot.com >> >> On Jan 2, 2007, at 1:36 PM, Joey Geiger wrote: >> >>> This may have to do with log rotation. There was a thread about >>> similar issue posted in I believe December. You might want to try >>> searching the mailing list archives. >>> >>> You're probably rotating your logs based on a specific size, >>> which is >>> why it's happening every couple hours, instead of nightly. >>> >>> On 1/2/07, Michael Kovacs <[EMAIL PROTECTED]> wrote: >>>> Hi all, >>>> >>>> I've been having problems with the apache 2.2-mod_proxy_balancer- >>>> mongrel >>>> setup. >>>> >>>> My setup is: >>>> >>>> CentOS 4.3 >>>> apache 2.2.3 (compiled from source) with mod_proxy_balancer >>>> mysql 4.1 >>>> ruby 1.8.4 >>>> mongrel 0.3.14 (I know I need to update but I think this problem is >>>> independent of the mongrel version) >>>> mongrel_cluster 0.2.0 >>>> rails_machine 0.1.1 >>>> >>>> I have apache setup as per Coda's configuration on his blog >>>> posting from >>>> several months back. >>>> http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server- >>>> rails-mongrel-apache-capistrano-and-you/ >>>> >>>> I have 4 mongrels in my cluster. >>>> >>>> Things work fine for periods of time but after several hours of >>>> inactivity >>>> (I think 8 hours or so) I experience oddness where only 1 of the 4 >>>> mongrels >>>> is properly >>>> responding. I end up getting a "500 internal server error" 3 out >>>> of 4 >>>> requests as they round robin from mongrel to mongrel. There is >>>> nothing in >>>> the production >>>> log file nor in the mongrel log. I've reproduced this problem on >>>> my staging >>>> box as well as my production box. >>>> >>>> The last time I reproduced the problem I decided to run "top" and >>>> see what's >>>> going on when I hit the server. >>>> Mongrel does receive every request but mysql is only active on the >>>> 1 request >>>> that works. In the other mongrels it never spikes up in CPU usage. >>>> >>>> Looking at the mysql process list revealed that all of the >>>> processes had >>>> received the "sleep" command but one of the processes is still >>>> working properly. I've played with connection timeouts other than >>>> to set the >>>> timeout in my application's environment >>>> (ActiveRecord::Base.verification_timeout = 14400) as well as the >>>> mysql >>>> interactive_timeout variable but it seems that all the mongrels >>>> should work >>>> or they shouldn't. The fact that 1 out of 4 always works is rather >>>> puzzling >>>> to me. >>>> >>>> Trying a 'killall -USR1 mongrel_rails" to turn debug on simply >>>> killed the 4 >>>> threads running mongrel. So now I'm running the cluster in debug >>>> mode and am >>>> going to just let it sit there for several hours until it happens >>>> again and >>>> hopefully get some idea of where the breakdown is happening. I >>>> still think >>>> it has to be a mysql connection timeout but again, the fact that 1 >>>> of the 4 >>>> always works doesn't lend credence to the timeout theory. >>>> >>>> Has anyone experienced this phenomenon themselves? >>>> >>>> Thanks for any tips/pointers and thanks Zed for all your hard work >>>> with >>>> mongrel. >>>> >>>> >>>> >>>> -Michaelhttp://javathehutt.blogspot.com >>>> >>>> >>>> _______________________________________________ >>>> Mongrel-users mailing list >>>> Mongrel-users@rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/mongrel-users >>>> >>>> >>> _______________________________________________ >>> Mongrel-users mailing list >>> Mongrel-users@rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mongrel-users >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users@rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- [EMAIL PROTECTED] > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users _______________________________________________ Mongrel-users mailing list Mongrel-users@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users
