Could I have some more details (or just confirm that the following is correct):
1) set log_rotation_age = 2 in postgresql.conf What do you set the other log_rotation variables to? 2) write a php script that opens a connection and loops on a query Report all errors returned by the query Result is that in ~2minute, a query will issue to the db, and will never return. Do I have this right? Thanks, Bob On Dec 12, 2008, at 8:57 AM, Laser, Mary wrote: > Yes. This is how I was able to reproduce the problem. By forcing > the log to rotate every 2 minutes, we were able to cause a scheduler > hang AND we observed apache db connections hanging, too. > > Mary > >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Gobeille, >> Robert >> Sent: Thursday, December 11, 2008 10:01 PM >> To: Krawetz, Neal >> Cc: [email protected] >> Subject: Re: [FOSSology] Hang identified >> >> Did you try letting Postgres rotate its own logs instead of >> using logrotate? >> >> http://www.postgresql.org/docs/8.1/interactive/runtime-config- >> logging.html >> >> Bob >> >> >> On Dec 11, 2008, at 5:14 PM, Krawetz, Neal wrote: >> >>> Hi Folks, >>> >>> Mary and I just debugged the "every morning" scheduler >> hang. Not only >>> can we repeat it, we can make APACHE hang (without needing >>> FOSSology!). >>> >>> The problem: >>> When PostgreSQL 8.1 does a log rotate, it ends up hanging >> connections >>> to the DB. >>> >>> Since the Scheduler keeps one connection open forever, that >> connection >>> hangs. As a result, all of the scheduler hangs. >>> >>> With Apache, DB connections are sustained until they timeout. There >>> are also multiple DB connections. What we are seeing: some of the >>> Apache connections appear hung -- the PHP scripts that >> access the DB >>> return no data and sometimes the session resets. Simply reloading >>> tells Apache to use a different (random/round-robin) DB connection >>> that will probably work (unless you are unlucky and reload >> using the >>> same hung DB connection). >>> >>> >>> The test: >>> Mary configured postgresql to rotate its logs every 2 minutes. Poof: >>> PHP >>> scripts that connect to the DB under Apache begin to die -- SQL >>> queries act inconsistent (some return results, yet on reload they >>> return >>> nothing) and sessions randomly time out. Also, the scheduler hangs. >>> >>> >>> One possible workaround: >>> When postgresql rotates its logs, we need to restart the scheduler. >>> This >>> can be done in /etc/logrotate.d/postgresql-common. Add a >> section to >>> the >>> post-rotate: >>> >>> postrotate >>> /etc/init.d/fossology restart > /dev/null >>> >>> Alternately, this can be put in /etc/logrotate.d/fossology, but be >>> sure it runs AFTER postgres rotates, and not before. >>> >>> In either case, this is ugly since it means that a stopped >> scheduler >>> will be restarted the next time the logs rotate. >>> >>> I'm open to other solutions if anyone else knows a >> good/elegant way to >>> detect and reset DB connections. >>> >>> >>> FAQ: >>> Why haven't we seen the Apache hangs? >>> Apache disconnects from the DB after a period of inactivity. >>> Since logs rotate between 4:30am and 6:30am, nobody is >> around to play >>> with the UI. All connections time out so Apache does not notice the >>> rotation. >>> >>> Also, Apache's logs rotate around the same time as Postgres, and >>> apache's postrotate says to restart apache. Thus, any >> potentially hung >>> connections are released. >>> >>> >>> So to summarize: >>> The scheduler hangs are not due to the scheduler. They are due to >>> PostgreSQL screwing up active connections when they rotate >> their logs. >>> >>> -Neal >>> >>> >>> _______________________________________________ >>> fossology mailing list >>> [email protected] >>> http://fossology.org/mailman/listinfo/fossology >> >> _______________________________________________ >> fossology mailing list >> [email protected] >> http://fossology.org/mailman/listinfo/fossology >> >> _______________________________________________ fossology mailing list [email protected] http://fossology.org/mailman/listinfo/fossology
