We're running MySQL 3.23.56 w/ InnoDB on Red Hat Linux 8.0,
compiled natively with bundled gcc:
# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking
--host=i386-redhat-linux --with-system-zlib --enable-__cxa_atexit
Thread model: posix
gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
For the second time in the last week, mysqld has reached a point where
it started reporting "too many connections". We managed to connect
a root session, and 'show processlist' showed indeed max_connections
number of connections, with all of them frozen in some state or
another executing a query.
mysql> kill <process #>
had no effect, nothing changed.
Interestingly enough, the load average dropped to zero, which
lead me to try...
# ps auxw | grep mysql
which showed only the mysqld "master" thread. Usually there are a whole
mess of other threads too. The one master was using almost no CPU.
I straced the mysqld and got a whole lot of:
select(5, [3 4], NULL, NULL, NULL) = 1 (in [3])
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(3, {sin_family=AF_INET, sin_port=htons(50246),
sin_addr=inet_addr("10.0.0.2")}}, [16]) = 320
fcntl64(3, F_SETFL, O_RDWR) = 0
getsockname(320, {sin_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("10.0.0.8")}}, [16]) = 0
fcntl64(320, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(320, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(320, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(320, SOL_TCP, TCP_NODELAY, [1], 4) = 0
kill(32685, SIGRTMIN) = 0
kill(32685, SIGRTMIN) = 0
select(5, [3 4], NULL, NULL, NULL) = 1 (in [3])
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(3, {sin_family=AF_INET, sin_port=htons(56842),
sin_addr=inet_addr("10.0.0.3")}}, [16]) = 320
fcntl64(3, F_SETFL, O_RDWR) = 0
getsockname(320, {sin_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("10.0.0.8")}}, [16]) = 0
fcntl64(320, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(320, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(320, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(320, SOL_TCP, TCP_NODELAY, [1], 4) = 0
kill(32678, SIGRTMIN) = 0
kill(32678, SIGRTMIN) = 0
select(5, [3 4], NULL, NULL, NULL) = 1 (in [3])
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(3, {sin_family=AF_INET, sin_port=htons(37833),
sin_addr=inet_addr("10.0.0.6")}}, [16]) = 320
fcntl64(3, F_SETFL, O_RDWR) = 0
getsockname(320, {sin_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("10.0.0.8")}}, [16]) = 0
fcntl64(320, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(320, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(320, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(320, SOL_TCP, TCP_NODELAY, [1], 4) = 0
kill(32685, SIGRTMIN) = 0
kill(32685, SIGRTMIN) = 0
select(5, [3 4], NULL, NULL, NULL) = 1 (in [3])
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(3, {sin_family=AF_INET, sin_port=htons(50247),
sin_addr=inet_addr("10.0.0.2")}}, [16]) = 320
fcntl64(3, F_SETFL, O_RDWR) = 0
getsockname(320, {sin_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("10.0.0.8")}}, [16]) = 0
fcntl64(320, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(320, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(320, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(320, SOL_TCP, TCP_NODELAY, [1], 4) = 0
kill(32678, SIGRTMIN) = 0
kill(32678, SIGRTMIN) = 0
select(5, [3 4], NULL, NULL, NULL) = 1 (in [3])
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(3, {sin_family=AF_INET, sin_port=htons(37834),
sin_addr=inet_addr("10.0.0.6")}}, [16]) = 320
fcntl64(3, F_SETFL, O_RDWR) = 0
getsockname(320, {sin_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("10.0.0.8")}}, [16]) = 0
fcntl64(320, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(320, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(320, SOL_IP, IP_TOS, [8], 4) = 0
setsockopt(320, SOL_TCP, TCP_NODELAY, [1], 4) = 0
The 10.0.0.x connects are from our web servers. Wish I could've seen
what killed off all of the threads.
Nothing in the logs.
So, anyone have any idea what the heck's going on?
--
Michael Bacarella 24/7 phone: 1-646-641-8662
Netgraft Corporation http://netgraft.com/
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
