>> That looks interesting, could you do a "show full processlist" on >> your mysql server? > I assume you mean like this ? > > # mysql -e "show full processlist" > +----+------------+--------------------+---------+---------+------+-------+-----------------------+ > | Id | User | Host | db | Command | Time | State | > Info | > +----+------------+--------------------+---------+---------+------+-------+-----------------------+ > | 50 | postfix-ro | 172.16.0.112:49265 | postfix | Sleep | 0 | | > NULL | > | 52 | postfix-ro | 172.16.0.112:49266 | postfix | Sleep | 0 | | > NULL | > | 54 | postfix-ro | 172.16.0.112:49267 | postfix | Sleep | 0 | | > NULL | > | 56 | policyd | localhost | policyd | Sleep | 10 | | > NULL | > | 59 | policyd | localhost | policyd | Sleep | 4 | | > NULL | > | 64 | postfix-ro | 172.16.0.112:49269 | postfix | Sleep | 1 | | > NULL | > | 66 | postfix-ro | 172.16.0.112:49270 | postfix | Sleep | 1 | | > NULL | > | 67 | postfix-ro | 172.16.0.112:49271 | postfix | Sleep | 1 | | > NULL | > | 70 | postfix-ro | 172.16.0.112:49272 | postfix | Sleep | 1 | | > NULL | > | 72 | postfix-ro | 172.16.0.112:49273 | postfix | Sleep | 1 | | > NULL | > | 73 | postfix-ro | 172.16.0.112:49274 | postfix | Sleep | 1 | | > NULL | > | 75 | policyd | localhost | policyd | Sleep | 6 | | > NULL | > | 76 | policyd | localhost | policyd | Sleep | 0 | | > NULL | > | 77 | policyd | localhost | policyd | Sleep | 2 | | > NULL | > | 78 | policyd | localhost | policyd | Sleep | 1 | | > NULL | > | 79 | policyd | localhost | policyd | Sleep | 5 | | > NULL | > | 81 | policyd | localhost | policyd | Sleep | 2 | | > NULL | > | 82 | policyd | localhost | policyd | Sleep | 4 | | > NULL | > | 83 | policyd | localhost | policyd | Sleep | 1 | | > NULL | > | 87 | postfix-ro | 172.16.0.112:49304 | postfix | Sleep | 0 | | > NULL | > | 88 | postfix-ro | 172.16.0.112:49305 | postfix | Sleep | 0 | | > NULL | > | 89 | postfix-ro | 172.16.0.112:49306 | postfix | Sleep | 0 | | > NULL | > | 90 | postfix-ro | 172.16.0.112:49307 | postfix | Sleep | 0 | | > NULL | > | 91 | postfix-ro | 172.16.0.112:49308 | postfix | Sleep | 0 | | > NULL | > | 92 | postfix-ro | 172.16.0.112:49309 | postfix | Sleep | 0 | | > NULL | > | 96 | root | localhost | NULL | Query | 0 | NULL | > show full processlist | > +----+------------+--------------------+---------+---------+------+-------+-----------------------+ > >> Check how many connections are established to the box. Also check the >> query rate and load of the mysql server. I assume you using innodb? > top - 13:14:33 up 10 min, 1 user, load average: 0.06, 0.03, 0.00 > Tasks: 67 total, 2 running, 65 sleeping, 0 stopped, 0 zombie > Cpu(s): 2.0%us, 0.3%sy, 0.0%ni, 94.6%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 1547380k total, 150752k used, 1396628k free, 5660k buffers > Swap: 3906496k total, 0k used, 3906496k free, 64792k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 2337 mysql 20 0 130m 21m 5492 S 1.0 1.4 0:02.60 mysqld > > 2538 policyd 20 0 16960 11m 2496 S 0.7 0.7 0:00.62 cbpolicyd > > 2760 policyd 20 0 16960 11m 2492 S 0.7 0.7 0:00.22 cbpolicyd > > 2782 root 20 0 2392 1096 876 R 0.7 0.1 0:00.02 top > > 1 root 20 0 2104 684 588 S 0.0 0.0 0:01.10 init > > 2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd > > 3 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0 > > 4 root 15 -5 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 > > 5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 > > 6 root 15 -5 0 0 0 S 0.0 0.0 0:00.02 events/0 > > 7 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper > > 39 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0 > > 41 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid > > 42 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify > > 113 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod > > 149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush > > 150 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush > > > # mysql -e "show status" > +-----------------------------------+----------+ > | Variable_name | Value | > +-----------------------------------+----------+ > | Aborted_clients | 190 | > | Aborted_connects | 1 | > > | Compression | OFF | > | Connections | 200 | > | Created_tmp_disk_tables | 0 | > | Created_tmp_files | 5 | > | Created_tmp_tables | 1 | > | Delayed_errors | 0 | > | Delayed_insert_threads | 0 | > | Delayed_writes | 0 | > | Flush_commands | 1 | > > | Innodb_buffer_pool_pages_data | 90 | > | Innodb_buffer_pool_pages_dirty | 10 | > | Innodb_buffer_pool_pages_flushed | 310 | > | Innodb_buffer_pool_pages_free | 420 | > | Innodb_buffer_pool_pages_latched | 0 | > | Innodb_buffer_pool_pages_misc | 2 | > | Innodb_buffer_pool_pages_total | 512 | > | Innodb_buffer_pool_read_ahead_rnd | 5 | > | Innodb_buffer_pool_read_ahead_seq | 0 | > | Innodb_buffer_pool_read_requests | 27106 | > | Innodb_buffer_pool_reads | 43 | > | Innodb_buffer_pool_wait_free | 0 | > | Innodb_buffer_pool_write_requests | 6412 | > | Innodb_data_fsyncs | 962 | > | Innodb_data_pending_fsyncs | 0 | > | Innodb_data_pending_reads | 0 | > | Innodb_data_pending_writes | 0 | > | Innodb_data_read | 3592192 | > | Innodb_data_reads | 66 | > | Innodb_data_writes | 1173 | > | Innodb_data_written | 11215360 | > | Innodb_dblwr_pages_written | 310 | > | Innodb_dblwr_writes | 29 | > | Innodb_log_waits | 0 | > | Innodb_log_write_requests | 1586 | > | Innodb_log_writes | 890 | > | Innodb_os_log_fsyncs | 906 | > | Innodb_os_log_pending_fsyncs | 0 | > | Innodb_os_log_pending_writes | 0 | > | Innodb_os_log_written | 1049088 | > | Innodb_page_size | 16384 | > | Innodb_pages_created | 4 | > | Innodb_pages_read | 86 | > | Innodb_pages_written | 310 | > | Innodb_row_lock_current_waits | 0 | > | Innodb_row_lock_time | 114 | > | Innodb_row_lock_time_avg | 57 | > | Innodb_row_lock_time_max | 113 | > | Innodb_row_lock_waits | 2 | > | Innodb_rows_deleted | 0 | > | Innodb_rows_inserted | 11 | > | Innodb_rows_read | 2199 | > | Innodb_rows_updated | 888 | > | Key_blocks_not_flushed | 0 | > | Key_blocks_unused | 14482 | > | Key_blocks_used | 15 | > | Key_read_requests | 192 | > | Key_reads | 15 | > | Key_write_requests | 0 | > | Key_writes | 0 | > | Last_query_cost | 0.000000 | > | Max_used_connections | 39 | > > > | Open_files | 57 | > | Open_streams | 0 | > | Open_tables | 53 | > | Opened_tables | 0 | > | Prepared_stmt_count | 0 | > | Qcache_free_blocks | 1 | > | Qcache_free_memory | 16660496 | > | Qcache_hits | 7589 | > | Qcache_inserts | 1143 | > | Qcache_lowmem_prunes | 0 | > | Qcache_not_cached | 88 | > | Qcache_queries_in_cache | 96 | > | Qcache_total_blocks | 208 | > | Questions | 10121 | > | Rpl_status | NULL | > > | Table_locks_immediate | 2364 | > | Table_locks_waited | 0 | > > | Threads_cached | 0 | > | Threads_connected | 39 | > | Threads_created | 75 | > | Threads_running | 1 | > | Uptime | 604 | > | Uptime_since_flush_status | 604 | > +-----------------------------------+----------+ > > > I assume that means more to you than it does to me ! > > >> Also, are you getting the same errors in cbpolicyd? > the words "mysql", "warning", "error", and "fatal" don't appear in the log.
I'm unable to reproduce anything similar to this on our side, we have some Policyd boxes in production which handle a very large number of mails per day and maintain about 500 connections to a single MySQL server. I've seen maybe one of those errors in Policyd. Can you maybe shove Policyd into debug mode and see if it appears any query is getting stuck? possibly may be an idea to tcpdump -w the traffic out and see if there are any PSH's or repeated packet transmissions which may indicate packet loss? The MySQL docs say the most common cause of this error is a timeout... http://dev.mysql.com/doc/refman/5.0/en/gone-away.html Regards Nigel
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Users mailing list [email protected] http://lists.policyd.org/mailman/listinfo/users
