Hi Tom, I am assuming nothing relevant shows up in dmesg, right?
I have experienced random crashes like that and most of them turned to be HW issues - hard disk and memory banks related. Is it a HW RAID? Have you tried looking at the controller logs? (Megacli). And yes, corrupted tables would fail when restoring them (or even when backuping them). Good luck! Manuel 2012/11/26, Rick James <rja...@yahoo-inc.com>: > Nothing looks bad. > 96G for the buffer_pool is bigger than I have experienced, but I know of no > reason for it to fail (given that you have 128GB of RAM). > >> -----Original Message----- >> From: Tom [mailto:livefortheda...@gmail.com] >> Sent: Wednesday, November 21, 2012 5:17 PM >> To: mysql@lists.mysql.com >> Subject: Innodb, MySQL 5.5.28 - Would an incorrect setting in my.cnf >> cause mysqld to randomly crash on high load? >> >> We have a high-end server, 128GB RAM, 32 Core , Xeon, SSD RAID 10 - >> running Ubuntu 12.04 with MySQL 5.5.28 . Doing random imports to large >> InnoDB tables, over 50+ gigs, randomly after a few hours of heavy load, >> mysql does a Signal 11 and crashes. >> >> We have tried to move hardware. Doing a full dump (but not a restore >> yet) gives no issues. Usually on corrupted tables, a dump would fail >> no? >> >> Below is the crash log and my.cnf . >> >> --- >> 12:45:4 UTC - mysqld got signal 11 ; >> This could be because you hit a bug. It is also possible that this >> binary or one of the libraries it was linked against is corrupt, >> improperly built, or misconfigured. This error can also be caused by >> malfunctioning hardware. >> We will try our best to scrape up some info that will hopefully help >> diagnose the problem, but since we have already crashed, something is >> definitely wrong and this may fail. >> >> key_buffer_size=536870912 >> read_buffer_size=131072 >> max_used_connections=324 >> max_threads=200 >> thread_count=308 >> connection_count=308 >> It is possible that mysqld could use up to key_buffer_size + >> (read_buffer_size + sort_buffer_size)*max_threads = >> 965187 K bytes of memory >> Hope that's ok; if not, decrease some variables in the equation. >> >> Thread pointer: 0x7fc7eb1b5040 >> Attempting backtrace. You can use the following information to find out >> where mysqld died. If you see no messages after this, something went >> terribly wrong... >> stack_bottom = 7fadf6abfe60 thread_stack 0x30000 >> /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fc758522759] >> /usr/sbin/mysqld(handle_fatal_signal+0x483)[0x7fc7583e9ae3] >> /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fc75713bcb0] >> /usr/sbin/mysqld(+0x6671b0)[0x7fc75863a1b0] >> /usr/sbin/mysqld(+0x61d6b9)[0x7fc7585f06b9] >> /usr/sbin/mysqld(+0x630d12)[0x7fc758603d12] >> /usr/sbin/mysqld(+0x6319c2)[0x7fc7586049c2] >> /usr/sbin/mysqld(+0x631d85)[0x7fc758604d85] >> /usr/sbin/mysqld(+0x626e7d)[0x7fc7585f9e7d] >> /usr/sbin/mysqld(+0x633cea)[0x7fc758606cea] >> /usr/sbin/mysqld(+0x6347e2)[0x7fc7586077e2] >> /usr/sbin/mysqld(+0x624426)[0x7fc7585f7426] >> /usr/sbin/mysqld(+0x610871)[0x7fc7585e3871] >> /usr/sbin/mysqld(+0x5d4cb0)[0x7fc7585a7cb0] >> /usr/sbin/mysqld(+0x5b7c9c)[0x7fc75858ac9c] >> /usr/sbin/mysqld(_ZN7handler21read_multi_range_nextEPP18st_key_multi_ra >> nge+0x24)[0x7fc7583e9fe4] >> /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x3c)[0x7fc7584a3c8 >> c] >> /usr/sbin/mysqld(+0x4e9195)[0x7fc7584bc195] >> /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x71)[0x7fc7582f >> 1741] >> /usr/sbin/mysqld(+0x32f025)[0x7fc758302025] >> /usr/sbin/mysqld(_ZN4JOIN4execEv+0x4a5)[0x7fc758311155] >> /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_E >> S2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sele >> ct_lex+0x130)[0x7fc75830d000] >> /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x17c)[0x >> 7fc758312f5c] >> /usr/sbin/mysqld(+0x2f66b4)[0x7fc7582c96b4] >> /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x16d8)[0x7fc7582d1118] >> /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x10f)[0x7fc758 >> 2d5daf] >> /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x13 >> 80)[0x7fc7582d7200] >> /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1bd)[0x7fc75837b7a >> d] >> /usr/sbin/mysqld(handle_one_connection+0x50)[0x7fc75837b810] >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc757133e9a] >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc756864cbd] >> >> Trying to get some variables. >> Some pointers may be invalid and cause the dump to abort. >> Query (7faa4c18e440): is an invalid pointer Connection ID (thread ID): >> 2286 >> Status: NOT_KILLED >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> contains information that should help you find out what is causing the >> crash. >> 121120 12:48:48 [Note] Plugin 'FEDERATED' is disabled. >> 121120 12:48:48 InnoDB: The InnoDB memory heap is disabled >> 121120 12:48:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins >> 121120 12:48:48 InnoDB: Compressed tables use zlib 1.2.3.4 >> 121120 12:48:48 InnoDB: Initializing buffer pool, size = 96.0G >> 121120 12:48:56 InnoDB: Completed initialization of buffer pool >> 121120 12:48:57 InnoDB: highest supported file format is Barracuda. >> InnoDB: Log scan progressed past the checkpoint lsn 1341738337497 >> 121120 12:48:58 InnoDB: Database was not shut down normally! >> InnoDB: Starting crash recovery. >> InnoDB: Reading tablespace information from the .ibd files... >> InnoDB: Restoring possible half-written data pages from the doublewrite >> InnoDB: buffer... >> InnoDB: Doing recovery: scanned up to log sequence number 1341743580160 >> InnoDB: Doing recovery: scanned up to log sequence number 1341748823040 >> InnoDB: Doing recovery: scanned up to log sequence number 1341754065920 >> InnoDB: Doing recovery: scanned up to log sequence number 1341759308800 >> InnoDB: Doing recovery: scanned up to log sequence number 1341764551680 >> ... >> ================ >> my.cnf: >> =============== >> [client] >> port = 3306 >> socket = /var/run/mysqld/mysqld.sock >> >> >> [mysqld_safe] >> socket = /var/run/mysqld/mysqld.sock >> nice = 0 >> >> [mysqld] >> skip-name-resolve >> innodb_file_per_table >> default_storage_engine=InnoDB >> >> user = mysql >> socket = /var/run/mysqld/mysqld.sock >> port = 3306 >> basedir = /usr >> datadir = /data/mysql >> tmpdir = /tmp >> skip-external-locking >> >> key_buffer = 512M >> max_allowed_packet = 128M >> thread_stack = 192K >> thread_cache_size = 64 >> >> myisam-recover = BACKUP >> max_connections = 500 >> table_cache = 812 >> table_definition_cache = 812 >> >> #query_cache_limit = 4M >> #query_cache_size = 512M >> join_buffer_size = 512K >> >> >> innodb_additional_mem_pool_size = 20M >> innodb_buffer_pool_size = 96G >> #innodb_file_io_threads = 4 >> #innodb_thread_concurrency = 12 >> innodb_flush_log_at_trx_commit = 1 >> innodb_log_buffer_size = 8M >> innodb_log_file_size = 1024M >> innodb_log_files_in_group = 2 >> innodb_max_dirty_pages_pct = 90 >> innodb_lock_wait_timeout = 120 >> >> log_error = /var/log/mysql/error.log >> >> long_query_time = 5 >> slow_query_log = 1 >> slow_query_log_file = /var/log/mysql/slowlog.log >> >> [mysqldump] >> quick >> quote-names >> max_allowed_packet = 16M >> >> [mysql] >> >> [isamchk] >> key_buffer = 16M >> ====== >> >> Memory usage, disk space is fine. This only happens during high I/O. >> >> Would anything in the my.cnf file be causing this issue? >> >> tia. >> >> -- >> MySQL General Mailing List >> For list archives: http://lists.mysql.com/mysql >> To unsubscribe: http://lists.mysql.com/mysql > > > -- > MySQL General Mailing List > For list archives: http://lists.mysql.com/mysql > To unsubscribe: http://lists.mysql.com/mysql > > -- Manuel Aróstegui Systems Team tuenti.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql
