Hi, I have managed to get the dB open with [mysqld] innodb_force_recovery = 2
I did a mysqldump for all tables, and restarted with innodb_force_recordy but still have the same old error messages. Regards, S On Wed, Aug 28, 2013 at 12:12 PM, Simon Loewenthal < simon.loewent...@gmail.com> wrote: > Hi there, > > I just had my mysql dB crash really badly on a small server I'm running > and wonder if someone could point me in the right direction. I'm pretty > basic with mysql... I set up master/slave replication to another site, so > this could be a backup, so long as the errors were not replicated over > there. > > Mysql boots with these types of errors recorded in syslog. Also is the > my.cnf > > > If you know of ideas that could be of help, then I woul dbe really > grateful. I've been trying to get this to run for 4 hours solid. > > Best regards, Simon > > My.cnf > ( My server only has 512Mb of RAM so mysql has to use not a lot). > > > # grep -v ^$ /etc/mysql/my.cnf|grep -v ^# > [client] > port = 3306 > socket = /var/run/mysqld/mysqld.sock > [mysqld_safe] > socket = /var/run/mysqld/mysqld.sock > nice = 0 > [mysqld] > user = mysql > pid-file = /var/run/mysqld/mysqld.pid > socket = /var/run/mysqld/mysqld.sock > port = 3306 > basedir = /usr > datadir = /var/lib/mysql > tmpdir = /tmp > skip-external-locking > bind-address = 127.0.0.1 > max_allowed_packet = 1M > key_buffer = 16K > sort_buffer_size = 64K > read_buffer_size = 256K > thread_stack = 64K > thread_cache_size = 8 > myisam-recover = BACKUP > max_connections = 34 > log_slow_queries = /var/log/mysql/mysql-slow.log > query_cache_limit = 64K > query_cache_size = 2M > expire_logs_days = 10 > max_binlog_size = 100M > table_open_cache = 4 > table_cache = 48 > read_rnd_buffer_size = 256K > net_buffer_length = 128K > server-id = 1 > log-bin=/var/log/mysql/mysql-bin.log > innodb_flush_log_at_trx_commit = 2 > max_binlog_size = 15M > expire_logs_days = 2 > [mysqldump] > quick > quote-names > max_allowed_packet = 1M > [mysql] > [isamchk] > key_buffer = 16M > !includedir /etc/mysql/conf.d/ > [myisamchk] > key_buffer_size = 386K > sort_buffer_size = 64K > read_buffer_size = 256K > > Syslog records 36000 lines of errors from mysql and below please find a > snip it: > > Aug 28 11:50:14 emailserver mysqld_safe: Number of processes running now: 0 > Aug 28 11:50:14 emailserver mysqld_safe: mysqld restarted > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] option > 'thread_stack': unsigned value 65536 adjusted to 131072 > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] The syntax > '--log-slow-queries' is deprecated and will be removed in a future release. > Please use '--slow-query-log'/'--slow-query-log-file' instead. > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Note] Plugin > 'FEDERATED' is disabled. > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: The InnoDB > memory heap is disabled > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Mutexes and > rw_locks use GCC atomic builtins > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Compressed > tables use zlib 1.2.3.4 > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Using Linux > native AIO > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Initializing > buffer pool, size = 128.0M > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Completed > initialization of buffer pool > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: highest > supported file format is Barracuda. > Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Waiting for > the background threads to start > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: 5.5.31 > started; log sequence number 1211163767 > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Recovering > after a crash using /var/log/mysql/mysql-bin > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Starting crash > recovery... > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Crash recovery > finished. > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server hostname > (bind-address): '127.0.0.1'; port: 3306 > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] - '127.0.0.1' > resolves to '127.0.0.1'; > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server socket > created on IP: '127.0.0.1'. > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Event > Scheduler: Loaded 0 events > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] > /usr/sbin/mysqld: ready for connections. > Aug 28 11:50:15 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log' > socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) > Aug 28 11:50:15 emailserver mysqld: InnoDB: Dump of the tablespace extent > descriptor: len 40; hex > 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa; > asc f ; > Aug 28 11:50:15 emailserver mysqld: InnoDB: Serious error! InnoDB is > trying to free page 815 > Aug 28 11:50:15 emailserver mysqld: InnoDB: though it is already marked as > free in the tablespace! > Aug 28 11:50:15 emailserver mysqld: InnoDB: The tablespace free space info > is corrupt. > Aug 28 11:50:15 emailserver mysqld: InnoDB: You may need to dump your > InnoDB tables and recreate the whole > Aug 28 11:50:15 emailserver mysqld: InnoDB: database! > Aug 28 11:50:15 emailserver mysqld: InnoDB: Please refer to > Aug 28 11:50:15 emailserver mysqld: InnoDB: > http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery. > Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: Assertion > failure in thread 139642230933248 in file fsp0fsp.c line 3309 > Aug 28 11:50:15 emailserver mysqld: InnoDB: We intentionally generate a > memory trap. > Aug 28 11:50:15 emailserver mysqld: InnoDB: Submit a detailed bug report > to http://bugs.mysql.com. > Aug 28 11:50:15 emailserver mysqld: InnoDB: If you get repeated assertion > failures or crashes, even > Aug 28 11:50:15 emailserver mysqld: InnoDB: immediately after the mysqld > startup, there may be > Aug 28 11:50:15 emailserver mysqld: InnoDB: corruption in the InnoDB > tablespace. Please refer to > Aug 28 11:50:15 emailserver mysqld: InnoDB: > http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery. > Aug 28 11:50:15 emailserver mysqld: 09:50:15 UTC - mysqld got signal 6 ; > Aug 28 11:50:15 emailserver mysqld: This could be because you hit a bug. > It is also possible that this binary > Aug 28 11:50:15 emailserver mysqld: or one of the libraries it was linked > against is corrupt, improperly built, > Aug 28 11:50:15 emailserver mysqld: or misconfigured. This error can also > be caused by malfunctioning hardware. > Aug 28 11:50:15 emailserver mysqld: We will try our best to scrape up some > info that will hopefully help > Aug 28 11:50:15 emailserver mysqld: diagnose the problem, but since we > have already crashed, > Aug 28 11:50:15 emailserver mysqld: something is definitely wrong and this > may fail. > Aug 28 11:50:15 emailserver mysqld: > Aug 28 11:50:15 emailserver mysqld: key_buffer_size=16384 > Aug 28 11:50:15 emailserver mysqld: read_buffer_size=262144 > Aug 28 11:50:15 emailserver mysqld: max_used_connections=0 > Aug 28 11:50:15 emailserver mysqld: max_threads=34 > Aug 28 11:50:15 emailserver mysqld: thread_count=0 > Aug 28 11:50:15 emailserver mysqld: connection_count=0 > Aug 28 11:50:15 emailserver mysqld: It is possible that mysqld could use > up to > Aug 28 11:50:15 emailserver mysqld: key_buffer_size + (read_buffer_size + > sort_buffer_size)*max_threads = 11283 K bytes of memory > Aug 28 11:50:15 emailserver mysqld: Hope that's ok; if not, decrease some > variables in the equation. > Aug 28 11:50:15 emailserver mysqld: > Aug 28 11:50:15 emailserver mysqld: Thread pointer: 0x0 > Aug 28 11:50:15 emailserver mysqld: Attempting backtrace. You can use the > following information to find out > Aug 28 11:50:15 emailserver mysqld: where mysqld died. If you see no > messages after this, something went > Aug 28 11:50:15 emailserver mysqld: terribly wrong... > Aug 28 11:50:15 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000 > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f011094b14e] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f0110802843] > Aug 28 11:50:15 emailserver mysqld: > /lib/libpthread.so.0(+0xeff0)[0x7f010ffbbff0] > Aug 28 11:50:15 emailserver mysqld: > /lib/libc.so.6(gsignal+0x35)[0x7f010e86d1b5] > Aug 28 11:50:15 emailserver mysqld: > /lib/libc.so.6(abort+0x180)[0x7f010e86ffc0] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x682405)[0x7f0110a6b405] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x682735)[0x7f0110a6b735] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x6394b9)[0x7f0110a224b9] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x6d1b0f)[0x7f0110abab0f] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x6d2438)[0x7f0110abb438] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x6c8255)[0x7f0110ab1255] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x613435)[0x7f01109fc435] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x60656c)[0x7f01109ef56c] > Aug 28 11:50:15 emailserver mysqld: > /usr/sbin/mysqld(+0x6092a3)[0x7f01109f22a3] > Aug 28 11:50:15 emailserver mysqld: > /lib/libpthread.so.0(+0x68ca)[0x7f010ffb38ca] > Aug 28 11:50:15 emailserver mysqld: > /lib/libc.so.6(clone+0x6d)[0x7f010e90ab6d] > Aug 28 11:50:15 emailserver mysqld: The manual page at > http://dev.mysql.com/doc/mysql/en/crashing.html contains > Aug 28 11:50:15 emailserver mysqld: information that should help you find > out what is causing the crash. > Aug 28 11:50:15 emailserver mysqld_safe: Number of processes running now: 0 > Aug 28 11:50:15 emailserver mysqld_safe: mysqld restarted > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] option > 'thread_stack': unsigned value 65536 adjusted to 131072 > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] The syntax > '--log-slow-queries' is deprecated and will be removed in a future release. > Please use '--slow-query-log'/'--slow-query-log-file' instead. > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Note] Plugin > 'FEDERATED' is disabled. > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: The InnoDB > memory heap is disabled > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Mutexes and > rw_locks use GCC atomic builtins > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Compressed > tables use zlib 1.2.3.4 > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Using Linux > native AIO > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Initializing > buffer pool, size = 128.0M > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Completed > initialization of buffer pool > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: highest > supported file format is Barracuda. > Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Waiting for > the background threads to start > Aug 28 11:50:16 emailserver mysqld_safe: mysqld from pid file > /var/run/mysqld/mysqld.pid ended > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: 5.5.31 > started; log sequence number 1211163767 > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Recovering > after a crash using /var/log/mysql/mysql-bin > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Starting crash > recovery... > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Crash recovery > finished. > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server hostname > (bind-address): '127.0.0.1'; port: 3306 > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] - '127.0.0.1' > resolves to '127.0.0.1'; > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server socket > created on IP: '127.0.0.1'. > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Event > Scheduler: Loaded 0 events > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] > /usr/sbin/mysqld: ready for connections. > Aug 28 11:50:17 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log' > socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) > Aug 28 11:50:17 emailserver mysqld: InnoDB: Dump of the tablespace extent > descriptor: len 40; hex > 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa; > asc f ; > Aug 28 11:50:17 emailserver mysqld: InnoDB: Serious error! InnoDB is > trying to free page 815 > Aug 28 11:50:17 emailserver mysqld: InnoDB: though it is already marked as > free in the tablespace! > Aug 28 11:50:17 emailserver mysqld: InnoDB: The tablespace free space info > is corrupt. > Aug 28 11:50:17 emailserver mysqld: InnoDB: You may need to dump your > InnoDB tables and recreate the whole > Aug 28 11:50:17 emailserver mysqld: InnoDB: database! > Aug 28 11:50:17 emailserver mysqld: InnoDB: Please refer to > Aug 28 11:50:17 emailserver mysqld: InnoDB: > http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery. > Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: Assertion > failure in thread 140076906338048 in file fsp0fsp.c line 3309 > Aug 28 11:50:17 emailserver mysqld: InnoDB: We intentionally generate a > memory trap. > Aug 28 11:50:17 emailserver mysqld: InnoDB: Submit a detailed bug report > to http://bugs.mysql.com. > Aug 28 11:50:17 emailserver mysqld: InnoDB: If you get repeated assertion > failures or crashes, even > Aug 28 11:50:17 emailserver mysqld: InnoDB: immediately after the mysqld > startup, there may be > Aug 28 11:50:17 emailserver mysqld: InnoDB: corruption in the InnoDB > tablespace. Please refer to > Aug 28 11:50:17 emailserver mysqld: InnoDB: > http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery. > Aug 28 11:50:17 emailserver mysqld: 09:50:17 UTC - mysqld got signal 6 ; > Aug 28 11:50:17 emailserver mysqld: This could be because you hit a bug. > It is also possible that this binary > Aug 28 11:50:17 emailserver mysqld: or one of the libraries it was linked > against is corrupt, improperly built, > Aug 28 11:50:17 emailserver mysqld: or misconfigured. This error can also > be caused by malfunctioning hardware. > Aug 28 11:50:17 emailserver mysqld: We will try our best to scrape up some > info that will hopefully help > Aug 28 11:50:17 emailserver mysqld: diagnose the problem, but since we > have already crashed, > Aug 28 11:50:17 emailserver mysqld: something is definitely wrong and this > may fail. > Aug 28 11:50:17 emailserver mysqld: > Aug 28 11:50:17 emailserver mysqld: key_buffer_size=16384 > Aug 28 11:50:17 emailserver mysqld: read_buffer_size=262144 > Aug 28 11:50:17 emailserver mysqld: max_used_connections=0 > Aug 28 11:50:17 emailserver mysqld: max_threads=34 > Aug 28 11:50:17 emailserver mysqld: thread_count=0 > Aug 28 11:50:17 emailserver mysqld: connection_count=0 > Aug 28 11:50:17 emailserver mysqld: It is possible that mysqld could use > up to > Aug 28 11:50:17 emailserver mysqld: key_buffer_size + (read_buffer_size + > sort_buffer_size)*max_threads = 11283 K bytes of memory > Aug 28 11:50:17 emailserver mysqld: Hope that's ok; if not, decrease some > variables in the equation. > Aug 28 11:50:17 emailserver mysqld: > Aug 28 11:50:17 emailserver mysqld: Thread pointer: 0x0 > Aug 28 11:50:17 emailserver mysqld: Attempting backtrace. You can use the > following information to find out > Aug 28 11:50:17 emailserver mysqld: where mysqld died. If you see no > messages after this, something went > Aug 28 11:50:17 emailserver mysqld: terribly wrong... > Aug 28 11:50:17 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000 > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f664541014e] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f66452c7843] > Aug 28 11:50:17 emailserver mysqld: > /lib/libpthread.so.0(+0xeff0)[0x7f6644a80ff0] > Aug 28 11:50:17 emailserver mysqld: > /lib/libc.so.6(gsignal+0x35)[0x7f66433321b5] > Aug 28 11:50:17 emailserver mysqld: > /lib/libc.so.6(abort+0x180)[0x7f6643334fc0] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x682405)[0x7f6645530405] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x682735)[0x7f6645530735] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x6394b9)[0x7f66454e74b9] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x6d1b0f)[0x7f664557fb0f] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x6d2438)[0x7f6645580438] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x6c8255)[0x7f6645576255] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x613435)[0x7f66454c1435] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x60656c)[0x7f66454b456c] > Aug 28 11:50:17 emailserver mysqld: > /usr/sbin/mysqld(+0x6092a3)[0x7f66454b72a3] > Aug 28 11:50:17 emailserver mysqld: > /lib/libpthread.so.0(+0x68ca)[0x7f6644a788ca] > Aug 28 11:50:17 emailserver mysqld: > /lib/libc.so.6(clone+0x6d)[0x7f66433cfb6d] > Aug 28 11:50:17 emailserver mysqld: The manual page at > http://dev.mysql.com/doc/mysql/en/crashing.html contains > Aug 28 11:50:17 emailserver mysqld: information that should help you find > out what is causing the crash. > > > > > -- -- GSM: +31.6.2000.5427
