Re: Server Won't Start "Next record offset is nonsensical"

Heikki Tuuri Thu, 18 Nov 2004 08:40:29 -0800

Clif,

I have analyzed some of the hex dumps now:

First there was corruption in the buffer pool:

" 041001 21:44:51 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.21-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 Official MySQL RPM InnoDB: Next record offset is nonsensical 21360 in record at offset 10289 ...

The hex addresses of the last records in the record chain look otherwise ok, but note the odd one: 0x2263. If it were 0x2363, it would be fine. Apparently, one bit has changed in memory.

After that, the ibdata file got corrupted. These are different pages:

" 041106 23:33:03 InnoDB: Page checksum 1633315220, prior-to-4.0.14-form checksum 4288906249 InnoDB: stored checksum 505188979, prior-to-4.0.14-form stored checksum 4288906249 InnoDB: Page lsn 0 222902106, low 4 bytes of lsn at page end 222902106 InnoDB: Page may be an index page where index id is 0 36 InnoDB: (index `PRIMARY` of table `perfparse/perfdata_service_bin`) InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 6452. "

" 041116 13:11:39 InnoDB: Page checksum 2281460337, prior-to-4.0.14-form checksum 3049742901 InnoDB: stored checksum 853267344, prior-to-4.0.14-form stored checksum 3049742901 InnoDB: Page lsn 0 35147564, low 4 bytes of lsn at page end 35147564 "

Page checksum errors are almost certainly caused by a hardware fault or an OS bug in file I/O. Since you suddenly get so many of them, I would suspect a hardware fault.

Heikki Innobase Oy InnoDB - transactions, row level locking, and foreign keys for MySQL InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM tables http://www.innodb.com/order.php

Order MySQL support from http://www.mysql.com/support/index.html

----- Alkuperäinen viesti ----- Lähettäjä: "Clif Smith" <[EMAIL PROTECTED]> Vastaanottaja: <[EMAIL PROTECTED]> Lähetetty: Thursday, November 18, 2004 4:31 AM Aihe: Re: Server Won't Start "Next record offset is nonsensical"

Yes, initially I had thought that there was an issue with the installation itself, so I upgraded to v4.1.7-0. I've attached the log.

Thanks, Clif

Cliff,

your OS or hardware has probably corrupted the ibdata file.

Next record offset is nonsensical 28769 in record at offset 7022

Before writing an index page to the file, InnoDB checks that offsets
are sensible (< 16 kB).

InnoDB: rec address 407b1b6e, first buffer frame 401c0000
InnoDB: buffer pool high end 409c0000, buf fix count 1
041116 17:17:10  InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex

<large snip>

";InnoDB: End of page dump
74 041116 17:17:10  InnoDB: Page checksum 3244520732,
prior-to-4.0.14-form checksum 1495873249
InnoDB: stored checksum 4145305205, prior-to-4.0.14-form stored
checksum 0


Hmm... the 'old checksum' at offsets 16 kB - 8 ... 16 kB - 4 is zero.
That is probably file corruption.

InnoDB: Resetting space id's in the doublewrite buffer


"
       if (mach_read_from_4(doublewrite +
TRX_SYS_DOUBLEWRITE_SPACE_ID_STORED)
           != TRX_SYS_DOUBLEWRITE_SPACE_ID_STORED_N) {

               /* We are upgrading from a version < 4.1.x to a version
where
               multiple tablespaces are supported. We must reset the
space id
               field in the pages in the doublewrite buffer because
starting
               from this version the space id is stored to
               FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID. */

               trx_doublewrite_must_reset_space_ids = TRUE;

               fprintf(stderr,
"InnoDB: Resetting space id's in the doublewrite buffer\n");
       } else {
               trx_sys_multiple_tablespace_format = TRUE;
       }
"

The printout looks like you tried a downgrade and upgrade of MySQL to
resolve the crash? Is that true? Which 4.1.x version you are running?


Yes, initially I had thought that there was an issue with the
installation itself, so I upgraded to v4.1.7-0.

Please send the FULL .err log to me [EMAIL PROTECTED] for
more detailed analysis. Do not cut anything off.

Your lsn is only 300 MB. High-end users of InnoDB on a stable Linux
computer can easily reach 300 GB without any corruption.

Best regards,

Heikki Tuuri
Innobase Oy
Foreign keys, transactions, and row level locking for MySQL
InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up
MyISAM tables
http://www.innodb.com/order.php

Order MySQL technical support from https://order.mysql.com/


----- Original Message ----- From: "Clif Smith" <[EMAIL PROTECTED]>
Newsgroups: mailing.database.myodbc
Sent: Wednesday, November 17, 2004 4:53 AM
Subject: Server Won't Start "Next record offset is nonsensical"

Everything was fine...I haven't installed anything lately, etc.  I've
got a Fedora FC1 system running MySQL v4.  I noticed my db exports
failing this morning.  The db wasn't running and now won't startup.  I'm
googling but...  Here's what's in the log:

41116 17:17:09  mysqld started
041116 17:17:09 [Warning] Asked for 196608 thread stack, but got 126976
041116 17:17:09  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Resetting space id's in the doublewrite buffer
041116 17:17:10  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 296311265.
041116 17:17:10  InnoDB: Starting an apply batch of log records to the
database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 InnoDB:
Next record offset is nonsensical 28769 in record at offset 7022

InnoDB: rec address 407b1b6e, first buffer frame 401c0000
InnoDB: buffer pool high end 409c0000, buf fix count 1
041116 17:17:10  InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex

<large snip>

";InnoDB: End of page dump
74 041116 17:17:10  InnoDB: Page checksum 3244520732,
prior-to-4.0.14-form checksum 1495873249
InnoDB: stored checksum 4145305205, prior-to-4.0.14-form stored
checksum 0
InnoDB: Page lsn 0 296329762, low 4 bytes of lsn at page end 296329762
InnoDB: Page number (if stored to page already) 6570,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 38
041116 17:17:10InnoDB: Assertion failure in thread 12292 in file
../include/page0page.ic line 494
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
75 mysqld got signal 11;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

76 thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbff3ecb8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x808d737
0x82e17a8
0x825f3dd
0x825ee95
0x820d264
0x820e2c5
0x81f2751
0x8231a83
0x813ed39
0x82def5c
0x83088da
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html
and
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html
contains
information that should help you find out what is causing the crash.
77 78 041116 17:17:10  mysqld ended
<end of log>

I read the above and came up with the following:
0x808d737 handle_segfault + 423
0x82e17a8 pthread_sighandler + 184
0x825f3dd page_cur_insert_rec_low + 1261
0x825ee95 page_cur_parse_insert_rec + 3749
0x820d264 recv_parse_or_apply_log_rec_body + 68
0x820e2c5 recv_recover_page + 2933
0x81f2751 buf_page_io_complete + 593
0x8231a83 fil_aio_wait + 899
0x813ed39 io_handler_thread + 25
0x82def5c pthread_start_thread + 220
0x83088da thread_start + 4

But I'm just sad sys admin reading greek at this point...  Any ideas?

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=1

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:    http://lists.mysql.com/[EMAIL PROTECTED]

Re: Server Won't Start "Next record offset is nonsensical"

Reply via email to