Our customer encountered a rare bug of PostgreSQL which prevents a cascaded 
standby from starting up.  The attached patch is a fix for it.  I hope this 
will be back-patched.  I'll add this to the next CF.


The PostgreSQL version is 9.5.  The cluster consists of a master, a cascading 
standby (SB1), and a cascaded standby (SB2).  The WAL flows like this: master 
-> SB1 -> SB2.

The user shut down SB2 and tried to restart it, but failed with the following 

    FATAL:  invalid memory alloc request size 3075129344

The master and SB1 continued to run.


total_len in the code below was about 3GB, so palloc() rejected the memory 
allocation request.

        record = (XLogRecord *) (state->readBuf + RecPtr % XLOG_BLCKSZ);
        total_len = record->xl_tot_len;
         * Enlarge readRecordBuf as needed.
        if (total_len > state->readRecordBufSize &&
                !allocate_recordbuf(state, total_len))

Why was XLogRecord->xl_tot_len so big?  That's because the XLOG reader read the 
garbage portion in a WAL file, which is immediately after the end of valid WAL 

Why was there the garbage?  Because the cascading standby sends just the valid 
WAL records, not the whole WAL blocks, to the cascaded standby.  When the 
cascaded standby receives those WAL records and write them in a recycled WAL 
file, the last WAL block in the file contains the mix of valid WAL records and 
the garbage left over.

Why did the XLOG reader try to allocate memory for a garbage?  Doesn't it stop 
reading the WAL?  As the following code shows, when the WAL record header 
crosses a WAL block boundary, the XLOG reader first allocates memory for the 
whole record, and then checks the validity of the record header as soon as it 
reads the entire header.

         * If the whole record header is on this page, validate it immediately.
         * Otherwise do just a basic sanity check on xl_tot_len, and validate 
         * rest of the header after reading it from the next page.  The 
         * check is necessary here to ensure that we enter the "Need to 
         * record" code path below; otherwise we might fail to apply
         * ValidXLogRecordHeader at all.
        if (targetRecOff <= XLOG_BLCKSZ - SizeOfXLogRecord)
                if (!ValidXLogRecordHeader(state, RecPtr, state->ReadRecPtr, 
                        goto err;
                gotheader = true;
                /* XXX: more validation should be done here */
                if (total_len < SizeOfXLogRecord)
record length at %X/%X: wanted %u, got %u",
(RecPtr >> 32), (uint32) RecPtr,
SizeOfXLogRecord, total_len);
                        goto err;
                gotheader = false;


One idea is to defer the memory allocation until the entire record header is 
read and ValidXLogRecordHeader() is called.  However, ValidXLogRecordHeader() 
might misjudge the validity if the garbage contains xl_prev seeming correct.

So, I modified walreceiver to zero-fill the last partial WAL block.  This is 
the guard that the master does in AdvanceXLInsertBuffer() as below:

                 * Be sure to re-zero the buffer so that bytes beyond what we've
                 * written will look like zeroes and not valid XLOG records...
                MemSet((char *) NewPage, 0, XLOG_BLCKSZ);

FYI, the following unsolved problem may be solved, too.


Takayuki Tsunakawa

Attachment: zerofill_walblock_on_standby.patch
Description: zerofill_walblock_on_standby.patch

Reply via email to