It seems that it was the Postgres bug on replica, after upgrading minor version to 9.1.21 on replica1, the corruption goes away.
Thanks everyone for the help On Tue, Apr 5, 2016 at 1:32 AM, Soni M <diptat...@gmail.com> wrote: > Hello Adrian, thanks for the response. > > master data also located on SAN > > Yes, each replica is it own VM with its own virtual disk/volume as served > up from the same SAN > > Raw disk mappings are a way for ESX to present a SAN volume directly to a > VM instead of creating a virtual disk. > > no unexpected messages detected. > > On Sun, Apr 3, 2016 at 11:23 PM, Adrian Klaver <adrian.kla...@aklaver.com> > wrote: > >> On 04/02/2016 08:38 PM, Soni M wrote: >> >>> Hello Everyone, >>> >>> We face TOAST table corruption. >>> >>> One master and two streaming replicas. The corruption happen only on >>> both streaming replicas. >>> >>> We did found the corrupted rows. Selecting on this row, return (on both >>> replica) : unexpected chunk number 0 (expected 1) for toast value >>> 1100613112 in pg_toast_112517 >>> selecting this row on master does not return corruption error, but >>> return correct result instead. >>> >>> Previously, dump on a replica return : unexpected chunk number 0 >>> (expected 1) for toast value 3234098599 in pg_toast_112517 (please note >>> the toast value is different) >>> >>> This table size is 343 GB, contain around 206,179,697 live tuples. We >>> found that the corruption happen on the biggest column (this column and >>> its pkey sized around 299 GB total). >>> >>> >> on both replica : >>> fsync NEVER turned off. >>> none unexpected power loss nor OS crash. >>> >>> How can the corruption occurs ? and how can I resolve them ? >>> >> >> Meant to add to previous post. >> >> Do you see anything in the replica Postgres logs that indicate a problem >> with the replication process? >> >> Or any other unexpected messages prior to the point you did the select on >> the replica(s)? >> >> >> >>> Thank so much for the help. >>> >>> Cheers \o/ >>> >>> -- >>> Regards, >>> >>> Soni Maula Harriz >>> >> >> >> -- >> Adrian Klaver >> adrian.kla...@aklaver.com >> > > > > -- > Regards, > > Soni Maula Harriz > -- Regards, Soni Maula Harriz
