Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-10 Thread Adam Sjøgren
Pavan writes: >> FWIW one of our support customers reported a very similar TOAST table >> corruption issue last week which nearly caused an outage. After a lot of >> analysis, I think I've now fully understood the reasons behind the >> corruption, the underlying bug(s) and possible remedy. I am

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-10 Thread Pavan Deolasee
On Fri, Apr 6, 2018 at 8:55 AM, Pavan Deolasee wrote: > > > On Fri, Apr 6, 2018 at 2:34 AM, Tom Lane wrote: > >> a...@novozymes.com (Adam =?utf-8?Q?Sj=C3=B8gren?=) writes: >> >> [... still waiting for the result, I will return with what it said >>

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-05 Thread Tom Lane
a...@novozymes.com (Adam =?utf-8?Q?Sj=C3=B8gren?=) writes: >> [... still waiting for the result, I will return with what it said >> when the server does ...] > It did eventually finish, with the same result: Huh. So what we have here, apparently, is that regular MVCC snapshots think there is

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-05 Thread Adam Sjøgren
Tom writes: >> And when I run the suggested query, I get: > >> efamroot@kat efam=# select chunk_id, chunk_seq, ctid, xmin, xmax, >> length(chunk_data) from pg_toast.pg_toast_10919630 where chunk_id = >> 1698936148 order by 1,2; >> chunk_id | chunk_seq | ctid |xmin| xmax |

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-05 Thread Tom Lane
a...@novozymes.com (Adam =?utf-8?Q?Sj=C3=B8gren?=) writes: > Here's a statement which currently gives an unexpected chunk error: > efamroot@kat efam=# SELECT * FROM efam.sendreference WHERE id = '189909908'; > ERROR: unexpected chunk number 0 (expected 1) for toast value 1698936148 > in

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-04-05 Thread Adam Sjøgren
Adam writes: > Here's a statement which currently gives an unexpected chunk error: > > efamroot@kat efam=# SELECT * FROM efam.sendreference WHERE id = '189909908'; > ERROR: unexpected chunk number 0 (expected 1) for toast value 1698936148 > in pg_toast_10919630 > > And when I run the

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-01-17 Thread Michael Paquier
On Wed, Jan 17, 2018 at 12:16:19PM -0500, Tom Lane wrote: > The basic thrust of these messages is "I'm reading what should be > sequentially numbered data chunks for this toast OID, and the sequence > numbers are wrong". Both of these instances could be explained by > duplicate toast rows (or

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-01-17 Thread Tom Lane
a...@novozymes.com (Adam =?utf-8?Q?Sj=C3=B8gren?=) writes: > Also, the error we are getting is now: "unexpected chunk number 2 > (expected 3) for toast value 1498303849 in pg_toast_10919630", where > previously we've only seen "unexpected chunk number 0 (expected 1)". > We are kind of at a loss,

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-01-16 Thread Adam Sjøgren
Michael writes: > On Tue, Jan 16, 2018 at 07:05:19PM +0100, Adam Sjøgren wrote: >> This sounds very interesting - we are running PostgreSQL 9.3.20. > Which means that we may be looking at a new bug, 9.3.20 is the latest in > the 9.3 set as of today. Yes; unfortunately we have failed to

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-01-16 Thread Michael Paquier
On Tue, Jan 16, 2018 at 07:05:19PM +0100, Adam Sjøgren wrote: > This sounds very interesting - we are running PostgreSQL 9.3.20. Which means that we may be looking at a new bug, 9.3.20 is the latest in the 9.3 set as of today. > Did you ever find out exactly what the change that solved the

Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

2018-01-16 Thread Adam Sjøgren
vozymes.com> > Sent: Tuesday, January 16, 2018 7:18 AM > To: pgsql-gene...@postgresql.org > Subject: Re: ERROR: unexpected chunk number 0 (expected 1) for toast value > 76753264 in pg_toast_10920100 > > We are seeing these "ERROR: unexpected chunk number 0 (expected 1