Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Re: Harry Ambrose 2017-08-25 > Hi All, > > Sorry to open this can of worms again. However, we are still struggling > with this issue across quite a large amount of our estate. Hi, we've just seen exactly this error on a

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi All, Sorry to open this can of worms again. However, we are still struggling with this issue across quite a large amount of our estate. >From doing some further research I stumbled across the following which seems to sum up what we are seeing quite well...

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On Wed, Aug 9, 2017 at 6:27 AM, ADSJ (Adam Sjøgren) wrote: > On 2017-06-21 Adam Sjøgren wrote: > >> Adam Sjøgren wrote: > >>> Meanwhile, I can report that I have upgraded from 9.3.14 to 9.3.17 and >>> the errors keep appearing the log.

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Unfortunately we still see it frequently :( On 9 August 2017 at 14:29, Achilleas Mantzios wrote: > On 09/08/2017 15:27, ADSJ (Adam Sjøgren) wrote: > >> On 2017-06-21 Adam Sjøgren wrote: >> >> Adam Sjøgren wrote: >>>

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 09/08/2017 15:27, ADSJ (Adam Sjøgren) wrote: On 2017-06-21 Adam Sjøgren wrote: Adam Sjøgren wrote: Meanwhile, I can report that I have upgraded from 9.3.14 to 9.3.17 and the errors keep appearing the log. Just to close this, for the record: We

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 2017-06-21 Adam Sjøgren wrote: > Adam Sjøgren wrote: >> Meanwhile, I can report that I have upgraded from 9.3.14 to 9.3.17 and >> the errors keep appearing the log. Just to close this, for the record: We haven't seen the errors since 2017-06-30. We

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi Everyone, Still trying to fathom this one. I have added quite a few log lines to a copy of 9.4.12 and compiled it hoping to find the fault. Below is from the log (at DEBUG5). Apologies for my name in the log lines, it was the easiest way to grep them specifically I also apologise that its a

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Adam Sjøgren wrote: > Meanwhile, I can report that I have upgraded from 9.3.14 to 9.3.17 and > the errors keep appearing the log. Just a quick update with more observations: All the errors in the postgres.log from one of the tables are triggered by a stored procedure that

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose writes: > Not sure whether its relevant or not, however upon adding an ANALYSE before > the second vacuum the issue has not presented when testing. I have managed > 95 cycles thus far. I'm still unable to reproduce :-( --- I ran about two dozen cycles

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, Not sure whether its relevant or not, however upon adding an ANALYSE before the second vacuum the issue has not presented when testing. I have managed 95 cycles thus far. BEGIN; CREATE TABLE x (id BIGSERIAL PRIMARY KEY, payload1 VARCHAR, payload2 VARCHAR, payload3 VARCHAR, payload4 BIGINT,

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, BTW, how do you get that jar to make the test table on a non-default > tablespace? Or are you just putting the whole test DB on a tablespace? > > regards, tom lane > I have been putting the whole database on a tablespace. It seemed easier than modifying the jar.

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose writes: > - Custom tablespaces (where the errors occur) sat on 4 disk RAID-10 (ext3 > filesystem). BTW, how do you get that jar to make the test table on a non-default tablespace? Or are you just putting the whole test DB on a tablespace?

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Achilleas Mantzios writes: > After 2 full attempts, (and after bringing my poor - old workstation to its > knees) it still does not produce the supposed ERROR : Yeah, I've had little luck reproducing it either. However, I noticed a few messages back that Harry is

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

It seems to be very hit and miss... The below is from the machine described in this thread running PostgreSQL 9.4.10: update 10 update 12 update 14 update 16 update 18 Update all Vacuum org.postgresql.util.PSQLException: ERROR: unexpected chunk number 2285 (expected 0)

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, pls tell me, I am currently running 2nd run in my box, (New attempt 2), and > its in the "Attempting vacuum" phase. > What is it supposed to do next? > I got no errors , it has gotten my machine to its knees. > The jar has an endless while loop. Thus please kill the PID when you are done

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 12/06/2017 10:46, Harry Ambrose wrote: Hi, > Their suggestion is to upload to Google Drive. That or use a third party site, like Dropbox. I have uploaded the jar to dropbox, link below (please let me know if you have any issues downloading):

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, > Their suggestion is to upload to Google Drive. That or use a third party site, like Dropbox. I have uploaded the jar to dropbox, link below (please let me know if you have any issues downloading): https://www.dropbox.com/s/96vm465i7rwhcf8/toast-corrupter-aio.jar?dl=0 > So I guess you run

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Alvaro Herrera wrote: > ADSJ (Adam Sjøgren) wrote: > >> Our database has started reporting errors like this: >> >> 2017-05-31 13:48:10 CEST ERROR: unexpected chunk number 0 (expected 1) >> for toast value 14242189 in pg_toast_10919630 > Does the problem still

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 09/06/2017 19:02, Harry Ambrose wrote: Hi, No error messages found. - is your RAM ECC? Did you run any memtest? Yes, memory is ECC. No error messages found. So I guess you run memtest86+ and it reported that your memory is indeed ECC and also that it is working properly? Best

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 06/11/2017 08:34 AM, Tom Lane wrote: Alvaro Herrera writes: I'm unable to run this file. Maybe it was corrupted in transit, given that it was considered to be text. My copy came through fine, so I tried to forward it to you off-list, but gmail rejected it as a

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Alvaro Herrera writes: > I'm unable to run this file. Maybe it was corrupted in transit, given > that it was considered to be text. My copy came through fine, so I tried to forward it to you off-list, but gmail rejected it as a possible security hazard. Do you know

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose wrote: > Hi, > > Please find the jar attached (renamed with a .txt extension as I know some > email services deem jars a security issue). > > The jar accepts the following arguments: > > $1 = host > $2 = database > $3 = username > $4 = password > $5 = port > > It returns its

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

ADSJ (Adam Sjøgren) wrote: > Our database has started reporting errors like this: > > 2017-05-31 13:48:10 CEST ERROR: unexpected chunk number 0 (expected 1) for > toast value 14242189 in pg_toast_10919630 Does the problem still reproduce if you revert commit

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, Please find the jar attached (renamed with a .txt extension as I know some email services deem jars a security issue). The jar accepts the following arguments: $1 = host $2 = database $3 = username $4 = password $5 = port It returns its logging to STDOUT. Please let me know if you require

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, Please find the jar attached (renamed with a .txt extension as I know some email services deem jars a security issue). The jar accepts the following arguments: $1 = host $2 = database $3 = username $4 = password $5 = port It returns its logging to STDOUT. Please let me know if you require

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Robert Lakes writes: > I am new to postgress and I am trying to write my first function to insert, > update or delete and trap errors as a result of the table not existing , > the columns not exist or if any other error simply pass back the sqlstate Please do not hijack an

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Ha guys, I am new to postgress and I am trying to write my first function to insert, update or delete and trap errors as a result of the table not existing , the columns not exist or if any other error simply pass back the sqlstate here's my code can you help CREATE OR REPLACE FUNCTION

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, Maybe you could give some info on : > - your ext3 mkfs and mount options (journal, barriers, etc) > /etc/fstab details below: LABEL=/var/lib/pgsql/var/lib/pgsql ext3defaults 1 2 LABEL=/tablespace1 /tablespace1ext3defaults 1 2

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 09/06/2017 14:44, Harry Ambrose wrote: Hi Tom, Thanks for attempting to replicate the issue. Anyway, the bad news is I couldn't reproduce the problem then and I can't now. I don't know if it's a timing issue or if there's something critical about configuration that I'm not

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi Tom, Thanks for attempting to replicate the issue. Anyway, the bad news is I couldn't reproduce the problem then and I can't > now. I don't know if it's a timing issue or if there's something critical > about configuration that I'm not duplicating. Can you explain what sort > of platform

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose writes: > Please find the jar attached (renamed with a .txt extension as I know some > email services deem jars a security issue). Hmm, the output from this script reminds me quite a lot of one I was sent in connection with bug #1 awhile back:

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 06/08/2017 08:13 AM, ADSJ (Adam Sjøgren) wrote: Achilleas writes: Anyone has a handy little script lying around? http://www.databasesoup.com/2013/10/de-corrupting-toast-tables.html Thanks for the response! Adam -- Adrian Klaver adrian.kla...@aklaver.com -- Sent via

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry writes: > The second vacuum causes an ERROR identical to that you are reporting > below (unexpected chunk number n (expected n) for toast value...). > However it may take up to ten attempts to replicate it. Interesting. > Out of interest, are you using any tablespaces other than

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Achilleas writes: > First try to find which tables those toast relations refer to : > select 10919630::regclass , 10920100::regclass ; > Are those critical tables? Can you restore them somehow? They are our two big tables, containing the bulk of our data (one with 168M rows, the other with 320M

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose writes: > Tom - I can provide a jar that I have been using to replicate the issue. > Whats the best transport method to send it over? If it's not enormous, just send it as an email attachment. regards, tom lane -- Sent via

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, Thanks for the responses. > "One lesson I learned from the BSD camp when dealing with random freezes and > panics : when all else fails to give an answer it is time to start blaming my > hardware. Are those tablespaces on any cheap SSD's ?” The tablespaces are not sat on SSD’s. Something

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Harry Ambrose writes: > I have been following the updates to the 9.4 branch hoping a fix will appear, > but sadly no luck yet. I have manually replicated the issue on 9.4.4, 9.4.10 > and 9.4.12. My replication steps are: This is a very interesting report, but you

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 07/06/2017 17:49, Harry Ambrose wrote: Hi, Out of interest, are you using any tablespaces other than pg_default? I can only replicate the issue when using separately mounted tablespaces. One lesson I learned from the BSD camp when dealing with random freezes and panics : when all else fails

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Hi, I too have been experiencing this with a busy PostgreSQL instance. I have been following the updates to the 9.4 branch hoping a fix will appear, but sadly no luck yet. I have manually replicated the issue on 9.4.4, 9.4.10 and 9.4.12. My replication steps are: BEGIN; CREATE TABLE x (id

Re: [GENERAL] ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

On 07/06/2017 16:33, ADSJ (Adam Sjøgren) wrote: Our database has started reporting errors like this: 2017-05-31 13:48:10 CEST ERROR: unexpected chunk number 0 (expected 1) for toast value 14242189 in pg_toast_10919630 ... 2017-06-01 11:06:56 CEST ERROR: unexpected chunk number 0