Re: pg_verify_checksums failure with hash indexes

2018-09-05 Thread Michael Paquier
On Wed, Sep 05, 2018 at 12:16:00AM -0400, Tom Lane wrote: > Amit Kapila writes: >> Does anybody else have any idea on how can we write a test for >> non-default block size or if we already have anything similar? > > Build with a non-default BLCKSZ and see if the regression tests pass. > There's

Re: pg_verify_checksums failure with hash indexes

2018-09-04 Thread Amit Kapila
On Wed, Sep 5, 2018 at 9:46 AM Tom Lane wrote: > > Amit Kapila writes: > > Does anybody else have any idea on how can we write a test for > > non-default block size or if we already have anything similar? > > Build with a non-default BLCKSZ and see if the regression tests pass. > There's no way

Re: pg_verify_checksums failure with hash indexes

2018-09-04 Thread Amit Kapila
On Tue, Sep 4, 2018 at 1:42 PM Dilip Kumar wrote: > > I have tested pg_upgrade with different block size (1K, 4K, 8K, 32K). > The upgrade is working fine from v10 to v11 and I am able to fetch > data with index scan on the hash index after an upgrade. > Thanks, do you see any way to write a

Re: pg_verify_checksums failure with hash indexes

2018-09-04 Thread Dilip Kumar
On Tue, Sep 4, 2018 at 10:49 AM, Dilip Kumar wrote: > On Tue, Sep 4, 2018 at 10:14 AM, Amit Kapila wrote: >> On Mon, Sep 3, 2018 at 2:44 PM Dilip Kumar wrote: >>> On Mon, Sep 3, 2018 at 8:37 AM, Amit Kapila wrote: >>> > On Sat, Sep 1, 2018 at 10:28 AM Dilip Kumar wrote: >>> >> >>> >> I think

Re: pg_verify_checksums failure with hash indexes

2018-09-03 Thread Dilip Kumar
On Tue, Sep 4, 2018 at 10:14 AM, Amit Kapila wrote: > On Mon, Sep 3, 2018 at 2:44 PM Dilip Kumar wrote: >> On Mon, Sep 3, 2018 at 8:37 AM, Amit Kapila wrote: >> > On Sat, Sep 1, 2018 at 10:28 AM Dilip Kumar wrote: >> >> >> >> I think if we compute with below formula which I suggested upthread

Re: pg_verify_checksums failure with hash indexes

2018-09-03 Thread Amit Kapila
On Mon, Sep 3, 2018 at 2:44 PM Dilip Kumar wrote: > On Mon, Sep 3, 2018 at 8:37 AM, Amit Kapila wrote: > > On Sat, Sep 1, 2018 at 10:28 AM Dilip Kumar wrote: > >> > >> I think if we compute with below formula which I suggested upthread > >> > >> #define HASH_MAX_BITMAPS

Re: pg_verify_checksums failure with hash indexes

2018-09-03 Thread Amit Kapila
On Mon, Sep 3, 2018 at 7:21 PM Tom Lane wrote: > > Amit Kapila writes: > > AFAICS, this problem exists in 9.6 and prior branches as well, > > although, I can't test it. I am not sure whether we need to backpatch > > this beyond 10 (where hash indexes are WAL logged) as prior to that > >

Re: pg_verify_checksums failure with hash indexes

2018-09-03 Thread Tom Lane
Amit Kapila writes: > AFAICS, this problem exists in 9.6 and prior branches as well, > although, I can't test it. I am not sure whether we need to backpatch > this beyond 10 (where hash indexes are WAL logged) as prior to that > hash-indexes are anyway not-reliable. What's your opinion?

Re: pg_verify_checksums failure with hash indexes

2018-09-03 Thread Dilip Kumar
On Mon, Sep 3, 2018 at 8:37 AM, Amit Kapila wrote: > On Sat, Sep 1, 2018 at 10:28 AM Dilip Kumar wrote: >> >> On Sat, Sep 1, 2018 at 8:22 AM, Robert Haas wrote: >> > On Thu, Aug 30, 2018 at 7:27 AM, Amit Kapila >> > wrote: >> > >> > I wouldn't bother bumping HASH_VERSION. First, the fix

Re: pg_verify_checksums failure with hash indexes

2018-09-02 Thread Amit Kapila
On Tue, Aug 28, 2018 at 6:43 PM Michael Paquier wrote: > > On Tue, Aug 28, 2018 at 11:21:34AM +0200, Peter Eisentraut wrote: > > The files in question correspond to > > > > hash_i4_index > > hash_name_index > > hash_txt_index > > The hash index code has been largely refactored in v10, so I would

Re: pg_verify_checksums failure with hash indexes

2018-09-02 Thread Amit Kapila
On Sat, Sep 1, 2018 at 10:28 AM Dilip Kumar wrote: > > On Sat, Sep 1, 2018 at 8:22 AM, Robert Haas wrote: > > On Thu, Aug 30, 2018 at 7:27 AM, Amit Kapila > > wrote: > > > > I wouldn't bother bumping HASH_VERSION. First, the fix needs to be > > back-patched, and you certainly can't back-patch

Re: pg_verify_checksums failure with hash indexes

2018-08-31 Thread Dilip Kumar
On Sat, Sep 1, 2018 at 8:22 AM, Robert Haas wrote: > On Thu, Aug 30, 2018 at 7:27 AM, Amit Kapila wrote: >> We have previously changed this define in 620b49a1 with the intent to >> allow many non-unique values in hash indexes without worrying to reach >> the limit of the number of overflow

Re: pg_verify_checksums failure with hash indexes

2018-08-31 Thread Robert Haas
On Thu, Aug 30, 2018 at 7:27 AM, Amit Kapila wrote: > We have previously changed this define in 620b49a1 with the intent to > allow many non-unique values in hash indexes without worrying to reach > the limit of the number of overflow pages. I think this didn't occur > to us that it won't work

Re: pg_verify_checksums failure with hash indexes

2018-08-30 Thread Amit Kapila
On Wed, Aug 29, 2018 at 4:05 PM Dilip Kumar wrote: > > On Wed, Aug 29, 2018 at 3:39 PM, Dilip Kumar wrote: > >> SHOW block_size ; > >> block_size > >> > >> 4096 > >> > >> CREATE TABLE foo(val text); > >> INSERT INTO foo VALUES('bernd'); > >> > >> CREATE INDEX ON foo USING

Re: pg_verify_checksums failure with hash indexes

2018-08-30 Thread Yugo Nagata
On Thu, 30 Aug 2018 15:01:24 +0900 (Tokyo Standard Time) Kyotaro HORIGUCHI wrote: > At Wed, 29 Aug 2018 20:10:15 +0900, Yugo Nagata wrote > in <20180829201015.d9d4fde2748910e86a13c...@sraoss.co.jp> > > On Wed, 29 Aug 2018 16:01:53 +0530 > > Amit Kapila wrote: > > > > > > By the way, I think

Re: pg_verify_checksums failure with hash indexes

2018-08-30 Thread Kyotaro HORIGUCHI
At Wed, 29 Aug 2018 20:10:15 +0900, Yugo Nagata wrote in <20180829201015.d9d4fde2748910e86a13c...@sraoss.co.jp> > On Wed, 29 Aug 2018 16:01:53 +0530 > Amit Kapila wrote: > > > > By the way, I think we can fix this also by clearing the header > > > information of the last > > > page instead of

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Yugo Nagata
On Wed, 29 Aug 2018 16:01:53 +0530 Amit Kapila wrote: > > By the way, I think we can fix this also by clearing the header information > > of the last > > page instead of setting a checksum to the unused page although I am not > > sure which way > > is better. > > > > I think that can

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Dilip Kumar
On Wed, Aug 29, 2018 at 3:39 PM, Dilip Kumar wrote: > On Tue, Aug 28, 2018 at 8:33 PM, Bernd Helmle wrote: >> Am Dienstag, den 28.08.2018, 11:21 +0200 schrieb Peter Eisentraut: >>> This is reproducible with PG11 and PG12: >>> >>> initdb -k data >>> postgres -D data >>> >>> make installcheck >>>

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Amit Kapila
On Wed, Aug 29, 2018 at 3:30 PM Yugo Nagata wrote: > > On Wed, 29 Aug 2018 14:39:10 +0530 > Amit Kapila wrote: > > > On Tue, Aug 28, 2018 at 2:51 PM Peter Eisentraut > > wrote: > > > > > > This is reproducible with PG11 and PG12: > > > > > > initdb -k data > > > postgres -D data > > > > > >

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Dilip Kumar
On Tue, Aug 28, 2018 at 8:33 PM, Bernd Helmle wrote: > Am Dienstag, den 28.08.2018, 11:21 +0200 schrieb Peter Eisentraut: >> This is reproducible with PG11 and PG12: >> >> initdb -k data >> postgres -D data >> >> make installcheck >> # shut down postgres with Ctrl-C >> > > I tried to reproduce

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Yugo Nagata
On Wed, 29 Aug 2018 14:39:10 +0530 Amit Kapila wrote: > On Tue, Aug 28, 2018 at 2:51 PM Peter Eisentraut > wrote: > > > > This is reproducible with PG11 and PG12: > > > > initdb -k data > > postgres -D data > > > > make installcheck > > # shut down postgres with Ctrl-C > > > .. > > > > The

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Yugo Nagata
On Tue, 28 Aug 2018 15:02:56 +0200 Michael Banck wrote: > Hi, > > On Tue, Aug 28, 2018 at 11:21:34AM +0200, Peter Eisentraut wrote: > > This is reproducible with PG11 and PG12: > > > > initdb -k data > > postgres -D data > > > > make installcheck > > # shut down postgres with Ctrl-C > > > >

Re: pg_verify_checksums failure with hash indexes

2018-08-29 Thread Amit Kapila
On Tue, Aug 28, 2018 at 2:51 PM Peter Eisentraut wrote: > > This is reproducible with PG11 and PG12: > > initdb -k data > postgres -D data > > make installcheck > # shut down postgres with Ctrl-C > .. > > The files in question correspond to > > hash_i4_index > hash_name_index > hash_txt_index >

Re: pg_verify_checksums failure with hash indexes

2018-08-28 Thread Bernd Helmle
Am Dienstag, den 28.08.2018, 11:21 +0200 schrieb Peter Eisentraut: > This is reproducible with PG11 and PG12: > > initdb -k data > postgres -D data > > make installcheck > # shut down postgres with Ctrl-C > I tried to reproduce this and by accident i had a blocksize=4 in my configure script,

Re: pg_verify_checksums failure with hash indexes

2018-08-28 Thread Amit Kapila
On Tue, Aug 28, 2018 at 6:43 PM Michael Paquier wrote: > > On Tue, Aug 28, 2018 at 11:21:34AM +0200, Peter Eisentraut wrote: > > The files in question correspond to > > > > hash_i4_index > > hash_name_index > > hash_txt_index > > The hash index code has been largely refactored in v10, so I would

Re: pg_verify_checksums failure with hash indexes

2018-08-28 Thread Michael Paquier
On Tue, Aug 28, 2018 at 11:21:34AM +0200, Peter Eisentraut wrote: > The files in question correspond to > > hash_i4_index > hash_name_index > hash_txt_index The hash index code has been largely refactored in v10, so I would imagine that you can see the problem as well there. [... digging

Re: pg_verify_checksums failure with hash indexes

2018-08-28 Thread Michael Banck
Hi, On Tue, Aug 28, 2018 at 11:21:34AM +0200, Peter Eisentraut wrote: > This is reproducible with PG11 and PG12: > > initdb -k data > postgres -D data > > make installcheck > # shut down postgres with Ctrl-C > > pg_verify_checksums data > > pg_verify_checksums: checksum verification failed in