RE: Wrong statistics for size of XLOG_SWITCH during pg_waldump.

2021-01-06 Thread Shinya11.Kato
>Thanks for review, and sorry for reply so later. > >>I reviewed the patch and found some problems. >>>+ if(startSegNo != endSegNo) >>>+ else if(record->ReadRecPtr / XLOG_BLCKSZ != >>>+ if(rmid == RM_XLOG_ID && info == XLOG_SWITCH) >>>+ if(ri == RM_XLOG_ID) >>>+ if(info == XLOG_SWITCH) >>You need

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Peter Smith
Hi Amit. PSA the v12 patch for the Tablesync Solution1. Differences from v11: + Added PG docs to mention the tablesync slot + Refactored tablesync slot drop (done by DropSubscription/process_syncing_tables_for_sync) + Fixed PG docs mentioning wrong state code + Fixed wrong code comment

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-06 Thread Amul Sul
On Thu, Jan 7, 2021 at 12:45 PM Bharath Rupireddy wrote: > > On Thu, Jan 7, 2021 at 11:32 AM Amul Sul wrote: > > Snip from CreateCheckPoint(): > > -- > > /* > > * Acquire CheckpointLock to ensure only one checkpoint happens at a time. > > * (This is just pro forma, since in the present system

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 07/01/2021 06:15, Michael Paquier wrote: On Wed, Jan 06, 2021 at 03:58:22PM +0200, Heikki Linnakangas wrote: contrib/pgcrypto/internal-sha2.c and src/backend/utils/adt/cryptohashfuncs.c: the call to pg_cryptohash_create() is missing check for NULL result. With your latest patch, that's OK

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-06 Thread Dilip Kumar
On Thu, Jan 7, 2021 at 12:45 PM Bharath Rupireddy wrote: > > On Thu, Jan 7, 2021 at 11:32 AM Amul Sul wrote: > > Snip from CreateCheckPoint(): > > -- > > /* > > * Acquire CheckpointLock to ensure only one checkpoint happens at a time. > > * (This is just pro forma, since in the present system

Re: A failure of standby to follow timeline switch

2021-01-06 Thread Kyotaro Horiguchi
At Thu, 7 Jan 2021 11:55:33 +0900, Fujii Masao wrote in > > > On 2021/01/05 17:26, Kyotaro Horiguchi wrote: > > At Mon, 4 Jan 2021 19:00:21 +0900, Fujii Masao > > wrote in > >> > >> > >> On 2021/01/04 12:06, Kyotaro Horiguchi wrote: > >>> At Sat, 26 Dec 2020 02:15:06 +0900, Fujii Masao > >>>

Re: CheckpointLock needed in CreateCheckPoint()?

2021-01-06 Thread Bharath Rupireddy
On Thu, Jan 7, 2021 at 11:32 AM Amul Sul wrote: > Snip from CreateCheckPoint(): > -- > /* > * Acquire CheckpointLock to ensure only one checkpoint happens at a time. > * (This is just pro forma, since in the present system structure there is > * only one process that is allowed to issue

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 1:30 PM Fujii Masao wrote: > > > > On 2021/01/07 12:42, Masahiko Sawada wrote: > > On Thu, Jan 7, 2021 at 10:59 AM Fujii Masao > > wrote: > >> > >> > >> > >> On 2021/01/07 10:01, Masahiko Sawada wrote: > >>> On Wed, Jan 6, 2021 at 3:37 PM wrote: > > > +#define

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Amit Kapila
On Thu, Jan 7, 2021 at 3:32 AM Peter Geoghegan wrote: > > On Wed, Jan 6, 2021 at 1:29 PM Michael Banck > wrote: > > That one seems to be 5min everywhere, and one can change it except on > > Azure. > > Okay, thanks for clearing that up. Looks like all of the big 3 cloud > providers use Postgres

Re: [PATCH] Identify LWLocks in tracepoints

2021-01-06 Thread Craig Ringer
On Mon, 28 Dec 2020 at 20:09, Masahiko Sawada wrote: > Hi Craig, > > On Sat, Dec 19, 2020 at 2:00 PM Craig Ringer > wrote: > > > > Hi all > > > > The attached patch set follows on from the discussion in [1] "Add LWLock > blocker(s) information" by adding the actual LWLock* and the numeric >

CheckpointLock needed in CreateCheckPoint()?

2021-01-06 Thread Amul Sul
Hi ALL, Snip from CreateCheckPoint(): -- /* * Acquire CheckpointLock to ensure only one checkpoint happens at a time. * (This is just pro forma, since in the present system structure there is * only one process that is allowed to issue checkpoints at any given * time.) */

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2021-01-06 Thread Craig Ringer
On Wed, 6 Jan 2021 at 18:23, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2021-01-06 00:30, Craig Ringer wrote: > > Perhaps debug_invalidate_system_caches_always ? It's a bit long but we > > use the debug prefix elsewhere too. > > Committed with that name. > Thanks very much.

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 4:51 PM Tom Lane wrote: > Thomas Munro writes: > > One of the strange things about these errors is that they're > > asynchronous/unsolicited, but they appear to the client to be the > > response to their next request (if it doesn't eat ECONNRESET instead). > > Right, which

Re: plpgsql variable assignment with union is broken

2021-01-06 Thread Pavel Stehule
čt 7. 1. 2021 v 4:20 odesílatel Merlin Moncure napsal: > On Tue, Jan 5, 2021 at 3:40 PM Tom Lane wrote: > > > > easter...@verfriemelt.org writes: > > > i found, that the behaviour of variable assignment in combination with > union is not working anymore: > > > DO $$ > > > DECLARE t bool; >

Re: Terminate the idle sessions

2021-01-06 Thread Li Japin
-- Best regards Japin Li On Jan 7, 2021, at 10:03 AM, Thomas Munro mailto:thomas.mu...@gmail.com>> wrote: * I'm not entirely comfortable with the name "idle_session_timeout", because it sounds like it applies to all idle states, but actually it only applies when we're not in a transaction. I

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Fujii Masao
On 2021/01/07 12:42, Masahiko Sawada wrote: On Thu, Jan 7, 2021 at 10:59 AM Fujii Masao wrote: On 2021/01/07 10:01, Masahiko Sawada wrote: On Wed, Jan 6, 2021 at 3:37 PM wrote: +#define Query_for_list_of_cursors \ +" SELECT name FROM pg_cursors"\ This query should be the following?

Re: Parallel Inserts in CREATE TABLE AS

2021-01-06 Thread Zhihong Yu
Thanks for the clarification. w.r.t. the commit message, maybe I was momentarily sidetracked by the phrase: With this change. It seems the scenarios you listed are known parallel safety constraints. Probably rephrase that sentence a little bit to make this clearer. Cheers On Wed, Jan 6, 2021

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Amit Kapila
On Wed, Jan 6, 2021 at 3:39 PM Amit Kapila wrote: > > On Wed, Jan 6, 2021 at 2:13 PM Peter Smith wrote: > > > > I think it makes sense. If there can be a race between the tablesync > > re-launching (after error), and the AlterSubscription_refresh removing > > some table’s relid from the

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2021-01-06 Thread Fujii Masao
On 2021/01/05 16:56, Bharath Rupireddy wrote: On Sat, Jan 2, 2021 at 11:19 AM Bharath Rupireddy wrote: I'm sorry for the mess. I missed adding the new files into the v6-0001 patch. Please ignore the v6 patch set and consder the v7 patch set for further review. Note that 0002 and 0003

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Michael Paquier
On Wed, Jan 06, 2021 at 03:58:22PM +0200, Heikki Linnakangas wrote: > contrib/pgcrypto/internal-sha2.c and > src/backend/utils/adt/cryptohashfuncs.c: the call to pg_cryptohash_create() > is missing check for NULL result. With your latest patch, that's OK because > the subsequent

Re: plpgsql variable assignment with union is broken

2021-01-06 Thread Merlin Moncure
On Wed, Jan 6, 2021 at 9:39 PM Tom Lane wrote: > > Merlin Moncure writes: > > On Tue, Jan 5, 2021 at 3:40 PM Tom Lane wrote: > >> easter...@verfriemelt.org writes: > >>> i found, that the behaviour of variable assignment in combination with > >>> union is not working anymore: > >>> DO $$ > >>>

Re: Parallel Inserts in CREATE TABLE AS

2021-01-06 Thread Bharath Rupireddy
On Thu, Jan 7, 2021 at 5:12 AM Zhihong Yu wrote: > > Hi, > For v20-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch : > > workers to Gather node to 0. With this change, there are chances > that the planner may choose the parallel plan. > > It would be nice if the scenarios where a

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-01-06 Thread Tang, Haiying
>I'd like take a look at them and redo some of the tests using my machine. I'll >send my test reults in a separate email after this. I did the same tests with Kirk's scripts using the latest patch on my own machine. The results look pretty good and similar with Kirk's. average of 5 runs.

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Peter Smith
Thankyou for the feedback. On Thu, Jan 7, 2021 at 12:45 PM Hou, Zhijie wrote: > > > PSA the v11 patch for the Tablesync Solution1. > > > > Difference from v10: > > - Addresses several recent review comments. > > - pg_indent has been run > > > Hi > > I took a look into the patch and have some

Re: Terminate the idle sessions

2021-01-06 Thread Tom Lane
Thomas Munro writes: > One of the strange things about these errors is that they're > asynchronous/unsolicited, but they appear to the client to be the > response to their next request (if it doesn't eat ECONNRESET instead). Right, which is what makes class 57 (operator intervention) seem

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 5:39 PM Masahiko Sawada wrote: > > On Mon, Dec 28, 2020 at 5:17 AM Peter Geoghegan wrote: > > > > Simply decreasing vacuum_cost_page_dirty seems like a low risk way of > > making the VACUUM costing more useful within autovacuum workers. > > Halving vacuum_cost_page_dirty

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 10:59 AM Fujii Masao wrote: > > > > On 2021/01/07 10:01, Masahiko Sawada wrote: > > On Wed, Jan 6, 2021 at 3:37 PM wrote: > >> > >>> +#define Query_for_list_of_cursors \ > >>> +" SELECT name FROM pg_cursors"\ > >>> > >>> This query should be the following? > >>> > >>> "

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Michael Paquier
On Wed, Jan 06, 2021 at 03:27:03PM +0200, Heikki Linnakangas wrote: > Looks fine to me. Thanks, I have been able to get this part done as of 55fe26a. -- Michael signature.asc Description: PGP signature

Re: Some more hackery around cryptohashes (some fixes + SHA1)

2021-01-06 Thread Michael Paquier
On Mon, Dec 14, 2020 at 12:48:15PM +0900, Michael Paquier wrote: > This is a nice cleanup, so I have moved ahead and applied it. A > rebased version of the SHA1 business is attached. Rebased version attached to address the conflicts caused by 55fe26a. I have fixed three places in pgcrypto where

Re: plpgsql variable assignment with union is broken

2021-01-06 Thread Tom Lane
Merlin Moncure writes: > On Tue, Jan 5, 2021 at 3:40 PM Tom Lane wrote: >> easter...@verfriemelt.org writes: >>> i found, that the behaviour of variable assignment in combination with >>> union is not working anymore: >>> DO $$ >>> DECLARE t bool; >>> begin >>> t := a FROM ( SELECT true WHERE

Re: Phrase search vs. multi-lexeme tokens

2021-01-06 Thread Alexander Korotkov
Hi! On Wed, Jan 6, 2021 at 8:18 PM Tom Lane wrote: > Alexander Korotkov writes: > > # select to_tsvector('pg_class foo') @@ websearch_to_tsquery('"pg_class > > foo"'); > > ?column? > > -- > > f > > Yeah, surely this is wrong. Thank you for confirming my thoughts. I also felt that

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 3:03 PM Thomas Munro wrote: > On Thu, Jan 7, 2021 at 12:55 PM Tom Lane wrote: > > * The SQLSTATE you chose for the new error condition seems pretty > > random. I do not see it in the SQL standard, so using a code that's > > within the spec-reserved code range is certainly

Re: plpgsql variable assignment with union is broken

2021-01-06 Thread Merlin Moncure
On Tue, Jan 5, 2021 at 3:40 PM Tom Lane wrote: > > easter...@verfriemelt.org writes: > > i found, that the behaviour of variable assignment in combination with > > union is not working anymore: > > DO $$ > > DECLARE t bool; > > begin > > t := a FROM ( SELECT true WHERE false ) t(a)

Re: A failure of standby to follow timeline switch

2021-01-06 Thread Fujii Masao
On 2021/01/05 17:26, Kyotaro Horiguchi wrote: At Mon, 4 Jan 2021 19:00:21 +0900, Fujii Masao wrote in On 2021/01/04 12:06, Kyotaro Horiguchi wrote: At Sat, 26 Dec 2020 02:15:06 +0900, Fujii Masao wrote in On 2020/12/25 12:03, Kyotaro Horiguchi wrote: The attached is the fixed

Re: Transactions involving multiple postgres foreign servers, take 2

2021-01-06 Thread Zhihong Yu
Hi, For pg-foreign/v31-0004-Add-PrepareForeignTransaction-API.patch : However these functions are not neither committed nor aborted at I think the double negation was not intentional. Should be 'are neither ...' For FdwXactShmemSize(), is another MAXALIGN(size) needed prior to the return

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 5:39 PM Masahiko Sawada wrote: > Perhaps you meant to decrease vacuumm_cost_page_miss instead of > vacuum_cost_page_dirty? You're right. Evidently I didn't write this email very carefully. Sorry about that. To say it again: I think that a miss (without dirtying the page)

RE: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > I wondered, couldn't backup management tools utilize the information > in the backup that is needed to be made when wal_level is changed to "none" > for example ? IIRC, someone proposed in the original thread that the change count can be recorded in

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 12:55 PM Tom Lane wrote: > * Thomas' patch for improving timeout.c seems like a great idea, but > it did indeed have a race condition, and I felt the comments could do > with more work. Oops, and thanks! Very happy to see this one in the tree. > * I'm not entirely

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Fujii Masao
On 2021/01/07 10:01, Masahiko Sawada wrote: On Wed, Jan 6, 2021 at 3:37 PM wrote: +#define Query_for_list_of_cursors \ +" SELECT name FROM pg_cursors"\ This query should be the following? " SELECT pg_catalog.quote_ident(name) "\ " FROM pg_catalog.pg_cursors "\ " WHERE

RE: When (and whether) should we improve the chapter on parallel query to accommodate parallel data updates?

2021-01-06 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I think each feature should develop the docs as part of feature > development but if we want to see some additional work like improving > overall docs for parallel execution as you are suggesting then it can > be done separately as well. > > I think you have a valid point but

RE: Single transaction in the tablesync worker?

2021-01-06 Thread Hou, Zhijie
> PSA the v11 patch for the Tablesync Solution1. > > Difference from v10: > - Addresses several recent review comments. > - pg_indent has been run > Hi I took a look into the patch and have some comments. 1. * So the state progression is always: INIT -> DATASYNC -> SYNCWAIT -> - *

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-06 Thread Masahiko Sawada
On Mon, Dec 28, 2020 at 5:17 AM Peter Geoghegan wrote: > > Simply decreasing vacuum_cost_page_dirty seems like a low risk way of > making the VACUUM costing more useful within autovacuum workers. > Halving vacuum_cost_page_dirty to 5 would be a good start, though I > think that a value as low as

Re: [PATCH] Simple progress reporting for COPY command

2021-01-06 Thread Justin Pryzby
On Wed, Jan 06, 2021 at 10:44:49PM +0100, Tomas Vondra wrote: > On 1/5/21 11:02 AM, Josef Šimánek wrote: > > I'm attaching the whole patch since commitfest failed to ingest the > > last incremental on CI. > > > > Yeah, the whole patch needs to be attached for the commitfest tester to work >

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
On Wed, Jan 6, 2021 at 3:37 PM wrote: > > >+#define Query_for_list_of_cursors \ > >+" SELECT name FROM pg_cursors"\ > > > >This query should be the following? > > > >" SELECT pg_catalog.quote_ident(name) "\ > >" FROM pg_catalog.pg_cursors "\ > >" WHERE

Re: list of extended statistics on psql

2021-01-06 Thread Tomas Vondra
On 1/7/21 1:46 AM, Tatsuro Yamada wrote: Hi Tomas, Thanks, and Happy new year to you too. I almost pushed this, but I have one more question. listExtendedStats first checks the server version, and errors out for pre-10 servers. Shouldn't the logic building query check the server version

Re: list of extended statistics on psql

2021-01-06 Thread Tatsuro Yamada
Hi Tomas, Thanks, and Happy new year to you too. I almost pushed this, but I have one more question. listExtendedStats first checks the server version, and errors out for pre-10 servers. Shouldn't the logic building query check the server version too, to decide whether to check the MCV

Re: logical replication worker accesses catalogs in error context callback

2021-01-06 Thread Masahiko Sawada
On Wed, Jan 6, 2021 at 11:42 AM Masahiko Sawada wrote: > > On Wed, Jan 6, 2021 at 11:02 AM Andres Freund wrote: > > > > Hi, > > > > Due to a debug ereport I just noticed that worker.c's > > slot_store_error_callback is doing something quite dangerous: > > > > static void > >

Re: Terminate the idle sessions

2021-01-06 Thread Tom Lane
Li Japin writes: > [ v9-0001-Allow-terminating-the-idle-sessions.patch ] I've reviewed and pushed this. A few notes: * Thomas' patch for improving timeout.c seems like a great idea, but it did indeed have a race condition, and I felt the comments could do with more work. * I'm not entirely

Re: Parallel Inserts in CREATE TABLE AS

2021-01-06 Thread Zhihong Yu
Hi, For v20-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch : workers to Gather node to 0. With this change, there are chances that the planner may choose the parallel plan. It would be nice if the scenarios where parallel plan is not chosen are listed. + if

Re: list of extended statistics on psql

2021-01-06 Thread Tomas Vondra
On 1/5/21 5:26 AM, Tatsuro Yamada wrote: Hi, I rebased the patch set on the master (7e5e1bba03), and the regression test is good. Therefore, I changed the status of the patch: "needs review". Happy New Year! I rebased my patches on HEAD. Please find attached files. :-D Thanks, and

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 1:29 PM Michael Banck wrote: > That one seems to be 5min everywhere, and one can change it except on > Azure. Okay, thanks for clearing that up. Looks like all of the big 3 cloud providers use Postgres checksums in a straightforward way. I don't have much more to say on

Re: [PATCH] Simple progress reporting for COPY command

2021-01-06 Thread Tomas Vondra
On 1/5/21 11:02 AM, Josef Šimánek wrote: I'm attaching the whole patch since commitfest failed to ingest the last incremental on CI. Yeah, the whole patch needs to be attached for the commitfest tester to work correctly - it can't apply pieces from multiple messages, etc. Anyway, I pushed

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Hi, Am Mittwoch, den 06.01.2021, 13:19 -0800 schrieb Peter Geoghegan: > On Wed, Jan 6, 2021 at 1:08 PM Peter Geoghegan wrote: > > So you tested this using "show data_checksums;" in psql in each case? > > > > What does "show full_page_writes;" show you? > > Another consideration is

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 1:08 PM Peter Geoghegan wrote: > So you tested this using "show data_checksums;" in psql in each case? > > What does "show full_page_writes;" show you? Another consideration is checkpoint_timeout and max_wal_size. -- Peter Geoghegan

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Am Mittwoch, den 06.01.2021, 13:08 -0800 schrieb Peter Geoghegan: > On Wed, Jan 6, 2021 at 1:04 PM Michael Banck > wrote: > > At least data_checksums=on for Azure Managed Postgres, Amazon RDS and > > Google Cloud SQL Postgres. It might not be on for all types (I just > > checked the basic one

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 1:04 PM Michael Banck wrote: > At least data_checksums=on for Azure Managed Postgres, Amazon RDS and > Google Cloud SQL Postgres. It might not be on for all types (I just > checked the basic one each earlier today), but I guess it is. So you tested this using "show

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Hi, Am Mittwoch, den 06.01.2021, 12:56 -0800 schrieb Peter Geoghegan: > On Wed, Jan 6, 2021 at 12:30 PM Stephen Frost wrote: > > As already mentioned, it's also, at least today, far > > simpler to disable checksums than to enable them, which is something > > else to consider when thinking about

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 12:30 PM Stephen Frost wrote: > As already mentioned, it's also, at least today, far > simpler to disable checksums than to enable them, which is something > else to consider when thinking about what the default should be. That is a valid concern. I just don't think that

Re: psql \df choose functions by their arguments

2021-01-06 Thread Greg Sabino Mullane
On Sat, Jan 2, 2021 at 1:56 AM Thomas Munro wrote: > ... > It looks like there is a collation dependency here that causes the > test to fail on some systems: > Thanks for pointing that out. I tweaked the function definitions to hopefully sidestep the ordering issue - attached is v4. Cheers,

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Wed, Jan 6, 2021 at 12:03 PM Stephen Frost wrote: > > Do you really believe it to be wrong? Do we stop performing the correct > > write calls in the correct order to the kernel with fsync being off? If > > the kernel actually handles all

Re: [HACKERS] [PATCH] Generic type subscripting

2021-01-06 Thread Pavel Stehule
Hi út 5. 1. 2021 v 20:32 odesílatel Dmitry Dolgov <9erthali...@gmail.com> napsal: > > On Mon, Jan 04, 2021 at 06:56:17PM +0100, Pavel Stehule wrote: > > po 4. 1. 2021 v 14:58 odesílatel Dmitry Dolgov <9erthali...@gmail.com> > > napsal: > > postgres=# update foo set a['c']['c'][10] = '10'; > >

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 12:03 PM Stephen Frost wrote: > Do you really believe it to be wrong? Do we stop performing the correct > write calls in the correct order to the kernel with fsync being off? If > the kernel actually handles all of our write calls correctly and we > cleanly shut down and

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Wed, Jan 6, 2021 at 11:44 AM Stephen Frost wrote: > > Having fsync off won't actually cause corruption unless you have an OS > > crash or don't sync the disks when you reboot the system though- so it's > > a hedge against certain failure

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 11:44 AM Stephen Frost wrote: > Having fsync off won't actually cause corruption unless you have an OS > crash or don't sync the disks when you reboot the system though- so it's > a hedge against certain failure conditions, as is checksums. I find this argument baffling.

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 13:01:59 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > > imv, enabling page checksums is akin to having fsync enabled by default. > > > > Does it impact performance? Yes, surely quite a

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Peter Geoghegan
On Wed, Jan 6, 2021 at 9:55 AM Andres Freund wrote: > Vacuum performance is one of *THE* major complaints about > postgres. Making it run slower by a lot obviously exascerbates that > problem significantly. I think it'd be prohibitively expensive if it > were 1.5x, not to even speak of 15x. +1.

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
Hi, On 2021-01-06 13:01:59 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > > imv, enabling page checksums is akin to having fsync enabled by default. > > > Does it impact performance? Yes, surely quite a lot, but it's also the > > > safe and sane choice when it

Re: set_config() documentation clarification

2021-01-06 Thread Pavel Stehule
> > > >> SET >> g = year % 19, >> c = year / 100, >> h = (c - c/4 - (8*c + 13)/25 + 19*g + 15) % 30, >> i = h - (h/28)*(1 - (h/28)*(29/(h + 1))*((21 - g)/11)), >> j = year + year/4 + i + 2 - c + c/4) % 7, >> p = i - j, >> easter_month = 3 + (p + 26)/30, >> easter_day = 1 + (p + 27

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Am Mittwoch, den 06.01.2021, 19:07 +0100 schrieb Michael Banck: > Am Mittwoch, den 06.01.2021, 09:58 -0800 schrieb Andres Freund: > > It still requires running a binary locally on the DB server, no? Which > > means it'll not be an option on most cloud providers... > > At least Azure and RDS seem

Re: set_config() documentation clarification

2021-01-06 Thread Pavel Stehule
út 5. 1. 2021 v 22:15 odesílatel Joel Jacobson napsal: > On Tue, Jan 5, 2021, at 21:34, Pavel Stehule wrote: > > yes, it is supported. More the schema variables supports RESET to > default on transaction end, > > and supports access rights for usage in security definer functions. > > Nice. > > >

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Am Mittwoch, den 06.01.2021, 09:58 -0800 schrieb Andres Freund: > It still requires running a binary locally on the DB server, no? Which > means it'll not be an option on most cloud providers... At least Azure and RDS seem to have data_checksums on anyway, I don't have a GCP test instance around

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Michael Banck
Hi, On Wed, Jan 06, 2021 at 09:55:08AM -0800, Andres Freund wrote: > On 2021-01-06 12:02:40 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > > > This looks much better from the WAL size perspective, there's

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Magnus Hagander
On Wed, Jan 6, 2021 at 6:58 PM Andres Freund wrote: > > Hi, > > On 2021-01-06 18:27:48 +0100, Magnus Hagander wrote: > > The other argument is that admins can cheaply and quickly turn off > > checksums if they don't want them. > > > > The same cannot be said for turning them *on* again, that's a

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 18:27:48 +0100, Magnus Hagander wrote: > > The other argument is that admins can cheaply and quickly turn off > > checksums if they don't want them. > > > > The same cannot be said for turning them *on* again, that's a very >

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 12:02:40 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > > > Am Samstag, den 02.01.2021, 10:47 -0500 schrieb Stephen Frost: > > > > > I

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
Hi, On 2021-01-06 18:27:48 +0100, Magnus Hagander wrote: > The other argument is that admins can cheaply and quickly turn off > checksums if they don't want them. > > The same cannot be said for turning them *on* again, that's a very > slow offline operation at this time. > > Turning off

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
Hi, On 2021-01-06 12:02:40 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > > Am Samstag, den 02.01.2021, 10:47 -0500 schrieb Stephen Frost: > > > > I agree with this, but I'd also like to propose, again, as has

Re: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > You said > > The use case I imagined is that the user temporarily > > changes wal_level to 'none' from 'replica' or 'logical' to speed up loading > > and > > changes back to the normal. In this case, the backups

Re: Add Information during standby recovery conflicts

2021-01-06 Thread Fujii Masao
On 2020/12/15 0:20, Fujii Masao wrote: On 2020/12/14 21:31, Fujii Masao wrote: On 2020/12/05 12:38, Masahiko Sawada wrote: On Fri, Dec 4, 2020 at 7:22 PM Drouvot, Bertrand wrote: Hi, On 12/4/20 2:21 AM, Fujii Masao wrote: On 2020/12/04 9:28, Masahiko Sawada wrote: On Fri, Dec 4,

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Magnus Hagander
On Wed, Jan 6, 2021 at 8:31 AM Michael Banck wrote: > > Hi, > > Am Mittwoch, den 06.01.2021, 10:52 +0900 schrieb Michael Paquier: > > On Mon, Jan 04, 2021 at 07:11:43PM +0100, Michael Banck wrote: > > > So maybe we should switch on wal_compression if we enable data checksums > > > by default. > >

Re: Phrase search vs. multi-lexeme tokens

2021-01-06 Thread Tom Lane
Alexander Korotkov writes: > # select to_tsvector('pg_class foo') @@ websearch_to_tsquery('"pg_class > foo"'); > ?column? > -- > f Yeah, surely this is wrong. > # select to_tsquery('pg_class <-> foo'); > to_tsquery > -- > ( 'pg' & 'class' ) <->

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Jan 6, 2021 at 12:02:40PM -0500, Stephen Frost wrote: > > > It unfortunately also hurts other workloads. If we moved towards a saner > > > compression algorithm that'd perhaps not be an issue anymore... > > > > I agree that

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Bruce Momjian
On Wed, Jan 6, 2021 at 12:02:40PM -0500, Stephen Frost wrote: > > It unfortunately also hurts other workloads. If we moved towards a saner > > compression algorithm that'd perhaps not be an issue anymore... > > I agree that improving compression performance would be good but I don't > see that

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > Am Samstag, den 02.01.2021, 10:47 -0500 schrieb Stephen Frost: > > > * Michael Paquier (mich...@paquier.xyz) wrote: > > > > On Fri, Jan 01, 2021 at 08:34:34PM +0100, Michael Banck

Re: PoC/WIP: Extended statistics on expressions

2021-01-06 Thread Dean Rasheed
Looking over the statscmds.c changes, there are a few XXX's and FIXME's that need resolving, and I had a couple of other minor comments: + /* +* An expression using mutable functions is probably wrong, +* since if you aren't going to get the same result for the +

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-01-06 Thread Tang, Haiying
Hi Kirk, >And if you want to test, I have already indicated the detailed steps including >the scripts I used. Have fun testing! Thank you for your sharing of test steps and scripts. I'd like take a look at them and redo some of the tests using my machine. I'll send my test reults in a

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 25/12/2020 02:57, Michael Paquier wrote: On Mon, Dec 21, 2020 at 04:28:26PM -0500, Robert Haas wrote: TBH, I think there's no point in return an error here at all, because it's totally non-specific. You have no idea what failed, just that something failed. Blech. If we want to check that ctx

Re: Moving other hex functions to /common

2021-01-06 Thread Bruce Momjian
On Wed, Jan 6, 2021 at 01:10:28PM +0900, Michael Paquier wrote: > > It was very confusing, and this attached patch fixes all of that. I > > also added the pg_ prefix you suggrested. If we want to add dstlen to > > all the functions, we have to do it for all types --- not sure it is > > worth

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 06/01/2021 13:42, Michael Paquier wrote: On Sat, Dec 19, 2020 at 03:13:50PM +0900, Michael Paquier wrote: At the same time, I have taken care of your comment from upthread to return a failure if the caller passes NULL for the context, and adjusted some comments. What do you think of the

[PATCH] Partial foreign key updates in referential integrity triggers

2021-01-06 Thread Paul Martinez
Hey, hackers! I've created a patch to better support referential integrity constraints when using composite primary and foreign keys. This patch allows creating a foreign key using the syntax: FOREIGN KEY (tenant_id, fk_id) REFERENCES fktable ON DELETE SET NULL (fk_id) which means that only

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-01-06 Thread k.jami...@fujitsu.com
On Wed, January 6, 2021 7:04 PM (JST), I wrote: > I will resume the test similar to Tang, because she also executed the original > failover test which I have been doing before. > To avoid confusion and to check if the results from mine and Tang are > consistent, I also did the recovery/failover

Re: New Table Access Methods for Multi and Single Inserts

2021-01-06 Thread Bharath Rupireddy
On Wed, Jan 6, 2021 at 12:56 PM Luc Vlaming wrote: > The main reason for me for wanting a single API is that I would like the > decision of using single or multi inserts to move to inside the tableam. > For e.g. a heap insert we might want to put the threshold at e.g. 100 > rows so that the

Re: Track replica origin progress for Rollback Prepared

2021-01-06 Thread Amit Kapila
On Wed, Jan 6, 2021 at 5:18 PM Michael Paquier wrote: > > On Tue, Jan 05, 2021 at 04:24:21PM +0530, Amit Kapila wrote: > > There are already tests [1] in one of the upcoming patches for logical > > decoding of 2PC which covers this code using which I have found this > > problem. So, I thought

Re: Track replica origin progress for Rollback Prepared

2021-01-06 Thread Michael Paquier
On Tue, Jan 05, 2021 at 04:24:21PM +0530, Amit Kapila wrote: > There are already tests [1] in one of the upcoming patches for logical > decoding of 2PC which covers this code using which I have found this > problem. So, I thought those would be sufficient. I have not checked > the feasibility of

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Michael Paquier
On Sat, Dec 19, 2020 at 03:13:50PM +0900, Michael Paquier wrote: > At the same time, I have taken care of your comment from upthread to > return a failure if the caller passes NULL for the context, and > adjusted some comments. What do you think of the attached? I have looked again at this

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2021-01-06 Thread Peter Eisentraut
On 2021-01-06 00:30, Craig Ringer wrote: Perhaps debug_invalidate_system_caches_always ? It's a bit long but we use the debug prefix elsewhere too. Committed with that name. In your original patch, this hunk in pg_config_manual.h: + * You can define these by editing pg_config_manual.h but

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Amit Kapila
On Wed, Jan 6, 2021 at 2:13 PM Peter Smith wrote: > > I think it makes sense. If there can be a race between the tablesync > re-launching (after error), and the AlterSubscription_refresh removing > some table’s relid from the subscription then there could be lurking > slot/origin tablesync

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-01-06 Thread k.jami...@fujitsu.com
On Sunday, January 3, 2021 10:35 PM (JST), Amit Kapila wrote: > On Sat, Jan 2, 2021 at 7:47 PM k.jami...@fujitsu.com > wrote: > > > > Happy new year. The V38 LGTM. > > Apologies for a bit of delay on posting the test results, but since > > it's the start of commitfest, here it goes and the

RE: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread osumi.takami...@fujitsu.com
Hello, Sawada-san I'll continue the discussion of [2]. We talked about how to recognize the time or LSN when/where wal_level is changed to 'none' there. You said > The use case I imagined is that the user temporarily > changes wal_level to 'none' from 'replica' or 'logical' to speed up loading

Enhance traceability of wal_level changes for backup management

2021-01-06 Thread osumi.takami...@fujitsu.com
Hi, This thread came from another thread about wal_level [1]. Mainly from backup management tools perspective such as pgBackRest, EDB's BART and pg_probackup, it seems worth talking about a way comprehensively to trace and recognize wal_level changes for various purposes and values like

  1   2   >