Re: REINDEX SCHEMA/DATABASE/SYSTEM weak with dropped relations

2020-09-01 Thread Michael Paquier
On Tue, Sep 01, 2020 at 09:41:48PM -0500, Justin Pryzby wrote: > I think you probably intended to write: 1<<2 Thanks, fixed. -- Michael signature.asc Description: PGP signature

Re: Parallel copy

2020-09-01 Thread vignesh C
On Tue, Sep 1, 2020 at 3:39 PM Greg Nancarrow wrote: > > Hi Vignesh, > > >Can you share with me the script you used to generate the data & the ddl of > >the table, so that it will help me check that >scenario you faced the > >>problem. > > Unfortunately I can't directly share it (considered

builtin functions, parameter names and psql's \df

2020-09-01 Thread Andres Freund
Hi, on a regular basis I remember a builtin function's name, or can figure it out using \df etc, but can't remember the argument order. A typical example is regexp_*, where I never remember whether the pattern or the input string comes first. Unfortunatly \df does not really help with that: =#

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-01 Thread Amit Kapila
On Tue, Sep 1, 2020 at 8:33 PM Amit Kapila wrote: > > I have fixed all the comments except .. > 3. +# Change the local values of the extra columns on the subscriber, > +# update publisher, and check that subscriber retains the expected > +# values > +$node_subscriber->safe_psql('postgres',

A micro-optimisation for walkdir()

2020-09-01 Thread Thomas Munro
Hello hackers, You don't need to call stat() just to find out if a dirent is a file or directory, most of the time. Please see attached. From cc2f0fd4a078728a67d862e2deec0332fb8b3555 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 2 Sep 2020 16:15:09 +1200 Subject: [PATCH] Skip

describe-config issue

2020-09-01 Thread vignesh C
Hi, Postgres's describe-config option prints reset_val for int & real configuration parameters which is not useful as it is not updated. Printing boot_val is better in this case. reset_val is updated with boot_val while the server is getting started but in case of postgres --describe-config this

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Justin Pryzby
On Tue, Sep 01, 2020 at 09:24:10PM -0500, Justin Pryzby wrote: > On Wed, Sep 02, 2020 at 10:00:12AM +0900, Michael Paquier wrote: > > On Tue, Sep 01, 2020 at 11:48:30AM -0400, Alvaro Herrera wrote: > > > On 2020-Sep-01, Justin Pryzby wrote: > > >> The question isn't whether to use a parenthesized

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-09-01 Thread David Rowley
On Sat, 29 Aug 2020 at 02:54, David Rowley wrote: > > On Wed, 26 Aug 2020 at 03:52, Andres Freund wrote: > > There'll be a significant reduction in increase in performance. > > So I did a very rough-cut change to the patch to have the caching be > part of Nested Loop. It can be applied on top

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread k.jami...@fujitsu.com
On Wednesday, September 2, 2020 10:31 AM, Kyotaro Horiguchi wrote: > Hello. > > At Tue, 1 Sep 2020 13:02:28 +, "k.jami...@fujitsu.com" > wrote in > > On Tuesday, August 18, 2020 3:05 PM (GMT+9), Amit Kapila wrote: > > > Today, again thinking about this point it occurred to me that during > >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread Tom Lane
Amit Kapila writes: > Even if the relation is locked, background processes like checkpointer > can still touch the relation which might cause problems. Consider a > case where we extend the relation but didn't flush the newly added > pages. Now during truncate operation, checkpointer can still

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread Amit Kapila
On Wed, Sep 2, 2020 at 7:01 AM Kyotaro Horiguchi wrote: > > Hello. > > At Tue, 1 Sep 2020 13:02:28 +, "k.jami...@fujitsu.com" > wrote in > > On Tuesday, August 18, 2020 3:05 PM (GMT+9), Amit Kapila wrote: > > > Today, again thinking about this point it occurred to me that during > > >

Re: REINDEX SCHEMA/DATABASE/SYSTEM weak with dropped relations

2020-09-01 Thread Justin Pryzby
> diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h > index 47d4c07306..23840bb8e6 100644 > --- a/src/include/nodes/parsenodes.h > +++ b/src/include/nodes/parsenodes.h > @@ -3352,6 +3352,7 @@ typedef struct ConstraintsSetStmt > /* Reindex options */ > #define

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-01 Thread Bruce Momjian
On Wed, Sep 2, 2020 at 10:45:30AM +0900, Kyotaro Horiguchi wrote: > At Tue, 1 Sep 2020 11:47:34 -0400, Bruce Momjian wrote in > > OK, I have developed the attached patch based on yours. I reordered the > > tests, simplified the documentation, and removed the hint since they > > Looks good to

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Justin Pryzby
On Wed, Sep 02, 2020 at 10:00:12AM +0900, Michael Paquier wrote: > On Tue, Sep 01, 2020 at 11:48:30AM -0400, Alvaro Herrera wrote: > > On 2020-Sep-01, Justin Pryzby wrote: > >> The question isn't whether to use a parenthesized option list. I realized > >> that > >> long ago (even though Alexey

Re: SyncRepLock acquired exclusively in default configuration

2020-09-01 Thread Andres Freund
On 2020-09-02 10:58:58 +0900, Fujii Masao wrote: > Asim and Sawada-san, thanks for the review! I pushed the patch. Thanks for all your combined work!

Re: SyncRepLock acquired exclusively in default configuration

2020-09-01 Thread Fujii Masao
On 2020/08/28 21:20, Masahiko Sawada wrote: On Fri, 28 Aug 2020 at 10:33, Fujii Masao wrote: On 2020/08/27 15:59, Asim Praveen wrote: On 26-Aug-2020, at 11:10 PM, Fujii Masao wrote: I added the following comments based on the suggestion by Sawada-san upthread. Thought? + *

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Michael Paquier
On Tue, Sep 01, 2020 at 09:29:28PM -0400, Alvaro Herrera wrote: > Seems sensible, but only to be done when actually needed, right? Of course. -- Michael signature.asc Description: PGP signature

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-01 Thread Kyotaro Horiguchi
Hello. At Tue, 1 Sep 2020 11:47:34 -0400, Bruce Momjian wrote in > On Tue, Sep 1, 2020 at 01:59:25PM +0900, Kyotaro Horiguchi wrote: > > At Mon, 31 Aug 2020 11:34:29 -0400, Bruce Momjian wrote > > in > > > On Mon, Aug 31, 2020 at 05:56:58PM +0900, Kyotaro Horiguchi wrote: > > > > Ok, this

Re: 回复:how to create index concurrently on partitioned table

2020-09-01 Thread Michael Paquier
On Fri, Aug 14, 2020 at 09:29:45AM +0900, Michael Paquier wrote: > Once this gets done, we should then be able to get rid of the extra > session locking taken when building the list of partitions, limiting > session locks to only be taken during the concurrent reindex of a > single partition (the

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread Kyotaro Horiguchi
I'd like make a subtle correction. At Wed, 02 Sep 2020 10:31:22 +0900 (JST), Kyotaro Horiguchi wrote in > By the way > > > #define BUF_DROP_THRESHOLD 500 /* NBuffers divided by 2 */ > > NBuffers is not a constant. Even if we wanted to set the macro as > described in the comment,

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread Kyotaro Horiguchi
Hello. At Tue, 1 Sep 2020 13:02:28 +, "k.jami...@fujitsu.com" wrote in > On Tuesday, August 18, 2020 3:05 PM (GMT+9), Amit Kapila wrote: > > Today, again thinking about this point it occurred to me that during > > recovery > > we can reliably find the relation size and after Thomas's

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Sep-02, Michael Paquier wrote: > Yeah, I am all for removing "concurrent" from ReindexStmt, but I don't > think that the proposed 0002 is that, because it is based on the > assumption that we'd want more than just boolean-based options in > those statements, and this case is not justified

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Michael Paquier
On Tue, Sep 01, 2020 at 11:48:30AM -0400, Alvaro Herrera wrote: > On 2020-Sep-01, Justin Pryzby wrote: >> The question isn't whether to use a parenthesized option list. I realized >> that >> long ago (even though Alexey didn't initially like it). Check 0002, which >> gets >> rid of "bool

Re: Maximum password length

2020-09-01 Thread Tom Lane
I wrote: > Peter Eisentraut writes: >> ISTM that it's only going to be a matter of time before that will be >> exceeded. Why have a limit at all? Accept whatever StringInfo accepts. > Hmm, that would require some refactoring of simple_prompt for starters. To use StringInfo, we have to move

Re: Dependencies for partitioned indexes are still a mess

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-12, Alvaro Herrera wrote: > On 2020-Jul-15, Tom Lane wrote: > > (There seem to be some other problems as well, but most of the 54 complaints > > are related to partitioned indexes/constraints.) > > In my run of it there's a good dozen remaining problems, all alike: we > do DROP TYPE

Re: Manager for commit fest 2020-09

2020-09-01 Thread Michael Paquier
On Wed, Sep 02, 2020 at 09:28:07AM +0900, Ian Barwick wrote: > 2020-11 is also now showing as "in progress", is that correct? It was not. Thanks! -- Michael signature.asc Description: PGP signature

Re: Manager for commit fest 2020-09

2020-09-01 Thread Ian Barwick
On 2020/09/01 10:43, Michael Paquier wrote: On Mon, Aug 31, 2020 at 04:37:12PM +0900, Michael Paquier wrote: We are going to be in September in a couple of hours, meaning that the second commit fest for Postgres 14 will begin soon. Do we have any volunteers to take the role of CFM this time?

Re: logtape.c stats don't account for unused "prefetched" block numbers

2020-09-01 Thread Peter Geoghegan
On Tue, Sep 1, 2020 at 4:36 PM Alvaro Herrera wrote: > This open item hasn't received any replies. I think Peter knows how to > fix it already, but no patch has been posted ... It'd be good to get a > move on it. I picked this up again today. It's not obvious what we should do. It's true that

Re: REINDEX SCHEMA/DATABASE/SYSTEM weak with dropped relations

2020-09-01 Thread Michael Paquier
On Tue, Sep 01, 2020 at 01:25:27PM +0300, Anastasia Lubennikova wrote: > Yes, this version is good. Thanks. I have added an extra comment for the case of RELKIND_INDEX with REINDEXOPT_MISSING_OK while on it, as it was not really obvious why the parent relation needs to be locked (at least

Re: Manager for commit fest 2020-09

2020-09-01 Thread Michael Paquier
On Tue, Sep 01, 2020 at 10:43:47AM +0900, Michael Paquier wrote: > As of the moment this message is written, 10 hours remain until we are > the 1st of September AoE, where I'll switch the commit fest as > officially in progress. It will not be possible to register new > patches to 2020-09 after

Re: logtape.c stats don't account for unused "prefetched" block numbers

2020-09-01 Thread Alvaro Herrera
On 2020-Jul-30, Peter Geoghegan wrote: > Commit 896ddf9b added prefetching to logtape.c to avoid excessive > fragmentation in the context of hash aggs that spill and have many > batches/tapes. Apparently the preallocation doesn't actually perform > any filesystem operations, so the new mechanism

Re: [HACKERS] Custom compression methods

2020-09-01 Thread Mark Dilger
> On Aug 13, 2020, at 4:48 AM, Dilip Kumar wrote: > > v1-0001: As suggested by Robert, it provides the syntax support for > setting the compression method for a column while creating a table and > adding columns. However, we don't support changing the compression > method for the existing

Re: Problems with the FSM, heap fillfactor, and temporal locality

2020-09-01 Thread Peter Geoghegan
On Wed, Aug 26, 2020 at 1:46 AM John Naylor wrote: > The fact that that logic extends by 20 * numwaiters to get optimal > performance is a red flag that resources aren't being allocated > efficiently. I agree that that's pretty suspicious. > I have an idea to ignore fp_next_slot entirely if we

Re: Group by reordering optimization

2020-09-01 Thread Peter Geoghegan
On Tue, Sep 1, 2020 at 2:09 PM Tomas Vondra wrote: > >* Instead of changing the order directly, now patch creates another patch > >with > > modifier order of clauses. It does so for the normal sort as well as for > > incremental sort. The whole thing is done in two steps: first it finds a > >

Re: Maximum password length

2020-09-01 Thread Tom Lane
Peter Eisentraut writes: > ISTM that it's only going to be a matter of time before that will be > exceeded. Why have a limit at all? Accept whatever StringInfo accepts. Hmm, that would require some refactoring of simple_prompt for starters. I agree there's no hard reason why we have to have

Re: Disk-based hash aggregate's cost model

2020-09-01 Thread Peter Geoghegan
On Tue, Sep 1, 2020 at 2:19 AM Tomas Vondra wrote: > FWIW I suspect some of this difference may be due to logical vs. > physical I/O. iosnoop only tracks physical I/O sent to the device, but > maybe we do much more logical I/O and it simply does not expire from > page cache for the sort. It might

Re: v13: show extended stats target in \d

2020-09-01 Thread Tomas Vondra
On Tue, Sep 01, 2020 at 05:08:25PM -0400, Alvaro Herrera wrote: +1 on fixing this, since the ability to change stats target is new in pg13. On 2020-Aug-31, Justin Pryzby wrote: Maybe it should have a comma, like ", STATISTICS %s"? It does need some separator. Maybe a comma is sufficient,

Re: Disk-based hash aggregate's cost model

2020-09-01 Thread Tomas Vondra
On Tue, Sep 01, 2020 at 12:58:59PM -0700, Jeff Davis wrote: On Tue, 2020-09-01 at 11:19 +0200, Tomas Vondra wrote: Why? I don't think we need to change costing of in-memory HashAgg. My assumption was we'd only tweak startup_cost for cases with spilling by adding something like

Re: Maximum password length

2020-09-01 Thread Peter Eisentraut
On 2020-09-01 02:54, Tom Lane wrote: Therefore, I propose setting this up with a #define symbol in pg_config_manual.h and leaving it at that. Giving documentation in pg_config_manual.h seems sufficient to me. Attached is a revised version of Nathan's patches that does it like that. I set the

Re: Group by reordering optimization

2020-09-01 Thread Tomas Vondra
On Tue, Sep 01, 2020 at 01:15:31PM +0200, Dmitry Dolgov wrote: Hi, Better late than never, to follow up on the original thread [1] I would like to continue the discussion with the another version of the patch for group by reordering optimization. To remind, it's about reordering of group by

Re: v13: show extended stats target in \d

2020-09-01 Thread Alvaro Herrera
+1 on fixing this, since the ability to change stats target is new in pg13. On 2020-Aug-31, Justin Pryzby wrote: > Maybe it should have a comma, like ", STATISTICS %s"? It does need some separator. Maybe a comma is sufficient, but I'm not sure: that will fail when we add cross-relation stats,

Re: Maximum password length

2020-09-01 Thread Bossart, Nathan
On 8/31/20, 5:55 PM, "Tom Lane" wrote: > I set the proposed limit at 1024 bytes, but given that we now know > of use-cases needing up to 800 bytes, maybe there should be a little > more headroom? I don't want to make it enormous, though, seeing that > we're allocating static buffers of that

Re: Disk-based hash aggregate's cost model

2020-09-01 Thread Jeff Davis
On Tue, 2020-09-01 at 11:19 +0200, Tomas Vondra wrote: > Why? I don't think we need to change costing of in-memory HashAgg. My > assumption was we'd only tweak startup_cost for cases with spilling > by > adding something like (cpu_operator_cost * npartitions * ntuples). The code above (the

Re: Maximum password length

2020-09-01 Thread Tom Lane
I wrote: > Note this patch is intended to be applied over my patch at [2], > since it modifies the test case added there. I've now pushed that patch, so the patch in my previous mail should directly apply to HEAD. I'd originally been wondering whether we need to back-patch this patch. But unless

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-27, Robert Haas wrote: > On Wed, Aug 26, 2020 at 7:40 PM Alvaro Herrera > wrote: > > To mark it detached means to set pg_inherits.inhdetached=true. That > > column name is a bit of a misnomer, since that column really means "is > > in the process of being detached"; the pg_inherits

Re: Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread Tom Lane
Bruce Momjian writes: > On Tue, Sep 1, 2020 at 06:14:45AM +, Junfeng Yang wrote: >> Data in file "/tmp/data". >> >> 122,as\.d,adad >> 133,sa dad,adadad > So, you are using comma as the delimiter, but have \. (backslash-period) > as a data value. You need to double-up backslashes in your

Re: Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread Bruce Momjian
On Tue, Sep 1, 2020 at 06:14:45AM +, Junfeng Yang wrote: > Hi hackers, > > As described in the doc https://www.postgresql.org/docs/current/sql-copy.html, > the TEXT format recognizes > backslash-period (\.) as end-of-data marker. > > The example below will raise an error for the line

Re: Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread Bruce Momjian
On Tue, Sep 1, 2020 at 12:05:02PM -0400, Bruce Momjian wrote: > > copy test from '/tmp/data' DELIMITER ','; > > > > An end-of-copy marker corrupt error will be raised. > > > > This requires users to escape the end-of-data marker manually in their data. > > Why we don't have a mechanism to

Re: Reloptions for table access methods

2020-09-01 Thread Jeff Davis
On Tue, 2020-09-01 at 12:20 -0400, Alvaro Herrera wrote: > Hmm, I think that if we're going to do this, we should do it for all > AMs, not just table AMs, since surely index AMs also want extensible > reloptions; and I think that makes the 'validate' mechanism dead code > and so we should remove

Re: autovac issue with large number of tables

2020-09-01 Thread Kasahara Tatsuhito
Hi, On Wed, Aug 12, 2020 at 2:46 AM Tom Lane wrote: > So I think Kasahara-san's point is that the shared memory stats collector > might wipe out those costs, depending on how it's implemented. (I've not > looked at that patch in a long time either, so I don't know how much it'd > cut the

Re: v13: show extended stats target in \d

2020-09-01 Thread Georgios Kokolatos
The following review has been posted through the commitfest application: make installcheck-world: not tested Implements feature: not tested Spec compliant: not tested Documentation:not tested Hi, I will humbly disagree with the current review. I shall refrain from

Re: Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread David G. Johnston
On Tue, Sep 1, 2020 at 9:05 AM Bruce Momjian wrote: > On Tue, Sep 1, 2020 at 06:14:45AM +, Junfeng Yang wrote: > > Hi hackers, > > > > > Data in file "/tmp/data". > > > > 122,as\.d,adad > > 133,sa dad,adadad > > > > Then execute > > > > copy test from '/tmp/data' DELIMITER ','; > > > > An

Re: Kerberos support broken on MSVC builds for Windows x64?

2020-09-01 Thread Stephen Frost
Greetings, * Dave Page (dp...@pgadmin.org) wrote: > Attached is a patch against 12.4 for the build system in case anyone wants > to play (I'll do it properly against the head branch later). I'm guessing > this will work for < 12, as with 12 I'm now getting the following which > looks like it's

Re: Reloptions for table access methods

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-31, Jeff Davis wrote: > fillRelOptions() validates when filling in a struct to make sure there > aren't "leftover" options. It does this using a hard-coded parsing > table that is not extensible. Hmm, I think that if we're going to do this, we should do it for all AMs, not just table

Re: Kerberos support broken on MSVC builds for Windows x64?

2020-09-01 Thread Dave Page
On Tue, Sep 1, 2020 at 4:22 PM Dave Page wrote: > I was experimenting with building with MIT Kerberos support on 64 bit > Windows using MSVC and ran into a number of linker errors along the lines > of: > > "C:\Users\dpage\Downloads\postgresql-12.4\pgsql.sln" (default target) (1) > -> >

Re: Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread Bruce Momjian
On Tue, Sep 1, 2020 at 06:14:45AM +, Junfeng Yang wrote: > Hi hackers, > > As described in the doc https://www.postgresql.org/docs/current/sql-copy.html, > the TEXT format recognizes > backslash-period (\.) as end-of-data marker. > > The example below will raise an error for the line

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Sep-01, Justin Pryzby wrote: > On Tue, Sep 01, 2020 at 11:40:18AM -0400, Alvaro Herrera wrote: > > The advantage of using a parenthesized option list is that you can add > > *further* options without making the new keywords reserved. Of course, > > we already reserve CONCURRENTLY and

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-01 Thread Bruce Momjian
On Tue, Sep 1, 2020 at 01:59:25PM +0900, Kyotaro Horiguchi wrote: > At Mon, 31 Aug 2020 11:34:29 -0400, Bruce Momjian wrote in > > On Mon, Aug 31, 2020 at 05:56:58PM +0900, Kyotaro Horiguchi wrote: > > > Ok, this is that. If we spcify clientcert=no-verify other than for > > > "cert"

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Justin Pryzby
On Tue, Sep 01, 2020 at 11:40:18AM -0400, Alvaro Herrera wrote: > On 2020-Aug-11, Justin Pryzby wrote: > > On Tue, Aug 11, 2020 at 02:39:45PM +0900, Michael Paquier wrote: > > > > The grammar that has been committed was the one that for the most > > > support, so we need to live with that. I

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alvaro Herrera
On 2020-Aug-11, Justin Pryzby wrote: > On Tue, Aug 11, 2020 at 02:39:45PM +0900, Michael Paquier wrote: > > The grammar that has been committed was the one that for the most > > support, so we need to live with that. I wonder if we should simplify > > ReindexStmt and move the "concurrent" flag

Kerberos support broken on MSVC builds for Windows x64?

2020-09-01 Thread Dave Page
I was experimenting with building with MIT Kerberos support on 64 bit Windows using MSVC and ran into a number of linker errors along the lines of: "C:\Users\dpage\Downloads\postgresql-12.4\pgsql.sln" (default target) (1) -> "C:\Users\dpage\Downloads\postgresql-12.4\zic.vcxproj" (default target)

Re: Remove line length restriction in passwordFromFile()

2020-09-01 Thread Tom Lane
Fujii Masao writes: > The patch looks good to me, except the following minor thing. > + if (fgets(buf.data + buf.len, buf.maxlen - buf.len - 1, fp) == > NULL) > IIUC fgets() reads the data with the specified size - 1, so ISTM that -1 of > "buf.maxlen - buf.len - 1" is not necessary.

Re: Docs: inaccurate description about config settings

2020-09-01 Thread Li Japin
On Sep 1, 2020, at 8:20 PM, Ian Lawrence Barwick mailto:barw...@gmail.com>> wrote: 2020年9月1日(火) 19:37 Li Japin mailto:japi...@hotmail.com>>: Hi, hackers When I setup a stream replication I found that the documentation says that promote_trigger_file parameter can only be set in the

Re: New default role- 'pg_read_all_data'

2020-09-01 Thread Georgios Kokolatos
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: not tested Spec compliant: not tested Documentation:not tested Version 2 of the patch, implements a useful feature. Based on the

Re: Online checksums patch - once again

2020-09-01 Thread Daniel Gustafsson
> On 28 Jul 2020, at 04:33, Justin Pryzby wrote: > > On Thu, Jun 25, 2020 at 11:43:00AM +0200, Daniel Gustafsson wrote: >> The attached v19 fixes a few doc issues I had missed. > > + They can also be enabled or disabled at a later timne, either as an > offline > => time Fixed. > + *

Re: clarify "rewritten" in pg_checksums docs

2020-09-01 Thread Daniel Gustafsson
> On 1 Sep 2020, at 15:34, Michael Banck wrote: > Am Dienstag, den 01.09.2020, 15:29 +0200 schrieb Daniel Gustafsson: >> Isn't "modified in-place" a more accurate description of the process? > > AIUI we do rewrite the whole file (block by block, after updating the > page header with the

Re: WIP: WAL prefetch (another approach)

2020-09-01 Thread Tomas Vondra
On Wed, Sep 02, 2020 at 02:05:10AM +1200, Thomas Munro wrote: On Wed, Sep 2, 2020 at 1:14 AM Tomas Vondra wrote: from the archive Ahh, so perhaps that's the key. Maybe. For the record, the commands look like this: archive_command = 'gzip -1 -c %p > /mnt/raid/wal-archive/%f.gz'

Re: WIP: WAL prefetch (another approach)

2020-09-01 Thread Thomas Munro
On Wed, Sep 2, 2020 at 1:14 AM Tomas Vondra wrote: > from the archive Ahh, so perhaps that's the key. > I've tested this applied on 6ca547cf75ef6e922476c51a3fb5e253eef5f1b6, > and the failure seems fairly similar to what I reported before, except > that now it happened right at the very

Re: clarify "rewritten" in pg_checksums docs

2020-09-01 Thread Michael Banck
Hi, Am Dienstag, den 01.09.2020, 15:29 +0200 schrieb Daniel Gustafsson: > > On 1 Sep 2020, at 15:13, Michael Banck wrote: > > the pg_checksums docs mention that "When enabling checksums, every file > > in the cluster is rewritten". > > > > From IRC discussions, "rewritten" seems ambiguous, it

Re: clarify "rewritten" in pg_checksums docs

2020-09-01 Thread Daniel Gustafsson
> On 1 Sep 2020, at 15:13, Michael Banck wrote: > the pg_checksums docs mention that "When enabling checksums, every file > in the cluster is rewritten". > > From IRC discussions, "rewritten" seems ambiguous, it could mean that a > second copy of the file is written and then switched over,

Re: WIP: WAL prefetch (another approach)

2020-09-01 Thread Tomas Vondra
On Thu, Aug 13, 2020 at 06:57:20PM +1200, Thomas Munro wrote: On Thu, Aug 6, 2020 at 10:47 PM Tomas Vondra wrote: On Thu, Aug 06, 2020 at 02:58:44PM +1200, Thomas Munro wrote: >On Tue, Aug 4, 2020 at 3:47 AM Tomas Vondra >> Any luck trying to reproduce thigs? Should I try again and collect

clarify "rewritten" in pg_checksums docs

2020-09-01 Thread Michael Banck
Hi, the pg_checksums docs mention that "When enabling checksums, every file in the cluster is rewritten". >From IRC discussions, "rewritten" seems ambiguous, it could mean that a second copy of the file is written and then switched over, implying increased storage demand during the operation.

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-01 Thread k.jami...@fujitsu.com
On Tuesday, August 18, 2020 3:05 PM (GMT+9), Amit Kapila wrote: > On Fri, Aug 7, 2020 at 9:33 AM Tom Lane wrote: > > > > Amit Kapila writes: > > > On Sat, Aug 1, 2020 at 1:53 AM Andres Freund > wrote: > > >> We could also just use pg_class.relpages. It'll probably mostly be > > >> accurate

Re: Evaluate expression at planning time for two more cases

2020-09-01 Thread Surafel Temesgen
Hi , Thank you for looking into this On Fri, Aug 28, 2020 at 9:48 AM Ashutosh Bapat wrote: > } > else > has_nonconst_input = true; > @@ -3382,7 +3395,47 @@ eval_const_expressions_mutator(Node *node, > > + > +

Re: Docs: inaccurate description about config settings

2020-09-01 Thread Ian Lawrence Barwick
2020年9月1日(火) 19:37 Li Japin : > > Hi, hackers > > When I setup a stream replication I found that the documentation says that > promote_trigger_file > parameter can only be set in the postgresql.conf file or on the server > command line, however, it > can also be put into postgresql.auto.conf. If

Group by reordering optimization

2020-09-01 Thread Dmitry Dolgov
Hi, Better late than never, to follow up on the original thread [1] I would like to continue the discussion with the another version of the patch for group by reordering optimization. To remind, it's about reordering of group by clauses to do sorting more efficiently. The patch is rebased and

Docs: inaccurate description about config settings

2020-09-01 Thread Li Japin
Hi, hackers When I setup a stream replication I found that the documentation says that promote_trigger_file parameter can only be set in the postgresql.conf file or on the server command line, however, it can also be put into postgresql.auto.conf. If I use include to import a new config, it

Re: Reloptions for table access methods

2020-09-01 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, 1 September 2020 09:18, Jeff Davis wrote: > A custom table access method might want to add a new reloption to > control something specific to that table access method. Unfortunately, > if you add a new option of type RELOPT_KIND_HEAP, it will

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Alexey Kondratov
On 2020-09-01 13:12, Justin Pryzby wrote: This patch seems to be missing a call to RelationAssumeNewRelfilenode() in reindex_index(). That's maybe the related to the cause of the crashes I pointed out earlier this year. Alexey's v4 patch changed RelationSetNewRelfilenode() to accept a

Re: Include access method in listTables output

2020-09-01 Thread Georgios
‐‐‐ Original Message ‐‐‐ On Tuesday, 1 September 2020 07:41, Michael Paquier wrote: > On Thu, Aug 20, 2020 at 08:16:19AM +, Georgios wrote: > > > Please find version 7 attached which hopefully addresses the error along > > with a proper > > expansion of the test coverage and

Re: REINDEX SCHEMA/DATABASE/SYSTEM weak with dropped relations

2020-09-01 Thread Anastasia Lubennikova
On 01.09.2020 04:38, Michael Paquier wrote: I have added some extra comments. There is one in ReindexRelationConcurrently() to mention that there should be no extra use of MISSING_OK once the list of indexes is built as session locks are taken where needed. Great, it took me a moment to

[patch] Fix checksum verification in base backups for zero page headers

2020-09-01 Thread Michael Banck
Hi, as a continuation of [1], I've split-off the zero page header case from the last patch, as this one seems less contentious. Michael [1] https://commitfest.postgresql.org/28/2308/ -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email:

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-01 Thread Justin Pryzby
This patch seems to be missing a call to RelationAssumeNewRelfilenode() in reindex_index(). That's maybe the related to the cause of the crashes I pointed out earlier this year. Alexey's v4 patch changed RelationSetNewRelfilenode() to accept a tablespace parameter, but Michael seemed to object

Re: Parallel copy

2020-09-01 Thread Greg Nancarrow
Hi Vignesh, >Can you share with me the script you used to generate the data & the ddl of >the table, so that it will help me check that >scenario you faced the >problem. Unfortunately I can't directly share it (considered company IP), though having said that it's only doing something that is

Re: New statistics for tuning WAL buffer size

2020-09-01 Thread Fujii Masao
On 2020/08/24 21:00, Masahiro Ikeda wrote: On 2020-08-24 20:45, Masahiro Ikeda wrote: Hi, thanks for useful comments. I agree to expose the number of WAL write caused by full of WAL buffers. It's helpful when tuning wal_buffers size. Haribabu separated that number into two fields in his

Re: Switch to multi-inserts for pg_depend

2020-09-01 Thread Daniel Gustafsson
> On 14 Aug 2020, at 20:23, Alvaro Herrera wrote: > The logic to keep track number of used slots used is baroque, though -- that > could use a lot of simplification. What if slot_init was an integer which increments together with the loop variable until max_slots is reached? If so, maybe it

Re: [patch] demote

2020-09-01 Thread Jehan-Guillaume de Rorthais
On Tue, 18 Aug 2020 17:41:31 +0200 Jehan-Guillaume de Rorthais wrote: > Hi, > > Please find in attachment v5 of the patch set rebased on master after various > conflicts. > > Regards, > > On Wed, 5 Aug 2020 00:04:53 +0200 > Jehan-Guillaume de Rorthais wrote: > > > Demote now keeps backends

Re: Disk-based hash aggregate's cost model

2020-09-01 Thread Tomas Vondra
On Mon, Aug 31, 2020 at 11:34:34PM -0700, Jeff Davis wrote: On Sun, 2020-08-30 at 17:03 +0200, Tomas Vondra wrote: So I'm wondering if the hashagg is not ignoring similar non-I/O costs for the spilling case. In particular, the initial section computing startup_cost seems to ignore that we may

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-01 Thread Masahiko Sawada
On Fri, 28 Aug 2020 at 23:39, Robert Haas wrote: > > On Fri, Aug 28, 2020 at 4:07 AM Masahiko Sawada > wrote: > > You've removed the description about executing VACUUM with > > DISABLE_PAGE_SKIPPING option on the target relation after using > > pg_surgery functions from the doc but I guess it’s

Re: Documentation patch for backup manifests in protocol.sgml

2020-09-01 Thread Bernd Helmle
Am Montag, den 31.08.2020, 18:48 -0400 schrieb Bruce Momjian: > > So confirmed. > > > Patch applied through 13. Thanks!

RE: [EXTERNAL] Re: WIP: WAL prefetch (another approach)

2020-09-01 Thread Sait Talha Nisanci
Hi, The WAL size for "SSD, full_page_writes=on" was 36GB. I currently don't have the exact size for the other rows because my test VMs got auto-deleted. I can possibly redo the benchmark to get pg_waldump stats for each row. Best, Talha. -Original Message- From: Stephen Frost Sent:

Is it possible to set end-of-data marker for COPY statement.

2020-09-01 Thread Junfeng Yang
Hi hackers, As described in the doc https://www.postgresql.org/docs/current/sql-copy.html, the TEXT format recognizes backslash-period (\.) as end-of-data marker. The example below will raise an error for the line contains `\.`. CREATE TABLE test ( id int, name text, dep text ) Data in file

Re: Disk-based hash aggregate's cost model

2020-09-01 Thread Jeff Davis
On Sun, 2020-08-30 at 17:03 +0200, Tomas Vondra wrote: > So I'm wondering if the hashagg is not ignoring similar non-I/O costs > for the spilling case. In particular, the initial section computing > startup_cost seems to ignore that we may need to so some of the stuff > repeatedly - for example

Reloptions for table access methods

2020-09-01 Thread Jeff Davis
A custom table access method might want to add a new reloption to control something specific to that table access method. Unfortunately, if you add a new option of type RELOPT_KIND_HEAP, it will immediately fail because of the validation that happens in fillRelOptions(). Right now, heap