Re: backup manifests

2020-03-30 Thread Noah Misch
On Mon, Mar 30, 2020 at 12:16:31PM -0700, Andres Freund wrote: > On 2020-03-30 15:04:55 -0400, Robert Haas wrote: > > I guess I'd like to be clear here that I have no fundamental > > disagreement with taking this tool in any direction that people would > > like it to go. For me it's just a

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-30 Thread Fabien COELHO
Hello, As I wrote about an earlier version of the patch, ISTM that instead of reinventing, extending, adapting various ls variants (with/without metadata, which show only files, which shows target of links, which shows directory, etc.) we would just need *one* postgres "ls" implementation

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-30 Thread Kyotaro Horiguchi
Thank you for looking this and trouble rebasing! At Mon, 30 Mar 2020 20:03:27 -0300, Alvaro Herrera wrote in > I rebased this patch; it's failing to apply due to minor concurrent > changes in PostgresNode.pm. I squashed the patches in a series that > made the most sense to me. > > I have a

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-30 Thread Masahiko Sawada
On Tue, 31 Mar 2020 at 12:58, Amit Kapila wrote: > > On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada > wrote: > > > > The patch for vacuum conflicts with recent changes in vacuum. So I've > > attached rebased one. > > > > + /* > + * Next, accumulate buffer usage. (This must wait for the

Re: Internal key management system

2020-03-30 Thread Masahiko Sawada
On Tue, 31 Mar 2020 at 09:36, Cary Huang wrote: > > Hi > I had a look on kms_v9 patch and have some comments > > --> pg_upgrade.c > keys are copied correctly, but as pg_upgrade progresses further, it will try > to start the new_cluster from "issue_warnings_and_set_wal_level()" function, > which

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-03-30 Thread Amit Kapila
On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada wrote: > > The patch for vacuum conflicts with recent changes in vacuum. So I've > attached rebased one. > + /* + * Next, accumulate buffer usage. (This must wait for the workers to + * finish, or we might get incomplete data.) + */ + for (i = 0;

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-30 Thread David Rowley
On Sat, 28 Mar 2020 at 22:22, David Rowley wrote: > I'm unsure yet if this has caused an instability on lousyjack's run in > [1]. pogona has just joined in on the fun [1], so, we're not out the woods on this yet. I'll start having a look at this in more detail. [1]

Re: error context for vacuum to include block number

2020-03-30 Thread Justin Pryzby
On Tue, Mar 31, 2020 at 07:50:45AM +0530, Amit Kapila wrote: > On Mon, Mar 30, 2020 at 9:56 PM Justin Pryzby wrote: > > > > On Mon, Mar 30, 2020 at 02:31:53PM +0530, Amit Kapila wrote: > > > The v37-0003-Avoid-some-calls-to-RelationGetRelationName.patch looks > > > good to me. I have added the

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-30 Thread Fujii Masao
On 2020/03/31 3:16, Julien Rouhaud wrote: On Mon, Mar 30, 2020 at 6:36 PM Fujii Masao wrote: On 2020/03/30 17:03, Julien Rouhaud wrote: On Mon, Mar 30, 2020 at 01:56:43PM +0900, Fujii Masao wrote: On 2020/03/29 15:15, Julien Rouhaud wrote: On Fri, Mar 27, 2020 at 03:42:50PM +0100,

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-03-30 Thread Tom Lane
[ not a review, just some drive-by comments on David's comments ] David Rowley writes: > 2. The following change does not seem like it should be part of this > patch. I understand you perhaps have done as you think it will > improve the performance of checking if an expression is in a list of >

Re: improve transparency of bitmap-only heap scans

2020-03-30 Thread Amit Kapila
On Mon, Mar 30, 2020 at 9:59 AM Tom Lane wrote: > > Amit Kapila writes: > > On Sat, Mar 28, 2020 at 8:02 PM James Coleman wrote: > >> I'm curious if Tom's objection is mostly on the grounds that we should > >> be consistent in what's displayed, or that he thinks the information > >> is likely

Re: [PATCH] Opclass parameters

2020-03-30 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 06:05:51PM +0300, Alexander Korotkov wrote: > On Wed, Mar 18, 2020 at 3:28 AM Nikita Glukhov > wrote: > > Attached new version of reordered patches. > > I'm going to push this if no objections. Find attached patch with editorial corrections to docs for this commit.

Re: error context for vacuum to include block number

2020-03-30 Thread Amit Kapila
On Mon, Mar 30, 2020 at 9:56 PM Justin Pryzby wrote: > > On Mon, Mar 30, 2020 at 02:31:53PM +0530, Amit Kapila wrote: > > The v37-0003-Avoid-some-calls-to-RelationGetRelationName.patch looks > > good to me. I have added the commit message in the patch. > > I realized the 0003 patch has an error

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-03-30 Thread David Rowley
On Sun, 29 Mar 2020 at 20:50, Andy Fan wrote: > Some other changes made in the new patch: > 1. Fixed bug for UniqueKey calculation for OUTER join. > 2. Fixed some typo error in comments. > 3. Renamed the field "grantee" as "guarantee". I've had a look over this patch. Thank for you doing

Very outdated comments in heapam_index_build_range_scan.

2020-03-30 Thread Andres Freund
Hi, heapam_index_build_range_scan() has the following, long standing, comment: /* * When dealing with a HOT-chain of updated tuples, we want to index * the values of the live tuple (if any), but index it under the TID * of the

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-03-30 Thread Justin Pryzby
On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julien Rouhaud wrote: > Rebase due to conflict with 3ec20c7091e97. This is failing to apply probably since 4a539a25ebfc48329fd656a95f3c1eb2cda38af3. Could you rebase? (Also, not sure if this can be set as RFC?) -- Justin

Re: pgsql: Improve handling of parameter differences in physical replicatio

2020-03-30 Thread Fujii Masao
On 2020/03/31 3:10, Andres Freund wrote: Hi, On 2020-03-30 19:41:43 +0900, Fujii Masao wrote: On 2020/03/30 16:58, Peter Eisentraut wrote: Improve handling of parameter differences in physical replication When certain parameters are changed on a physical replication primary, this is

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-30 Thread Tomas Vondra
On Mon, Mar 30, 2020 at 06:53:47PM -0400, James Coleman wrote: On Mon, Mar 30, 2020 at 8:24 AM Tomas Vondra wrote: On Sun, Mar 29, 2020 at 10:16:53PM -0400, James Coleman wrote: >On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra > wrote: >> >> Hi, >> >> Attached is a slightly reorganized patch

Re: Internal key management system

2020-03-30 Thread Cary Huang
Hi I had a look on kms_v9 patch and have some comments --> pg_upgrade.c keys are copied correctly, but as pg_upgrade progresses further, it will try to start the new_cluster from "issue_warnings_and_set_wal_level()" function, which is called after key copy. The new cluster will fail to

Re: Corruption during WAL replay

2020-03-30 Thread Andres Freund
Hi, On 2020-03-24 18:18:12 +0900, Kyotaro Horiguchi wrote: > At Mon, 23 Mar 2020 20:56:59 +, Teja Mupparti > wrote in > > The original bug reporting-email and the relevant discussion is here > ... > > The crux of the fix is, in the current code, engine drops the buffer and > > then

Re: backup manifests

2020-03-30 Thread David Steele
On 3/30/20 4:16 PM, Robert Haas wrote: On Fri, Mar 27, 2020 at 3:51 PM David Steele wrote: > { "Path": "backup_label", "Size": 224, "Last-Modified": "2020-03-27 18:33:18 GMT", "Checksum-Algorithm": "CRC32C", "Checksum": "b914bec9" }, Storing the checksum type with each file seems pretty

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-03-30 Thread Alvaro Herrera
I rebased this patch; it's failing to apply due to minor concurrent changes in PostgresNode.pm. I squashed the patches in a series that made the most sense to me. I have a question about static variable lastFoundOldestSeg in FindOldestXLogFileSegNo. It may be set the first time the function

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Tom Lane
Nikita Glukhov writes: > I think now it looks as simple as the whole algorithm is. Yeah, I think we've gotten checkCond to the point of "there's no longer anything to take away". I've marked this RFC, and will push tomorrow unless somebody wants to object to the loss of backwards compatibility.

Re: backup manifests

2020-03-30 Thread David Steele
On 3/30/20 5:08 PM, Andres Freund wrote: The data in the backup label isn't sufficient though. Without having parsed the timeline file there's no way to verify that the correct WAL is present. I guess we can also add client side tools to parse timelines, add command the fetch all of the

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Nikita Glukhov
On 31.03.2020 1:35, Tom Lane wrote: Nikita Glukhov writes: And we even can simply transform this tail call into a loop: -if (tlen > 0 && qlen > 0) +while (tlen > 0 && qlen > 0) Yeah, the same occurred to me ... and then we can drop the other loop too. I think now it looks as simple as the

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Tom Lane
Nikita Glukhov writes: > And we even can simply transform this tail call into a loop: > -if (tlen > 0 && qlen > 0) > +while (tlen > 0 && qlen > 0) Yeah, the same occurred to me ... and then we can drop the other loop too. I've got it down to this now: /* * Try to match an lquery (of qlen

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Nikita Glukhov
On 31.03.2020 1:12, Tom Lane wrote: I wrote: I dunno, that doesn't really seem clearer to me (although some of it might be that you expended no effort on making the comments match the new code logic). ... although looking closer, this formulation does have one very nice advantage: for the

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Tom Lane
I wrote: > I dunno, that doesn't really seem clearer to me (although some of it > might be that you expended no effort on making the comments match > the new code logic). ... although looking closer, this formulation does have one very nice advantage: for the typical non-star case with high = low

Re: [PATCH] remove condition always true (/src/backend/access/heap/vacuumlazy.c)

2020-03-30 Thread Ranier Vilela
Em seg., 30 de mar. de 2020 às 18:14, Andres Freund escreveu: > Hi, > > On 2020-03-30 14:10:29 -0700, Andres Freund wrote: > > On 2020-03-30 17:08:01 -0300, Ranier Vilela wrote: > > > Em seg., 30 de mar. de 2020 às 16:05, Andres Freund < > and...@anarazel.de> > > > escreveu: > > > > > > > Hi, >

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Tom Lane
Nikita Glukhov writes: > On 30.03.2020 21:00, Tom Lane wrote: >> Hence, new patch versions that do it like that. (0002 is unchanged.) > I tried to simplify a bit loops in checkCond() by merging two of them into > one with an explicit exit condition. Also I added return statement after > this

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Nikita Glukhov
On 30.03.2020 21:00, Tom Lane wrote: Hence, new patch versions that do it like that. (0002 is unchanged.) I tried to simplify a bit loops in checkCond() by merging two of them into one with an explicit exit condition. Also I added return statement after this loop, so it's now clear that we

Re: extension patch of CREATE OR REPLACE TRIGGER

2020-03-30 Thread Tom Lane
"osumi.takami...@fujitsu.com" writes: > Also, I'm waiting for other kind of feedbacks from anyone. As David pointed out, this needs to be rebased, though it looks like the conflict is pretty trivial. A few other notes from a quick look: * You missed updating equalfuncs.c/copyfuncs.c. Pretty

Re: backup manifests

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 15:23:08 -0400, Robert Haas wrote: > On Mon, Mar 30, 2020 at 2:59 PM Andres Freund wrote: > > I wonder if it'd not be best, independent of whether we build in this > > verification, to include that metadata in the manifest file. That's for > > sure better than having to build a

Re: Error on failed COMMIT

2020-03-30 Thread Shay Rojansky
Apologies for not responding earlier, busy times. Fourth, it is not clear how many applications would break if COMMIT >> started issuing an error rather than return success a with ROLLBACK tag. >> Certainly SQL scripts would be fine. They would have one additional >> error in the script output,

Re: backup manifests

2020-03-30 Thread Robert Haas
On Sun, Mar 29, 2020 at 9:05 PM David Steele wrote: > Yeah, that seems reasonable. > > In our case backups are nearly always compressed and/or encrypted so > even checking the original size is a bit of work. Getting the checksum > at the same time seems like an obvious win. Makes sense. If this

Re: [PATCH] Atomic pgrename on Windows

2020-03-30 Thread Andres Freund
Hi, On March 30, 2020 9:17:00 AM PDT, Tom Lane wrote: >So far as pg_stat_tmp is concerned, I think there is reasonable hope >that that problem is just going to go away in the near future. >I've not been paying attention to the shared-memory stats collector >thread so I'm not sure if that's

Re: backup manifests

2020-03-30 Thread Robert Haas
On Mon, Mar 30, 2020 at 2:59 PM Andres Freund wrote: > I wonder if it'd not be best, independent of whether we build in this > verification, to include that metadata in the manifest file. That's for > sure better than having to build a separate tool to parse timeline > history files. I don't

Re: backup manifests

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 15:04:55 -0400, Robert Haas wrote: > I guess I'd like to be clear here that I have no fundamental > disagreement with taking this tool in any direction that people would > like it to go. For me it's just a question of timing. Feature freeze > is now a week or so away, and

Re: tweaking perfect hash multipliers

2020-03-30 Thread John Naylor
On Tue, Mar 31, 2020 at 2:31 AM Andres Freund wrote: > > Hi, > > On 2020-03-30 21:33:14 +0800, John Naylor wrote: > > Then I used the attached program to measure various combinations of > > compiled instructions using two constant multipliers iterating over > > bytes similar to a generated hash

Re: [PATCH] remove condition always true (/src/backend/access/heap/vacuumlazy.c)

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 15:07:40 -0300, Ranier Vilela wrote: > I'm not sure that the patch is 100% correct. This is *NOT* correct. > But the fix is about expression about always true. > But if this patch is correct, he fix one possible bug. > > The comment says: > * Perform checking of FSM after

Re: backup manifests

2020-03-30 Thread Robert Haas
On Mon, Mar 30, 2020 at 2:24 AM Amit Kapila wrote: > > Between those two, I would use "pg_validatebackup" if there's a fair chance > > it > > will end up doing the pg_waldump check. Otherwise, I would use > > "pg_validatemanifest". > > +1. I guess I'd like to be clear here that I have no

Re: backup manifests

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 14:35:40 -0400, Robert Haas wrote: > On Sun, Mar 29, 2020 at 10:08 PM Andres Freund wrote: > > See the attached minimal prototype for what I am thinking of. > > > > This would not correctly handle the case where the timeline changes > > while taking a base backup. But I'm not

Re: backup manifests

2020-03-30 Thread Robert Haas
On Sun, Mar 29, 2020 at 10:08 PM Andres Freund wrote: > See the attached minimal prototype for what I am thinking of. > > This would not correctly handle the case where the timeline changes > while taking a base backup. But I'm not sure that'd be all that serious > a limitation for now? > > I'd

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-30 Thread Justin Pryzby
On Mon, Mar 30, 2020 at 09:02:22PM +0300, Alexey Kondratov wrote: > Hmm, I went through the well known to me SQL commands in Postgres and a bit > more. Parenthesized options list is mostly used in two common cases: There's also ANALYZE(VERBOSE), REINDEX(VERBOSE). There was debate a year ago [0]

Re: tweaking perfect hash multipliers

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 21:33:14 +0800, John Naylor wrote: > Then I used the attached program to measure various combinations of > compiled instructions using two constant multipliers iterating over > bytes similar to a generated hash function. It looks like you didn't attach the program? > -O2

Re: Recognizing superuser in pg_hba.conf

2020-03-30 Thread Tom Lane
Tomas Vondra writes: > I see this patch is marked as RFC since 12/30, but there seems to be > quite a lot of discussion about the syntax, keywords and how exactly to > identify the superuser. So I'll switch it back to needs review, which I > think is a better representation of the current state.

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-30 Thread Julien Rouhaud
On Mon, Mar 30, 2020 at 6:36 PM Fujii Masao wrote: > > On 2020/03/30 17:03, Julien Rouhaud wrote: > > On Mon, Mar 30, 2020 at 01:56:43PM +0900, Fujii Masao wrote: > >> > >> > >> On 2020/03/29 15:15, Julien Rouhaud wrote: > >>> On Fri, Mar 27, 2020 at 03:42:50PM +0100, Julien Rouhaud wrote: >

Re: pgsql: Improve handling of parameter differences in physical replicatio

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 19:41:43 +0900, Fujii Masao wrote: > On 2020/03/30 16:58, Peter Eisentraut wrote: > > Improve handling of parameter differences in physical replication > > > > When certain parameters are changed on a physical replication primary, > > this is communicated to standbys using the

[PATCH] remove condition always true (/src/backend/access/heap/vacuumlazy.c)

2020-03-30 Thread Ranier Vilela
Hi, I'm not sure that the patch is 100% correct. But the fix is about expression about always true. But if this patch is correct, he fix one possible bug. The comment says: * Perform checking of FSM after releasing lock, the fsm is * approximate, after all. But this is not what the code does,

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-30 Thread Alexey Kondratov
On 2020-03-28 03:11, Justin Pryzby wrote: On Thu, Mar 26, 2020 at 11:01:06PM -0500, Justin Pryzby wrote: > Another issue is this: > > +VACUUM ( FULL [, ...] ) [ TABLESPACE new_tablespace ] [ table_and_columns [, ...] ] > As you mentioned in your v1 patch, in the other cases, "tablespace >

Re: fix for BUG #3720: wrong results at using ltree

2020-03-30 Thread Tom Lane
I wrote: > Hence, attached are two revised patches that attack the problem > this way. The first one is somewhat unrelated to the original > point --- it's trying to clean up the error messages in ltree_in > and lquery_in so that they are more consistent and agree with > the terminology used in

Re: snapper vs. HEAD

2020-03-30 Thread Tom Lane
"Tom Turelinckx" writes: > Tom Lane wrote: >> Thanks! But it doesn't seem to have taken: snapper just did a new run >> that still failed, and it still seems to be using -O2. > Snapper did build using -O1 a few hours ago, but it failed the check stage > very early with a different error: >

Re: snapper vs. HEAD

2020-03-30 Thread Tom Lane
Andres Freund writes: > On 2020-03-30 12:24:06 -0400, Tom Lane wrote: >> (As of this weekend, it seemed to be impossible to find the wheezy sparc >> distribution images on-line anymore. > The installer downloads are still available at: > https://www.debian.org/releases/wheezy/debian-installer/

Re: snapper vs. HEAD

2020-03-30 Thread Tom Turelinckx
Hi, Tom Lane wrote: > Thanks! But it doesn't seem to have taken: snapper just did a new run > that still failed, and it still seems to be using -O2. Snapper did build using -O1 a few hours ago, but it failed the check stage very early with a different error: FATAL: null value in column

Re: snapper vs. HEAD

2020-03-30 Thread Andres Freund
Hi, On 2020-03-30 12:24:06 -0400, Tom Lane wrote: > "Tom Turelinckx" writes: > Yeah, I've got a couple of those myself. But perhaps it'd be sensible > to move to a newer Debian LTS release? Or have they dropped Sparc > support altogether? I think the 32bit sparc support has been phased out.

Re: materialization blocks hash join

2020-03-30 Thread Tom Lane
Tomas Vondra writes: > That's because eqjoinsel_inner won't have any statistics for either side > of the join, so it'll use default ndistinct values (200), resulting in > estimate of 0.5% for the join condition. Right. > But this should not affect the choice of join algorithm, I think, >

Re: Add A Glossary

2020-03-30 Thread Corey Huinker
On Sun, Mar 29, 2020 at 5:29 AM Jürgen Purtz wrote: > On 27.03.20 21:12, Justin Pryzby wrote: > > On Fri, Mar 20, 2020 at 11:32:25PM +0100, Jürgen Purtz wrote: > +Archiver > >>> Can you change that to archiver process ? > >> I prefer the short term without the addition of 'process' -

Re: ECPG: proposal for new DECLARE STATEMENT

2020-03-30 Thread David Steele
On 1/11/20 10:41 PM, Tomas Vondra wrote: On Sun, Jan 12, 2020 at 03:52:48AM +0100, Tomas Vondra wrote: ... I'm not an ecpg expert (in fact I've never even used it), so my review is pretty superficial, but I only found a couple of minor whitespace issues (adding/removing a line/tab) - see the

Re: materialization blocks hash join

2020-03-30 Thread Tomas Vondra
On Mon, Mar 30, 2020 at 06:14:42PM +0200, Pavel Stehule wrote: po 30. 3. 2020 v 18:06 odesílatel Pavel Stehule napsal: Hi when I was in talk with Silvio Moioli, I found strange hash join. Hash was created from bigger table.

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-03-30 Thread Fujii Masao
On 2020/03/30 17:03, Julien Rouhaud wrote: On Mon, Mar 30, 2020 at 01:56:43PM +0900, Fujii Masao wrote: On 2020/03/29 15:15, Julien Rouhaud wrote: On Fri, Mar 27, 2020 at 03:42:50PM +0100, Julien Rouhaud wrote: On Fri, Mar 27, 2020 at 2:01 PM Fujii Masao wrote: So what I'd like to

Re: Error on failed COMMIT

2020-03-30 Thread Vik Fearing
On 3/30/20 6:05 PM, Dave Cramer wrote: > So it appears this is currently languishing as unresolved and feature > freeze is imminent. > > What has to be done to get a decision one way or another before feature > freeze. > > I have provided a patch that could be reviewed and at least be considered

Re: error context for vacuum to include block number

2020-03-30 Thread Justin Pryzby
On Mon, Mar 30, 2020 at 02:31:53PM +0530, Amit Kapila wrote: > Now that the main patch is committed, I have reviewed the other two patches. Thanks for that On Mon, Mar 30, 2020 at 02:31:53PM +0530, Amit Kapila wrote: > The v37-0003-Avoid-some-calls-to-RelationGetRelationName.patch looks > good

Re: snapper vs. HEAD

2020-03-30 Thread Tom Lane
"Tom Turelinckx" writes: > In the past, I've already switched from gcc 4.6 to gcc 4.7 as a workaround > for a similar compiler bug, but I can't upgrade to a newer gcc without > backporting it myself, so for the moment I've switched snapper to use -O1 > instead of -O2, for HEAD only. Thanks!

Re: ALTER TABLE ... SET STORAGE does not propagate to indexes

2020-03-30 Thread Adam Brusselback
> ALTER TABLE ... SET STORAGE does not propagate to indexes, even though > indexes created afterwards get the new storage setting. So depending on > the order of commands, you can get inconsistent storage settings between > indexes and tables. I've absolutely noticed this behavior, I just

Re: Make autovacuum sort tables in descending order of xid_age

2020-03-30 Thread Mark Dilger
> On Mar 30, 2020, at 7:09 AM, David Steele wrote: > > On 1/11/20 12:53 PM, David Fetter wrote: >> I agree that it's a complex situation, and that many different >> approaches will eventually need to be brought to bear. >> What concerns me about introducing a big lump of complexity here is >>

Re: ALTER TABLE ... SET STORAGE does not propagate to indexes

2020-03-30 Thread Alvaro Herrera
On 2020-Feb-25, Peter Eisentraut wrote: > An alternative would be that we make this situation fully supported. Then > we'd probably need at least ALTER INDEX ... ALTER COLUMN ... SET STORAGE, > and some pg_dump support. I think this is a more promising direction. -- Álvaro Herrera

Re: [PATCH] Atomic pgrename on Windows

2020-03-30 Thread Tom Lane
David Steele writes: > On 1/11/20 5:13 PM, Alexander Korotkov wrote: >> Regarding "pg_stat_tmp/global.stat", which is a problem in particular >> case, we may evade file renaming altogether. Instead, we may >> implement shared-memory counter for filename. So, instead of >> renaming, new reads

Re: materialization blocks hash join

2020-03-30 Thread Pavel Stehule
po 30. 3. 2020 v 18:06 odesílatel Pavel Stehule napsal: > Hi > > when I was in talk with Silvio Moioli, I found strange hash join. Hash was > created from bigger table. > > > https://www.postgresql.org/message-id/79dd683d-3296-1b21-ab4a-28fdc2d98807%40suse.de > > Now it looks so materialized CTE

materialization blocks hash join

2020-03-30 Thread Pavel Stehule
Hi when I was in talk with Silvio Moioli, I found strange hash join. Hash was created from bigger table. https://www.postgresql.org/message-id/79dd683d-3296-1b21-ab4a-28fdc2d98807%40suse.de Now it looks so materialized CTE disallow hash create table bigger(a int); create table smaller(a int);

Re: Error on failed COMMIT

2020-03-30 Thread Dave Cramer
On Tue, 17 Mar 2020 at 19:32, Dave Cramer wrote: > > > On Tue, 17 Mar 2020 at 19:23, Bruce Momjian wrote: > >> On Tue, Mar 17, 2020 at 07:15:05PM -0400, Dave Cramer wrote: >> > On Tue, 17 Mar 2020 at 16:47, Bruce Momjian wrote: >> > Third, the idea that individual interfaces, e.g. JDBC,

Re: [PATCH] Atomic pgrename on Windows

2020-03-30 Thread David Steele
On 1/11/20 5:13 PM, Alexander Korotkov wrote: On Tue, Jan 7, 2020 at 11:04 PM Tom Lane wrote: "If the link named by the new argument exists and the file's link count becomes 0 when it is removed and no process has the file open, the space occupied by the file shall be freed and the file shall

Re: Unix-domain socket support on Windows

2020-03-30 Thread Peter Eisentraut
On 2020-03-27 18:52, Andrew Dunstan wrote: I have tested this on drongo/fairywren and it works fine. The patches apply cleanly (with a bit of fuzz) and a full buildfarm run is happy in both cases. Unfortunately I don't have a Windows machine that's young enough to support git master and old

Re: adding partitioned tables to publications

2020-03-30 Thread Amit Langote
I have updated the comments in apply_handle_tuple_routing() (see 0002) to better explain what's going on with UPDATE handling. I also rearranged the tests a bit for clarity. Attached updated patches. -- Thank you, Amit Langote EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-03-30 Thread Tomas Vondra
On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote: On Sun, Mar 29, 2020 at 9:01 PM Tomas Vondra wrote: On Sun, Mar 29, 2020 at 11:19:21AM +0530, Amit Kapila wrote: >On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra > wrote: >> >> Ummm, how is that different from what the patch is doing

Re: Can we get rid of GetLocaleInfoEx() yet?

2020-03-30 Thread Tom Lane
Michael Paquier writes: > On Sun, Mar 29, 2020 at 07:06:56PM +0200, Juan José Santamaría Flecha wrote: >> It works for the issue just fine, and more comments make a better a >> patch, so no objections from me. > +1 from me. And yes, we are still missing lc_codepage in newer > versions of VS.

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-30 Thread Tom Lane
Fabien COELHO writes: > As I wrote about an earlier version of the patch, ISTM that instead of > reinventing, extending, adapting various ls variants (with/without > metadata, which show only files, which shows target of links, which shows > directory, etc.) we would just need *one* postgres

[PATCH] Remove some reassigned values.

2020-03-30 Thread Ranier Vilela
Hi, This patch remove reassigned values, with safety. Plancat.c, needs a more careful review. Best regards Ranier Vilela remove_reassigned_values.patch Description: Binary data

PostgreSQL 13 Feature Freeze + Release Management Team (RMT)

2020-03-30 Thread Jonathan S. Katz
Hi, The Release Management Team (RMT) for the PostgreSQL 13 is assembled and has determined that the feature freeze date for the PostgreSQL 11 release will be April 7, 2020. This means that any feature for the PostgreSQL 13 release **must be committed by April 7, 2020 AOE** ("anywhere on

Re: Make autovacuum sort tables in descending order of xid_age

2020-03-30 Thread David Steele
On 1/11/20 12:53 PM, David Fetter wrote: I agree that it's a complex situation, and that many different approaches will eventually need to be brought to bear. What concerns me about introducing a big lump of complexity here is disentangling the effects of each part and of their interaction

Re: Improving connection scalability: GetSnapshotData()

2020-03-30 Thread Alexander Korotkov
On Sun, Mar 29, 2020 at 9:50 PM Andres Freund wrote: > On March 29, 2020 11:24:32 AM PDT, Alexander Korotkov > wrote: > > clearly a big win on majority > >of workloads, I think we still need to investigate different workloads > >on different hardware to ensure there is no regression. > >

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2020-03-30 Thread Alexey Kondratov
On 2020-03-30 04:56, Michael Paquier wrote: On Fri, Mar 27, 2020 at 07:47:29AM +0900, Michael Paquier wrote: On Thu, Mar 26, 2020 at 06:56:36PM -0300, Alvaro Herrera wrote: I don't think any such cleanup should hamper the patch at hand anyway. I don't think either, so I would do the split

Re: Conflict handling for COPY FROM

2020-03-30 Thread Surafel Temesgen
On Fri, Mar 27, 2020 at 3:27 PM Tom Lane wrote: > Surafel Temesgen writes: > > [ conflict-handling-copy-from-v16.patch ] > > I took a quick look at this patch, since it was marked "ready for > committer", but I don't see how it can possibly be considered committable. > > 1. Covering only the

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2020-03-30 Thread Alexander Korotkov
On Mon, Mar 30, 2020 at 4:57 AM Michael Paquier wrote: > On Fri, Mar 27, 2020 at 07:47:29AM +0900, Michael Paquier wrote: > > On Thu, Mar 26, 2020 at 06:56:36PM -0300, Alvaro Herrera wrote: > >> I don't think any such cleanup should hamper the patch at hand anyway. > > > > I don't think either,

tweaking perfect hash multipliers

2020-03-30 Thread John Naylor
Hi all, While playing around with Peter E.'s unicode normalization patch [1], I found that HEAD failed to build a perfect hash function for any of the four sets of 4-byte keys ranging from 1k to 17k in number. It probably doesn't help that codepoints have nul bytes and often cluster into

Re: WIP/PoC for parallel backup

2020-03-30 Thread Ahsan Hadi
On Mon, Mar 30, 2020 at 3:44 PM Rajkumar Raghuwanshi < rajkumar.raghuwan...@enterprisedb.com> wrote: > Thanks Asif, > > I have re-verified reported issue. expect standby backup, others are fixed. > Yes As Asif mentioned he is working on the standby issue and adding bandwidth throttling

Re: WAL usage calculation patch

2020-03-30 Thread Julien Rouhaud
On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote: > On Sun, Mar 29, 2020 at 5:49 PM Julien Rouhaud wrote: > > > > @@ -1249,6 +1250,16 @@ XLogInsertRecord(XLogRecData *rdata, > ProcLastRecPtr = StartPos; > XactLastRecEnd = EndPos; > > + /* Provide WAL update data to the

Re: Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-30 Thread Ranier Vilela
Em seg., 30 de mar. de 2020 às 06:06, Magnus Hagander escreveu: > On Sat, Mar 28, 2020 at 11:49 AM Ranier Vilela > wrote: > > > > Em sex., 27 de mar. de 2020 às 20:49, Tom Lane > escreveu: > >> > >> Ranier Vilela writes: > >> > Can someone check if there is a copy and paste error, at file: >

Re: truncating timestamps on arbitrary intervals

2020-03-30 Thread John Naylor
I wrote: > I'm going to look into implementing timezone while awaiting comments on v6. I attempted this in the attached v7. There are 4 new functions for truncating timestamptz on an interval -- with and without origin, and with and without time zone. Parts of it are hackish, and need more

Re: Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-30 Thread Ranier Vilela
Em seg., 30 de mar. de 2020 às 05:16, Michael Paquier escreveu: > On Sat, Mar 28, 2020 at 07:48:22AM -0300, Ranier Vilela wrote: > > I completely disagree. My tools have proven their worth, including > finding > > serious errors in the code, which fortunately have been fixed by other > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-30 Thread Tomas Vondra
On Sun, Mar 29, 2020 at 10:16:53PM -0400, James Coleman wrote: On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra wrote: Hi, Attached is a slightly reorganized patch series. I've merged the fixes into the appropriate matches, and I've also combined the two patches adding incremental sort paths to

Re: PostgreSQL proposal of Google Summer of Code

2020-03-30 Thread Fabrízio de Royes Mello
On Sun, Mar 29, 2020 at 4:48 PM Maryam Farrukh wrote: > > Hi Fabrizio, > Hi Maryam, please try to avoid top posting!! Returning the discussion to pgsql-hackers. > Thank you for reaching out. I have a question. I went through the links you provided me. There > are already some project ideas

Re: [PATCH] Redudant initilization

2020-03-30 Thread Ranier Vilela
Em dom., 29 de mar. de 2020 às 21:57, Kyotaro Horiguchi < horikyota@gmail.com> escreveu: > Hello. > > At Sat, 28 Mar 2020 19:04:00 -0300, Ranier Vilela > wrote in > > Hi, > > This patch fixes some redundant initilization, that are safe to remove. > > > diff --git

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-03-30 Thread k.jami...@fujitsu.com
On Wednesday, March 25, 2020 3:25 PM, Kirk Jamison wrote: > As for the performance and how it affects the read-only workloads. > Using pgbench, I've kept the overload to a minimum, less than 1%. > I'll post follow-up results. Here's the follow-up results. I executed the similar tests from top of

Re: Some problems of recovery conflict wait events

2020-03-30 Thread Masahiko Sawada
On Fri, 27 Mar 2020 at 17:54, Fujii Masao wrote: > > > > On 2020/03/04 14:31, Masahiko Sawada wrote: > > On Wed, 4 Mar 2020 at 13:48, Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/03/04 13:27, Michael Paquier wrote: > >>> On Wed, Mar 04, 2020 at 01:13:19PM +0900, Masahiko Sawada wrote: >

Re: WIP/PoC for parallel backup

2020-03-30 Thread Rajkumar Raghuwanshi
Thanks Asif, I have re-verified reported issue. expect standby backup, others are fixed. Thanks & Regards, Rajkumar Raghuwanshi On Fri, Mar 27, 2020 at 11:04 PM Asif Rehman wrote: > > > On Wed, Mar 25, 2020 at 12:22 PM Rajkumar Raghuwanshi < > rajkumar.raghuwan...@enterprisedb.com> wrote: >

Re: WAL usage calculation patch

2020-03-30 Thread Amit Kapila
On Sun, Mar 29, 2020 at 5:49 PM Julien Rouhaud wrote: > @@ -1249,6 +1250,16 @@ XLogInsertRecord(XLogRecData *rdata, ProcLastRecPtr = StartPos; XactLastRecEnd = EndPos; + /* Provide WAL update data to the instrumentation */ + if (inserted) + { + pgWalUsage.wal_bytes += rechdr->xl_tot_len; +

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-30 Thread David Rowley
On Mon, 30 Mar 2020 at 19:49, David Rowley wrote: > I'll see if I can come up with some way to do this in a more > deterministic way to determine which tables to add vacuums for, rather > than waiting for and reacting post-failure. I ended up running make installcheck on an instance with

Re: snapper vs. HEAD

2020-03-30 Thread Tom Turelinckx
Hi, Tom Lane wrote: > Confirmed: building gistget with --enable-cassert, and all of snapper's > compile/link options, produces something that passes regression. Skate uses buildfarm default configuration, whereas snapper uses settings which are used when building postgresql debian packages.

Re: Possible copy and past error? (\usr\backend\commands\analyze.c)

2020-03-30 Thread Magnus Hagander
On Sat, Mar 28, 2020 at 11:49 AM Ranier Vilela wrote: > > Em sex., 27 de mar. de 2020 às 20:49, Tom Lane escreveu: >> >> Ranier Vilela writes: >> > Can someone check if there is a copy and paste error, at file: >> > \usr\backend\commands\analyze.c, at lines 2225 and 2226? >> > int num_mcv =

Re: error context for vacuum to include block number

2020-03-30 Thread Amit Kapila
On Fri, Mar 27, 2020 at 5:03 AM Justin Pryzby wrote: > Now that the main patch is committed, I have reviewed the other two patches. v37-0002-Drop-reltuples 1. @@ -2289,11 +2289,10 @@ vacuum_one_index(Relation indrel, IndexBulkDeleteResult **stats, /* Do vacuum or cleanup of the index */ if

  1   2   >