Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-06 Thread Noah Misch
On Mon, Apr 06, 2020 at 09:46:31AM +0900, Kyotaro Horiguchi wrote: > At Sat, 4 Apr 2020 15:32:12 -0700, Noah Misch wrote in > > On Sat, Apr 04, 2020 at 06:24:34PM -0400, Tom Lane wrote: > > > Shouldn't the CF entry get closed? > > > > Once the buildfarm is clean for a day, sure. The buildfarm h

Re: 2pc leaks fds

2020-04-06 Thread Antonin Houska
Andres Freund wrote: > > From what I can see, the error is that the code only bothers closing > > WALOpenSegment->seg when switching to a new segment, but we need also > > to close it when finishing the business in XLogReaderFree(). > > Yea, I came to the same conclusion and locally fixed it the

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-06 Thread Amit Kapila
On Mon, Apr 6, 2020 at 11:19 AM Masahiko Sawada wrote: > > The attached patch changes to the above comment and removed the code > that is used to un-support only buffer usage accumulation. > So, IIUC, the purpose of this patch will be to count the buffer usage due to the heap scan (in heapam_inde

Re: 2pc leaks fds

2020-04-06 Thread Andres Freund
Hi, On 2020-04-06 09:12:32 +0200, Antonin Houska wrote: > Andres Freund wrote: > > > > From what I can see, the error is that the code only bothers closing > > > WALOpenSegment->seg when switching to a new segment, but we need also > > > to close it when finishing the business in XLogReaderFree(

Problems with GSS encryption and SSL in libpq in 12~

2020-04-06 Thread Michael Paquier
Hi all, A quick make check with Postgres 11 and 12 for src/test/ssl/ shows a lot of difference in run time, using the same set of options with SSL and the same compilation flags (OpenSSL 1.1.1f, with debugging and assertions enabled among other things FWIW), with 12 taking up to five minutes to co

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-06 Thread Masahiko Sawada
On Mon, 6 Apr 2020 at 16:16, Amit Kapila wrote: > > On Mon, Apr 6, 2020 at 11:19 AM Masahiko Sawada > wrote: > > > > The attached patch changes to the above comment and removed the code > > that is used to un-support only buffer usage accumulation. > > > > So, IIUC, the purpose of this patch will

RE: Index Skip Scan

2020-04-06 Thread Floris Van Nee
> > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > > > I was just wondering how the distinct will work with the "skip scan" > > > if we have some filter? I mean every time we select the unique row > > > based on the prefix key and that might get rejected by an external > > >

Re: Index Skip Scan

2020-04-06 Thread Dilip Kumar
On Mon, Apr 6, 2020 at 1:14 PM Floris Van Nee wrote: > > > > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > > > > > I was just wondering how the distinct will work with the "skip scan" > > > > if we have some filter? I mean every time we select the unique row > > > > based

Re: Problems with GSS encryption and SSL in libpq in 12~

2020-04-06 Thread Michael Paquier
On Mon, Apr 06, 2020 at 04:25:57PM +0900, Michael Paquier wrote: > It is possible to enforce this flag to false by using > gssencmode=disable, but that's not really user-friendly in my opinion > because nobody is going to remember that for connection strings with > SSL settings so a lot of applicat

Re: WAL usage calculation patch

2020-04-06 Thread Julien Rouhaud
On Mon, Apr 06, 2020 at 08:55:01AM +0530, Amit Kapila wrote: > On Sat, Apr 4, 2020 at 2:50 PM Julien Rouhaud wrote: > > > I have pushed pg_stat_statements and Explain related patches. I am > now looking into (auto)vacuum patch and have few comments. > Thanks! > @@ -614,6 +616,9 @@ heap_vacuum_

Re: d25ea01275 and partitionwise join

2020-04-06 Thread Amit Langote
On Mon, Apr 6, 2020 at 7:29 AM Tom Lane wrote: > Amit Langote writes: > > Okay, I tried that in the updated patch. I didn't try to come up with > > examples that might break it though. > > I looked through this. Thank you. > * I think your logic for building the coalesce combinations was just >

Re: SyncRepLock acquired exclusively in default configuration

2020-04-06 Thread Masahiko Sawada
On Mon, 6 Apr 2020 at 14:04, Andres Freund wrote: > > Hi, > > Due to the change below, when using the default postgres configuration > of ynchronous_commit = on, max_wal_senders = 10, will now acquire a new > exclusive lwlock after writing a commit record. Indeed. > > commit 48c9f4926562278a2fd2

Optimizing RelationFindReplTupleSeq() for CLOBBER_CACHE_ALWAYS

2020-04-06 Thread Noah Misch
When testing commit c6b9204 with CLOBBER_CACHE_ALWAYS, of the 20 hours for check-world, 001_rep_changes.pl took 1.8 hours. At commit 5406513, the test failed at a poll_query_until() timeout[1]. The slow part is the logical replication of "DELETE FROM tab_ins WHERE a > 0", which deletes 100 record

Re: WAL usage calculation patch

2020-04-06 Thread Amit Kapila
On Mon, Apr 6, 2020 at 1:53 PM Julien Rouhaud wrote: > > On Mon, Apr 06, 2020 at 08:55:01AM +0530, Amit Kapila wrote: > > > > Here, we are not displaying Buffers related data, so why do we think > > it is important to display WAL data? I see some point in displaying > > Buffers and WAL data in a

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-06 Thread Amit Kapila
On Mon, Apr 6, 2020 at 12:55 PM Masahiko Sawada wrote: > > On Mon, 6 Apr 2020 at 16:16, Amit Kapila wrote: > > > > On Mon, Apr 6, 2020 at 11:19 AM Masahiko Sawada > > wrote: > > > > > > The attached patch changes to the above comment and removed the code > > > that is used to un-support only buf

Re: WAL usage calculation patch

2020-04-06 Thread Julien Rouhaud
On Mon, Apr 06, 2020 at 02:34:36PM +0530, Amit Kapila wrote: > On Mon, Apr 6, 2020 at 1:53 PM Julien Rouhaud wrote: > > > > On Mon, Apr 06, 2020 at 08:55:01AM +0530, Amit Kapila wrote: > > > > > > Here, we are not displaying Buffers related data, so why do we think > > > it is important to display

[PATCH] optimization of VALGRIND_MEMPOOL_* usage

2020-04-06 Thread Ivan N. Taranov
Propose Optimization of using the family of macros VALGRIND_MEMPOOL_*. How? - add the GetMemoryChunkCapacity method, which returns the size of usable space in the chunk - use GetMemoryChunkCapacity on VALGRIND_MEMPOOL_* calls - call VALGRIND_MEMPOOL_CHANGED only for really changed chunks *

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Kyotaro Horiguchi
At Fri, 3 Apr 2020 20:14:03 -0300, Alvaro Herrera wrote in > So, the more I look at this patch, the less I like the way the slots are > handled. > > * I think it's a mistake to try to do anything in KeepLogSeg itself; > that function is merely in charge of some arithmetic. I propose to > m

PG compilation error with Visual Studio 2015/2017/2019

2020-04-06 Thread davinder singh
Hi All, I am working on “pg_locale compilation error with Visual Studio 2017”, Related threads[1],[2]. We are getting compilation error in static char *IsoLocaleName(const char *winlocname) function in pg_locale.c file. This function is trying to convert the locale name into the Unix style. For ex

[patch] Fix pg_checksums to allow checking of offline base backup directories

2020-04-06 Thread Michael Banck
Hi, Right now, pg_checksums cannot check a base backup directory taken by pg_basebackup: initdb -k data > /dev/null pg_ctl -D data -l logfile start > /dev/null pg_basebackup -D data_backup pg_checksums -D data_backup pg_checksums: error: cluster must be shut down So users need to start and then

Re: Make MemoryContextMemAllocated() more precise

2020-04-06 Thread David Rowley
On Sat, 28 Mar 2020 at 13:21, Jeff Davis wrote: > Attached refactoring patch. There's enough in here that warrants > discussion that I don't think this makes sense for v13 and I'm adding > it to the July commitfest. I had a read over this too. I noted down the following during my pass of it. 1.

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2020-04-06 Thread Ashutosh Bapat
On Fri, 3 Apr 2020 at 20:45, Etsuro Fujita wrote: > Hi, > > On Thu, Apr 2, 2020 at 2:12 AM Ashutosh Bapat > wrote: > > On Thu, 26 Mar 2020 at 00:35, Tomas Vondra > wrote: > >> I've started reviewing the patch a couple days ago. I haven't done any > >> extensive testing, but I do have a bunch of

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2020-04-06 Thread Ashutosh Bapat
Here are some changes suggested on top of v34 as per my previous mail. These are mostly comment changes. On Mon, 6 Apr 2020 at 13:59, Etsuro Fujita wrote: > On Sat, Apr 4, 2020 at 12:15 AM Etsuro Fujita > wrote: > > Attached is the original patch (0001) and one patch (0002) with > > changes inc

Re: backup manifests and contemporaneous buildfarm failures

2020-04-06 Thread Robert Haas
On Sun, Apr 5, 2020 at 4:07 PM Andrew Dunstan wrote: > Do I need to precede those with some recursive chmod commands? +1. > Perhaps > the client should refuse to run if there is still something left after > these. +1 to that, too. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The E

Re: WAL usage calculation patch

2020-04-06 Thread Euler Taveira
On Mon, 6 Apr 2020 at 00:25, Amit Kapila wrote: > > I have pushed pg_stat_statements and Explain related patches. I am > now looking into (auto)vacuum patch and have few comments. > > I wasn't paying much attention to this thread. May I suggest changing wal_num_fpw to wal_fpw? wal_records and wa

Re: adding partitioned tables to publications

2020-04-06 Thread Peter Eisentraut
On 2020-04-03 16:25, Amit Langote wrote: On Fri, Apr 3, 2020 at 6:34 PM Amit Langote wrote: I am checking test coverage at the moment and should have the patches ready by sometime later today. Attached updated patches. Committed 0001 now. I'll work on the rest tomorrow. -- Peter Eisentrau

Re: PostgreSQL 13 Feature Freeze + Release Management Team (RMT)

2020-04-06 Thread Jonathan S. Katz
On 3/30/20 10:18 AM, Jonathan S. Katz wrote: > Hi, > > The Release Management Team (RMT) for the PostgreSQL 13 is assembled and > has determined that the feature freeze date for the PostgreSQL 11 > release will be April 7, 2020. This means that any feature for the > PostgreSQL 13 release **must be

Re: WAL usage calculation patch

2020-04-06 Thread Julien Rouhaud
On Mon, Apr 06, 2020 at 10:12:55AM -0300, Euler Taveira wrote: > On Mon, 6 Apr 2020 at 00:25, Amit Kapila wrote: > > > > > I have pushed pg_stat_statements and Explain related patches. I am > > now looking into (auto)vacuum patch and have few comments. > > > > I wasn't paying much attention to t

Re: Improving connection scalability: GetSnapshotData()

2020-04-06 Thread Andres Freund
Hi, These benchmarks are on my workstation. The larger VM I used in the last round wasn't currently available. HW: 2 x Intel(R) Xeon(R) Gold 5215 (each 10 cores / 20 threads) 192GB Ram. data directory is on a Samsung SSD 970 PRO 1TB A bunch of terminals, emacs, mutt are open while the benchmark

Re: pg_upgrade fails with non-standard ACL

2020-04-06 Thread Anastasia Lubennikova
New version of the patch is attached. I fixed several issues in the check_for_changed_signatures(). Now it passes check without "test_rename_catalog_objects" and fails (generates script) with it. Test script pg_upgrade_ACL_test.sh demonstrates this. The only known issue left is the usage of

Re: d25ea01275 and partitionwise join

2020-04-06 Thread Tom Lane
Amit Langote writes: > On Mon, Apr 6, 2020 at 7:29 AM Tom Lane wrote: >> * I think your logic for building the coalesce combinations was just >> wrong. We need combinations of left-hand inputs with right-hand inputs, >> not left-hand with left-hand or right-hand with right-hand. Also the >> nes

Re: WAL usage calculation patch

2020-04-06 Thread Euler Taveira
On Mon, 6 Apr 2020 at 10:37, Julien Rouhaud wrote: > On Mon, Apr 06, 2020 at 10:12:55AM -0300, Euler Taveira wrote: > > On Mon, 6 Apr 2020 at 00:25, Amit Kapila > wrote: > > > > > > > > I have pushed pg_stat_statements and Explain related patches. I am > > > now looking into (auto)vacuum patch

Re: Issues with replication slots(which created manually) against logical replication

2020-04-06 Thread freebsdjlu
Hi I also met the same issue : WAL stream: ERROR: option "proto_version" = "1" is unknown do you know what is the root cause ? thanks ! -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: where should I stick that backup?

2020-04-06 Thread Stephen Frost
Greetings, * Noah Misch (n...@leadboat.com) wrote: > On Fri, Apr 03, 2020 at 10:19:21AM -0400, Robert Haas wrote: > > What I'm thinking about is: suppose we add an option to pg_basebackup > > with a name like --pipe-output. This would be mutually exclusive with > > -D, but would work at least with

Re: PG compilation error with Visual Studio 2015/2017/2019

2020-04-06 Thread Juan José Santamaría Flecha
On Mon, Apr 6, 2020 at 1:08 PM davinder singh wrote: > > I am working on “pg_locale compilation error with Visual Studio 2017”, > Related threads[1],[2]. > We are getting compilation error in static char *IsoLocaleName(const char > *winlocname) function in pg_locale.c file. This function is tryin

Re: bad wal on replica / incorrect resource manager data checksum in record / zfs

2020-04-06 Thread Alex Malek
On Thu, Apr 2, 2020 at 2:10 PM Andres Freund wrote: > Hi, > > On 2020-02-19 16:35:53 -0500, Alex Malek wrote: > > We are having a reoccurring issue on 2 of our replicas where replication > > stops due to this message: > > "incorrect resource manager data checksum in record at ..." > > Could you s

Re: WAL usage calculation patch

2020-04-06 Thread Peter Eisentraut
I noticed in some of the screenshots that were tweeted that for example in WAL: records=1 bytes=56 there are two spaces between pieces of data. This doesn't match the rest of the EXPLAIN output. Can that be adjusted? -- Peter Eisentraut http://www.2ndQuadrant.com/ Postgre

Re: adding partitioned tables to publications

2020-04-06 Thread Amit Langote
On Mon, Apr 6, 2020 at 10:25 PM Peter Eisentraut wrote: > On 2020-04-03 16:25, Amit Langote wrote: > > On Fri, Apr 3, 2020 at 6:34 PM Amit Langote wrote: > >> I am checking test coverage at the moment and should have the patches > >> ready by sometime later today. > > > > Attached updated patches

Re: where should I stick that backup?

2020-04-06 Thread Robert Haas
On Mon, Apr 6, 2020 at 10:45 AM Stephen Frost wrote: > For my 2c, at least, introducing more shell commands into critical parts > of the system is absolutely the wrong direction to go in. > archive_command continues to be a mess that we refuse to clean up or > even properly document and the projec

Re: d25ea01275 and partitionwise join

2020-04-06 Thread Amit Langote
On Mon, Apr 6, 2020 at 11:09 PM Tom Lane wrote: > Amit Langote writes: > > Oops, I thought I copy-pasted 4-way full join test not this one, but > > evidently didn't. > > Have you got such a query at hand? I wondered whether we shouldn't > use a 4-way rather than 3-way test case; it'd offer more

Re: WAL usage calculation patch

2020-04-06 Thread Justin Pryzby
On Mon, Apr 06, 2020 at 05:01:30PM +0200, Peter Eisentraut wrote: > I noticed in some of the screenshots that were tweeted that for example in > > WAL: records=1 bytes=56 > > there are two spaces between pieces of data. This doesn't match the rest of > the EXPLAIN output. Can that be adju

Re: User Interface for WAL usage data

2020-04-06 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 08:29:31AM +0200, Julien Rouhaud wrote: > > > "full page records" seems to be showing the number of full page > > > images, not the record having full page images. > > > > I am not sure what exactly is a difference but it is the records > > having full page images. Julien c

Re: Index Skip Scan

2020-04-06 Thread Dmitry Dolgov
> On Mon, Apr 6, 2020 at 1:14 PM Floris Van Nee > wrote: > > > > There's a number of ways we can force a 'filter' rather than an > > 'index condition'. Hm, I wasn't aware about this one, thanks for bringing this up. Btw, Floris, I would appreciate if in the future you can make it more visible th

Re: pg_upgrade fails with non-standard ACL

2020-04-06 Thread Anastasia Lubennikova
On 06.04.2020 16:49, Anastasia Lubennikova wrote: New version of the patch is attached. I fixed several issues in the check_for_changed_signatures(). Now it passes check without "test_rename_catalog_objects" and fails (generates script) with it. Test script pg_upgrade_ACL_test.sh demonstrates

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Kyotaro Horiguchi wrote: > > * Andres complained that the "distance" column was not a great value to > > expose (20171106132050.6apzynxrqrzgh...@alap3.anarazel.de). That's > > right: it changes both by the insertion LSN as well as the slot's > > consumption. Maybe we can ex

Re: where should I stick that backup?

2020-04-06 Thread Magnus Hagander
On Mon, Apr 6, 2020 at 4:45 PM Stephen Frost wrote: > > Greetings, > > * Noah Misch (n...@leadboat.com) wrote: > > On Fri, Apr 03, 2020 at 10:19:21AM -0400, Robert Haas wrote: > > > What I'm thinking about is: suppose we add an option to pg_basebackup > > > with a name like --pipe-output. This wou

Using the rr debugging tool to debug Postgres

2020-04-06 Thread Peter Geoghegan
rr is a tool that makes gdb much more useful by supporting recording and replaying of the program being debugged. I highly recommend trying rr if you're somebody that regularly uses gdb to debug Postgres. rr implements a gdbserver under the hood, so it's very easy to start using once you're already

Re: d25ea01275 and partitionwise join

2020-04-06 Thread Tom Lane
Amit Langote writes: > which does succeed in using partitionwise join. Please see attached > delta that applies on your v7 if that is what you'd rather have. I figured these queries were cheap enough that we could afford to run both. With that and some revision of the comments (per attached), I

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-06 Thread Peter Geoghegan
On Mon, Apr 6, 2020 at 2:21 AM Amit Kapila wrote: > AFAIU, it uses heapam_index_build_range_scan but for writing to index, > it doesn't use buffer manager. Right. It doesn't need to use the buffer manager to write to the index, unlike (say) GIN's CREATE INDEX. -- Peter Geoghegan

Re: Don't try fetching future segment of a TLI.

2020-04-06 Thread Fujii Masao
On 2020/03/19 22:22, Pavel Suderevsky wrote: Hi, I've tested patch provided by Kyotaro and do confirm it fixes the issue. The patch looks good to me. Attached is the updated version of the patch. I updated only comments. Barring any objection, I will commit this patch. Any chance it will

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-04-06 Thread Alexey Kondratov
On 2020-04-03 21:27, Justin Pryzby wrote: On Wed, Apr 01, 2020 at 08:08:36AM -0500, Justin Pryzby wrote: Or maybe you'd want me to squish my changes into yours and resend after any review comments ? I didn't hear any feedback, so I've now squished all "parenthesized" and "fix" commits. T

Re: Allow cluster owner to bypass authentication

2020-04-06 Thread David Steele
On 4/5/20 6:15 AM, Peter Eisentraut wrote: On 2020-03-27 15:58, David Steele wrote: Hi Peter, On 12/27/19 3:22 PM, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: I think it'd be great if this behavior could be implemented within the notation, because we could then just set up a

Re: archive recovery fetching wrong segments

2020-04-06 Thread David Steele
Hi Grigory, On 4/5/20 8:02 PM, Grigory Smolkin wrote: Hello, hackers! I`m investigating a complains from our clients about archive recovery speed been very slow, and I`ve noticed a really strange and, I think, a very dangerous recovery behavior. When running multi-timeline archive recovery,

Re: where should I stick that backup?

2020-04-06 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Apr 6, 2020 at 10:45 AM Stephen Frost wrote: > > For my 2c, at least, introducing more shell commands into critical parts > > of the system is absolutely the wrong direction to go in. > > archive_command continues to be a mess that

RE: Index Skip Scan

2020-04-06 Thread Floris Van Nee
> > Hm, I wasn't aware about this one, thanks for bringing this up. Btw, Floris, I > would appreciate if in the future you can make it more visible that changes > you > suggest contain some fixes. E.g. it wasn't clear for me from your previous > email > that that's the case, and it doesn't mak

Re: where should I stick that backup?

2020-04-06 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Mon, Apr 6, 2020 at 4:45 PM Stephen Frost wrote: > > * Noah Misch (n...@leadboat.com) wrote: > > > On Fri, Apr 03, 2020 at 10:19:21AM -0400, Robert Haas wrote: > > > > What I'm thinking about is: suppose we add an option to pg_basebac

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-04-06 Thread Justin Pryzby
On Mon, Apr 06, 2020 at 08:43:46PM +0300, Alexey Kondratov wrote: > Thanks for the input, but I am afraid that the patch set became a bit messy > now. I have eyeballed it and found some inconsistencies. > > const char *name; /* name of database to reindex > */ > -

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2020-04-06 Thread Tom Lane
Noah Misch writes: > On Wed, Mar 25, 2020 at 04:42:31PM -0400, Tom Lane wrote: >> So I think what we're actually trying to accomplish here is to >> ensure that instead of deleting up to half of the SLRU space >> before the cutoff, we delete up to half-less-one-segment. >> Maybe it should be half-l

Re: SyncRepLock acquired exclusively in default configuration

2020-04-06 Thread Ashwin Agrawal
On Mon, Apr 6, 2020 at 1:52 AM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > On Mon, 6 Apr 2020 at 14:04, Andres Freund wrote: > > > > commit 48c9f4926562278a2fd2b85e7486c6d11705f177 > > Author: Simon Riggs > > Date: 2017-12-29 14:30:33 + > > > > Fix race condition when c

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Kyotaro Horiguchi wrote: > At Fri, 3 Apr 2020 20:14:03 -0300, Alvaro Herrera > wrote in > Agreed and thanks for the code. The patch is enough to express the > intention. I fixed some compilation errors and made a clean up of > KeepLogSeg. InvalidateObsoleteReplicationSlots req

Re: Don't try fetching future segment of a TLI.

2020-04-06 Thread David Steele
On 4/6/20 1:43 PM, Fujii Masao wrote: On 2020/03/19 22:22, Pavel Suderevsky wrote: Hi, I've tested patch provided by Kyotaro and do confirm it fixes the issue. The patch looks good to me. Attached is the updated version of the patch. I updated only comments. Barring any objection, I will c

Re: DROP OWNED CASCADE vs Temp tables

2020-04-06 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Jan-14, Alvaro Herrera wrote: >> Hmm, it seems to be doing it differently. Maybe it should be acquiring >> locks on all objects in that nested loop and verified them for >> existence, so that when it calls performMultipleDeletions the objects >> are already locked

Re: archive recovery fetching wrong segments

2020-04-06 Thread Grigory Smolkin
On 4/6/20 9:17 PM, David Steele wrote: Hi Grigory, Hello! On 4/5/20 8:02 PM, Grigory Smolkin wrote: Hello, hackers! I`m investigating a complains from our clients about archive recovery speed been very slow, and I`ve noticed a really strange and, I think, a very dangerous recovery behav

Re: archive recovery fetching wrong segments

2020-04-06 Thread David Steele
On 4/6/20 3:23 PM, Grigory Smolkin wrote: On 4/6/20 9:17 PM, David Steele wrote: Hi Grigory, Hello! On 4/5/20 8:02 PM, Grigory Smolkin wrote: Hello, hackers! I`m investigating a complains from our clients about archive recovery speed been very slow, and I`ve noticed a really strange and,

Re: Online verification of checksums

2020-04-06 Thread Tom Lane
Michael Banck writes: > [ 0001-Fix-checksum-verification-in-base-backups-for-random_V3.patch ] I noticed that the cfbot wasn't testing this because of a minor merge conflict. I rebased it over that, and also readjusted things a little bit to avoid unnecessarily reindenting existing code, in hope

Re: backup manifests and contemporaneous buildfarm failures

2020-04-06 Thread Andrew Dunstan
On 4/6/20 7:53 AM, Robert Haas wrote: > On Sun, Apr 5, 2020 at 4:07 PM Andrew Dunstan > wrote: >> Do I need to precede those with some recursive chmod commands? > +1. > >> Perhaps >> the client should refuse to run if there is still something left after >> these. > +1 to that, too. > See https

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: Hi, I've pushed the fist part of this patch series - I've reorganized it a bit by moving the add_partial_path changes to the end. That way I've been able to add regression test demonstrating impact of the change on plans involving inc

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > Hmmm, I see the buildfarm is not happy about it - a couple of animals > failed, but some succeeded. The failure seems like a simple difference > in explain output, but it's not clear why would it happen (and I've > ran the tests many times but never seen this failure). Did

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 04:14:38PM -0400, Tom Lane wrote: Tomas Vondra writes: Hmmm, I see the buildfarm is not happy about it - a couple of animals failed, but some succeeded. The failure seems like a simple difference in explain output, but it's not clear why would it happen (and I've ran the

Re: Online verification of checksums

2020-04-06 Thread Tom Lane
I wrote: > Another thing that's bothering me is that the patch compares page LSN > against GetInsertRecPtr(); but that function says > ... > I'm not convinced that an approximation is good enough here. It seems > like a page that's just now been updated could have an LSN beyond the > current XLOG

Re: archive recovery fetching wrong segments

2020-04-06 Thread Grigory Smolkin
On 4/6/20 10:51 PM, David Steele wrote: On 4/6/20 3:23 PM, Grigory Smolkin wrote: On 4/6/20 9:17 PM, David Steele wrote: Hi Grigory, Hello! On 4/5/20 8:02 PM, Grigory Smolkin wrote: Hello, hackers! I`m investigating a complains from our clients about archive recovery speed been very s

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > On Mon, Apr 06, 2020 at 04:14:38PM -0400, Tom Lane wrote: >> Did you ever use force_parallel_mode = regress? > Ah, not sure - probably not in this round of tests and there were some > changes in the explain code. Thanks for the hint. Locally, things pass without force_para

Re: Improving connection scalability: GetSnapshotData()

2020-04-06 Thread Andres Freund
Hi, On 2020-04-06 06:39:59 -0700, Andres Freund wrote: > These benchmarks are on my workstation. The larger VM I used in the last > round wasn't currently available. One way to reproduce the problem at smaller connection counts / smaller machines is to take more snapshots. Doesn't fully reproduce

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Tom Lane wrote: > Locally, things pass without force_parallel_mode, but turning it on > produces failures that look similar to rhinoceros's (didn't examine > other BF members). FWIW I looked at the eight failures there were about fifteen minutes ago and they were all identical. I

Re: where should I stick that backup?

2020-04-06 Thread Robert Haas
On Mon, Apr 6, 2020 at 2:23 PM Stephen Frost wrote: > So, instead of talking about 'bzip2 > %f.bz2', and then writing into our > documentation that that's how this feature can be used, what about > proposing something that would actually work reliably with this > interface? Which properly fsync's

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: On 2020-Apr-06, Tom Lane wrote: Locally, things pass without force_parallel_mode, but turning it on produces failures that look similar to rhinoceros's (didn't examine other BF members). FWIW I looked at the eight failures there

ERROR: invalid input syntax for type circle

2020-04-06 Thread David Zhang
Hi, I got an error when I was trying to insert a circle using the syntax (the 3rd one) specified in the latest document. https://www.postgresql.org/docs/current/datatype-geometric.html#DATATYPE-CIRCLE < ( x , y ) , r > ( ( x , y ) , r )   ( x , y ) , r     x , y   , r Here is how to reproduce

Re: SyncRepLock acquired exclusively in default configuration

2020-04-06 Thread Andres Freund
Hi, On 2020-04-06 11:51:06 -0700, Ashwin Agrawal wrote: > On Mon, Apr 6, 2020 at 1:52 AM Masahiko Sawada < > masahiko.saw...@2ndquadrant.com> wrote: > > On Mon, 6 Apr 2020 at 14:04, Andres Freund wrote: > > > I'm really not ok with unneccessarily adding an exclusive lock > > > acquisition to such

Re: Online verification of checksums

2020-04-06 Thread Michael Banck
Hi, Am Montag, den 06.04.2020, 16:45 -0400 schrieb Tom Lane: > I wrote: > > Another thing that's bothering me is that the patch compares page LSN > > against GetInsertRecPtr(); but that function says > > ... > > I'm not convinced that an approximation is good enough here. It seems > > like a page

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >On 2020-Apr-06, Tom Lane wrote: > > > >> Locally, things pass without force_parallel_mode, but turning it on > >> produces failures that look similar to rhinoceros's (didn't ex

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > > >On 2020-Apr-06, Tom Lane wrote: > > > > > >> Locally, things pass without force_parallel_mode, but turning it o

Re: where should I stick that backup?

2020-04-06 Thread Robert Haas
On Mon, Apr 6, 2020 at 1:32 PM Magnus Hagander wrote: > Now, if we were just talking about compression, it would actually be > interesting to implement some sort of "postgres compression API" if > you will, that is implemented by a shared library. This library could > then be used from pg_baseback

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:22 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > > wrote: > > > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > > > >On 2020-Apr-06, Tom Lane wrote: > > > > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: On 2020-Apr-06, Tom Lane wrote: Locally, things pass without force_parallel_mode, but turning it on produces failures that look similar to rhinoceros's (didn't examine

Re: nbtree: assertion failure in _bt_killitems() for posting tuple

2020-04-06 Thread Peter Geoghegan
On Sun, Apr 5, 2020 at 5:15 PM Peter Geoghegan wrote: > I am thinking about pushing a fix along the lines of the attached > patch. This preserves the assertion, while avoiding the check in cases > where it doesn't apply, such as when a dirty snapshot is in use. Pushed. Thanks. -- Peter Geoghega

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >>On 2020-Apr-06, Tom Lane wrote: > >> > >>>Locally, things pass without force_parallel_mode, but turning it on

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > OK, I've pushed a fix - this should make the buildfarm happy again. Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. regards, tom lane

Re: [HACKERS] make async slave to wait for lsn to be replayed

2020-04-06 Thread Kartyshov Ivan
On 2020-04-04 03:14, Alexander Korotkov wrote: I think that now we would be fine with single LSN and single TIMEOUT. In future we may add multiple LSNs/TIMEOUTs or/and support for expressions as LSNs/TIMEOUTs if we figure out it's necessary. I also think it's good to couple waiting for lsn with

Re: Let people set host(no)ssl settings from initdb

2020-04-06 Thread Cary Huang
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Hi I applied the patch "v3-0001-Enable-setting-pg_hba.conf-

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra wrote: On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: >>On 2020-Apr-06, Tom Lane wrote: >> >>>Locally, thin

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Alvaro Herrera wrote: > Lastly, I noticed that we're now changing the slot's restart_lsn to > Invalid without being the slot's owner, which goes counter to what is > said in slot.h: > > * - Individual fields are protected by mutex where only the backend owning > * the slot is au

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: Tomas Vondra writes: OK, I've pushed a fix - this should make the buildfarm happy again. Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. Yeah, I've alread

Re: ERROR: invalid input syntax for type circle

2020-04-06 Thread Tom Lane
David Zhang writes: > I got an error when I was trying to insert a circle using the syntax > (the 3rd one) specified in the latest document. Hm. Presumably, that has never worked, and we've had no complaints to date. I'm halfway inclined to treat it as a documentation bug and remove the claim

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: >> Well, it's *less* unhappy. thorntail is showing that the number of >> workers field is not stable; that will need to be masked. > Yeah, I've already pushed a fix for that. But there seems to be another > failure

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 06:34:04PM -0400, Tom Lane wrote: Tomas Vondra writes: On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. Yeah, I've already pushed a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >> >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Al

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 7:09 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > > wrote: > > >> > > >> On Mon, Apr 06, 2020 at 11:12:32PM +0200,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 07:09:11PM -0400, James Coleman wrote: On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > wrote: >> >> On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wr

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Alvaro Herrera wrote: > I think there's a race condition in this: if we kill a walsender and it > restarts immediately before we (checkpoint) can acquire the slot, we > will wait for it to terminate on its own. Fixing this requires changing > the ReplicationSlotAcquire API so that

Re: Improving connection scalability: GetSnapshotData()

2020-04-06 Thread Andres Freund
Hi, On 2020-04-06 06:39:59 -0700, Andres Freund wrote: > > 3) Plain pgbench read-write (you already did it for sure) > > -s 100 -M prepared -T 700 > > autovacuum=off, fsync on: > clients tps master tps pgxact > 1 474 479 > 16 43564476 > 40

  1   2   >