Re: PG 13 release notes, first draft

2020-05-08 Thread Amit Kapila
On Tue, May 5, 2020 at 8:46 AM Bruce Momjian wrote: > > I have committed the first draft of the PG 13 release notes. You can > see them here: > > https://momjian.us/pgsql_docs/release-13.html > Thanks for the work. I was today going through the release notes and was wondering whether

Re: Back-branch minor release notes are up for review

2020-05-08 Thread Fujii Masao
On 2020/05/09 5:41, Tom Lane wrote: See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ac0e30e0d0fe402fbdb3099fd8b32e4bc6755a6a Thanks for making the release note! As usual, please send any corrections by Sunday + Fix possible undercounting of deleted B-tree

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread Tom Lane
James Coleman writes: > There are always full sort groups before any prefix groups can happen, > so we know (even though the tooling doesn't) that the 2nd test can > never contradict the first. So maybe an assertion enforcing that would be appropriate? Untested, but: - if

Re: Should smgrdounlink() be removed?

2020-05-08 Thread Peter Geoghegan
Fine with me. Peter Geoghegan (Sent from my phone)

Re: Should smgrdounlink() be removed?

2020-05-08 Thread Michael Paquier
On Thu, May 07, 2020 at 09:18:52AM -0700, Peter Geoghegan wrote: > On Thu, May 7, 2020 at 4:33 AM Michael Paquier wrote: >> So this gives the attached. Any thoughts? > > That seems fine. Thanks for the review. If there are no objections, I would like to apply that by tomorrow. So please let

Re: Back-patch is necessary? Re: Don't try fetching future segment of a TLI.

2020-05-08 Thread Fujii Masao
On 2020/05/08 14:23, Fujii Masao wrote: On 2020/05/07 17:57, Amit Kapila wrote: On Thu, May 7, 2020 at 12:13 PM Fujii Masao wrote: On 2020/05/02 20:40, Amit Kapila wrote: I don't see any obvious problem with the changed code but we normally don't backpatch performance improvements.  I

Re: stat() on Windows might cause error if target file is larger than 4GB

2020-05-08 Thread Alvaro Herrera
On 2018-Sep-13, Tom Lane wrote: > What I was vaguely imagining is that win32_port.h could #include > whichever Windows header defines these functions and structs, and > then do > > #define stat __stat64 > > static inline ... __stat64(...) { return _stat64(...); } > > What would need testing is

Re: Include sequence relation support in logical replication

2020-05-08 Thread Andres Freund
Hi, On 2020-05-08 16:32:38 -0700, Cary Huang wrote: > I have added more regression test cases to the sequence replication > patch with emphasis on transactions and rollback per your > suggestions. I find that when a transaction is aborted with rollback, > the decoder plugin will not receive the

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread Tomas Vondra
On Fri, May 08, 2020 at 07:33:03PM -0400, James Coleman wrote: On Fri, May 8, 2020 at 7:20 PM Tomas Vondra wrote: On Fri, May 08, 2020 at 07:25:36PM -0300, Ranier Vilela wrote: >Em sex., 8 de mai. de 2020 às 19:02, Tomas Vondra < >tomas.von...@2ndquadrant.com> escreveu: > >> On Thu, Apr 23,

Re: pendingOps table is not cleared with fsync=off

2020-05-08 Thread Thomas Munro
On Sat, May 9, 2020 at 9:21 AM Heikki Linnakangas wrote: > I noticed that commit 3eb77eba5a changed the logic in > ProcessSyncRequests() (formerly mdsync()) so that if you have fsync=off, > the entries are not removed from the pendingOps hash table. I don't > think that was intended. Perhaps we

Re: pg_restore error message

2020-05-08 Thread Alvaro Herrera
On 2020-May-07, Ranier Vilela wrote: > Can suggest improvements? > > 1. free (398 line) must be pg_free(buf)'; Yeah, there's a lot of frontend code that uses free() instead of pg_free(). There are too many of these that worrying about a single one would not improve things much. I guess we

Re: pg_restore error message

2020-05-08 Thread Alvaro Herrera
On 2020-May-07, Euler Taveira wrote: > While investigating a pg_restore error, I stumbled upon a message that is > not so useful. > > pg_restore: error: could not close data file: No such file or directory > > Which file? File name should be printed too like in the error check for > cfopen_read

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-05-08 Thread James Coleman
On Fri, May 8, 2020 at 7:14 PM Tomas Vondra wrote: > > On Fri, Apr 24, 2020 at 04:35:02PM -0400, James Coleman wrote: > >On Sun, Apr 19, 2020 at 12:14 PM James Coleman wrote: > >> > >> On Wed, Apr 15, 2020 at 2:04 PM James Coleman wrote: > >> > > >> > On Wed, Apr 15, 2020 at 11:02 AM James

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread James Coleman
On Fri, May 8, 2020 at 7:20 PM Tomas Vondra wrote: > > On Fri, May 08, 2020 at 07:25:36PM -0300, Ranier Vilela wrote: > >Em sex., 8 de mai. de 2020 às 19:02, Tomas Vondra < > >tomas.von...@2ndquadrant.com> escreveu: > > > >> On Thu, Apr 23, 2020 at 04:12:34PM -0400, James Coleman wrote: > >> >On

Re: Include sequence relation support in logical replication

2020-05-08 Thread Cary Huang
Hi Craig I have added more regression test cases to the sequence replication patch with emphasis on transactions and rollback per your suggestions. I find that when a transaction is aborted with rollback, the decoder plugin will not receive the change but the sequence value will in fact

Re: JSON output from psql

2020-05-08 Thread Gurjeet Singh
On Fri, May 8, 2020 at 12:10 PM Pavel Stehule wrote: > > > pá 8. 5. 2020 v 21:08 odesílatel Gurjeet Singh napsal: > >> >> On Fri, May 8, 2020 at 12:01 PM Pavel Stehule >> wrote: >> >>> Hi >>> >>> pá 8. 5. 2020 v 20:18 odesílatel Gurjeet Singh >>> napsal: >>> psql currently supports

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread Tomas Vondra
On Fri, May 08, 2020 at 07:25:36PM -0300, Ranier Vilela wrote: Em sex., 8 de mai. de 2020 às 19:02, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: On Thu, Apr 23, 2020 at 04:12:34PM -0400, James Coleman wrote: >On Thu, Apr 23, 2020 at 8:38 AM Ranier Vilela wrote: >> >> Hi, >> >> Per

Re: Incremental sorts and EXEC_FLAG_REWIND

2020-05-08 Thread Tomas Vondra
On Fri, Apr 24, 2020 at 04:35:02PM -0400, James Coleman wrote: On Sun, Apr 19, 2020 at 12:14 PM James Coleman wrote: On Wed, Apr 15, 2020 at 2:04 PM James Coleman wrote: > > On Wed, Apr 15, 2020 at 11:02 AM James Coleman wrote: > > > > On Tue, Apr 14, 2020 at 2:53 AM Michael Paquier wrote:

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread Ranier Vilela
Em sex., 8 de mai. de 2020 às 19:02, Tomas Vondra < tomas.von...@2ndquadrant.com> escreveu: > On Thu, Apr 23, 2020 at 04:12:34PM -0400, James Coleman wrote: > >On Thu, Apr 23, 2020 at 8:38 AM Ranier Vilela > wrote: > >> > >> Hi, > >> > >> Per Coverity. > >> > >> If has 0 full groups, "we don't

Re: [PATCH] Fix division by zero (explain.c)

2020-05-08 Thread Tomas Vondra
On Thu, Apr 23, 2020 at 04:12:34PM -0400, James Coleman wrote: On Thu, Apr 23, 2020 at 8:38 AM Ranier Vilela wrote: Hi, Per Coverity. If has 0 full groups, "we don't need to do anything" and need goes to next. Otherwise a integer division by zero, can raise. comments extracted trom

Re: refactoring basebackup.c

2020-05-08 Thread Andres Freund
Hi, On 2020-05-08 16:53:09 -0400, Robert Haas wrote: > They represent closely-related concepts, so much so that I initially > thought we could get by with just one new abstraction layer. I found > on experimentation that this did not work well, so I split it up into > two and that worked a lot

pendingOps table is not cleared with fsync=off

2020-05-08 Thread Heikki Linnakangas
Hi! I noticed that commit 3eb77eba5a changed the logic in ProcessSyncRequests() (formerly mdsync()) so that if you have fsync=off, the entries are not removed from the pendingOps hash table. I don't think that was intended. I propose the attached patch to move the test for enableFsync so

Re: 2pc leaks fds

2020-05-08 Thread Alvaro Herrera
On 2020-May-08, Kyotaro Horiguchi wrote: > I agree to the direction of this patch. Thanks for the explanation. > The patch looks good to me except the two points below. Thanks! I pushed the patch. I fixed the walsender commentary as you suggested, but I'm still of the opinion that we might

refactoring basebackup.c

2020-05-08 Thread Robert Haas
Hi, I'd like to propose a fairly major refactoring of the server's basebackup.c. The current code isn't horrific or anything, but the base backup mechanism has grown quite a few features over the years and all of the code knows about all of the features. This is going to make it progressively

Back-branch minor release notes are up for review

2020-05-08 Thread Tom Lane
See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ac0e30e0d0fe402fbdb3099fd8b32e4bc6755a6a As usual, please send any corrections by Sunday. regards, tom lane

Re: Improving estimates for TPC-H Q2

2020-05-08 Thread Tomas Vondra
On Fri, May 08, 2020 at 07:58:39AM -0400, Matt Daw wrote: Hi Tomas, there’s an interesting related paper in the April 2020 PVLDB, “Quantifying TPC-H Choke Points and Their Optimizations”: http://www.vldb.org/pvldb/vol13/p1206-dreseler.pdf. Thanks. Seems like an interesting and new paper,

Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h

2020-05-08 Thread Christoph Berg
Re: Tom Lane > But during "make check", that should be executing pg_config from the > the temporary installation, so we should get the right answer no? > > Conversely, in "make installcheck" scenarios, we definitely do want > the value from the installed file, or so I should think. > > Do you

Re: src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h

2020-05-08 Thread Tom Lane
Christoph Berg writes: > I believe check_pg_config as used by src/test/ssl/t/002_scram.pl > shouldn't rely on /usr/include/postgresql/pg_config.h but use the file > from the build tree instead: But during "make check", that should be executing pg_config from the the temporary installation, so we

src/test/perl/TestLib.pm: check_pg_config needs /usr/include/postgresql/pg_config.h

2020-05-08 Thread Christoph Berg
I believe check_pg_config as used by src/test/ssl/t/002_scram.pl shouldn't rely on /usr/include/postgresql/pg_config.h but use the file from the build tree instead: src/test/perl/TestLib.pm: Return the number of matches of the given regular expression within the installation's C.

Re: JSON output from psql

2020-05-08 Thread Pavel Stehule
pá 8. 5. 2020 v 21:08 odesílatel Gurjeet Singh napsal: > > On Fri, May 8, 2020 at 12:01 PM Pavel Stehule > wrote: > >> Hi >> >> pá 8. 5. 2020 v 20:18 odesílatel Gurjeet Singh napsal: >> >>> psql currently supports HTML, CSV, etc output formats. I was >>> wondering if supporting JSON format

Re: JSON output from psql

2020-05-08 Thread Gurjeet Singh
On Fri, May 8, 2020 at 12:01 PM Pavel Stehule wrote: > Hi > > pá 8. 5. 2020 v 20:18 odesílatel Gurjeet Singh napsal: > >> psql currently supports HTML, CSV, etc output formats. I was >> wondering if supporting JSON format was requested or discussed in past. If >> there's desire for this

fill_extraUpdatedCols is done in completely the wrong place

2020-05-08 Thread Tom Lane
I happened to notice $subject while working on the release notes. AFAICS, it is 100% inappropriate for the parser to compute the set of generated columns affected by an UPDATE, because that set could change before execution. It would be really easy to break this for an UPDATE in a stored rule,

Re: JSON output from psql

2020-05-08 Thread Pavel Stehule
Hi pá 8. 5. 2020 v 20:18 odesílatel Gurjeet Singh napsal: > psql currently supports HTML, CSV, etc output formats. I was wondering > if supporting JSON format was requested or discussed in past. If there's > desire for this feature, perhaps we can add it to the TODO list on wiki so >

JSON output from psql

2020-05-08 Thread Gurjeet Singh
psql currently supports HTML, CSV, etc output formats. I was wondering if supporting JSON format was requested or discussed in past. If there's desire for this feature, perhaps we can add it to the TODO list on wiki so someone can pick it up and work on it in future. Best regards, -- Gurjeet

MultiXact\SLRU buffers configuration

2020-05-08 Thread Andrey M. Borodin
Hi, hackers! *** The problem *** I'm investigating some cases of reduced database performance due to MultiXactOffsetLock contention (80% MultiXactOffsetLock, 20% IO DataFileRead). The problem manifested itself during index repack and constraint validation. Both being effectively full table

making update/delete of inheritance trees scale better

2020-05-08 Thread Amit Langote
Here is a sketch for implementing the design that Tom described here: https://www.postgresql.org/message-id/flat/357.1550612935%40sss.pgh.pa.us In short, we would like to have only one plan for ModifyTable to get tuples out of to update/delete, not N for N child result relations as is done

Is it possible to find out write_lsn on standby?

2020-05-08 Thread godjan •
On primary I can execute ’SELECT write_lsn FROM pg_stat_replication;’ and get write_lsn of standby. I didn’t find function like "pg_last_write_lsn()” to get write_lsn on standby. Is it possible?

POC and rebased patch for CSN based snapshots

2020-05-08 Thread Movead Li
Hello hackers, I have read the community mail from 'postgrespro' which the link below ①, a summary for the patch, it generals a CSN by timestamp when a transaction is committed and assigns a special value as CSN for abort transaction, and record them in CSN SLRU file. Now we can judge if a

Re: Improving estimates for TPC-H Q2

2020-05-08 Thread Matt Daw
Hi Tomas, there’s an interesting related paper in the April 2020 PVLDB, “Quantifying TPC-H Choke Points and Their Optimizations”: http://www.vldb.org/pvldb/vol13/p1206-dreseler.pdf. Matt

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-08 Thread godjan •
I got it, thank you. Can you recommend what to use to determine which quorum standby should be promoted in such case? We planned to use pg_last_wal_receive_lsn() to determine which has fresh data but if it returns the beginning of the segment on both replicas we can’t determine which standby

Re: Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-08 Thread Sergei Kornilov
Hello Yes, this is expected. Walreceiver always start streaming from beginning of the wal segment. ./src/backend/replication/walreceiverfuncs.c in RequestXLogStreaming: * We always start at the beginning of the segment. That prevents a broken * segment (i.e., with no records

Re: Dumping/restoring fails on inherited generated column

2020-05-08 Thread Peter Eisentraut
On 2020-05-06 16:29, Peter Eisentraut wrote: On 2020-04-23 08:35, Masahiko Sawada wrote: After investigating this issue, I think that current DDLs regarding inherited tables and generated columns seem not to work fine. Right, there were a number of combinations that were not properly handled.

Re: PG 13 release notes, first draft

2020-05-08 Thread Peter Eisentraut
On 2020-05-05 22:29, Bruce Momjian wrote: a01e1b8b9d Add new part SQL/MDA to information_schema.sql_parts 33e27c3785c5ce8a3264d6af2550ec5adcebc517 2fc2a88e67 Remove obsolete information schema tables Uh, that didn't seem significant. Maybe have one item "modernize information_schema", and

Strange decreasing value of pg_last_wal_receive_lsn()

2020-05-08 Thread godjan •
PG12 Steps to reproduce on 3 nodes cluster with quorum commit. 1. Cut off network on master with everything. 2. Pkill -9 PostgreSQL on each node. 3. Start PostgreSQL on each node. What was strange? I check every second pg_last_wal_replay_lsn() and pg_last_wal_receive_lsn(). All time it was the

COPY, lock release and MVCC

2020-05-08 Thread Laurenz Albe
I happened to notice that COPY TO releases the ACCESS SHARE lock on the table right when the command ends rather than holding it until the end of the transaction: >From backend/commands/copy.c: /* * Close the relation. If reading, we can release the AccessShareLock * we got; if writing, we

Re: Why no "array_sort" function?

2020-05-08 Thread Fabien COELHO
Hello Sergei, Aggregate functions have syntax for ordering: just "select array_agg(i order by i) from " Described here: https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES Great, that's indeed enough for my usage, thanks for the tip! The questions remains,

Postgres default FILLFACTOR value

2020-05-08 Thread Virender Singla
Why Postgres default FILLFACTOR for table is 100 and for Index is 90. Although Oracle is having completely different MVCC architecture, it uses default 90 for table and 100 for Index (exact reverse of Postgres) Postgres blocks needed more spaces for row update compares to Oracle (because Oracle

Re: Why no "array_sort" function?

2020-05-08 Thread Sergei Kornilov
Hello > mostly in an aggregation to show > in a compact way what items were grouped together. Aggregate functions have syntax for ordering: just "select array_agg(i order by i) from " Described here: https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-AGGREGATES regards,

Re: ALTER TABLE ... SET STORAGE does not propagate to indexes

2020-05-08 Thread Peter Eisentraut
On 2020-05-06 16:37, Peter Eisentraut wrote: On 2020-04-22 16:26, Peter Eisentraut wrote: On 2020-04-22 01:56, Alvaro Herrera wrote: I'm surprised that this hasn't applied yet, because: On 2020-Apr-09, Peter Eisentraut wrote: One thing to remember is that the current situation is broken.

Why no "array_sort" function?

2020-05-08 Thread Fabien COELHO
Hello devs, although having arrays is an anathema in a relational world, pg has them, and I find it useful for some queries, mostly in an aggregation to show in a compact way what items were grouped together. There are a few functions available to deal with arrays. Among these functions,

should INSERT SELECT use a BulkInsertState?

2020-05-08 Thread Justin Pryzby
Seems to me it should, at least conditionally. At least if there's a function scan or a relation or .. I mentioned a bit about our use-case here: https://www.postgresql.org/message-id/20200219173742.GA30939%40telsasoft.com => I'd prefer our loaders to write their own data rather than dirtying

Re: PG 13 release notes, first draft

2020-05-08 Thread Fabien COELHO
Hello Tom, Uh, can someone else give an opinion on this? I am not sure how hard or un-fun an item is should be used as criteria. Historically we don't document documentation changes at all, do we? ISTM that the "we did not do it previously" is as weak an argument as un-fun-ness:-)

Re: Implementing Incremental View Maintenance

2020-05-08 Thread Tatsuo Ishii
>> +1, This is a smart idea. How did you test it? AFAIK, we can test it > with: > > 1. For any query like SELECT xxx, we create view like CREATE MATERIAL VIEW > mv_name as SELECT xxx; to test if the features in the query are supported. No I didn't test the correctness of IVM with TPC-DS

Re: Logical replication subscription owner

2020-05-08 Thread Kyotaro Horiguchi
At Fri, 8 May 2020 01:02:11 -0400, Alvaro Herrera wrote in > On 2020-May-07, Tom Lane wrote: > > > FWIW, I would argue that LOGIN permits logging in on a regular SQL > > connection, while REPLICATION should permit logging in on a > > replication connection, and there's no reason for either to