Re: A really subtle lexer bug

2018-08-20 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> Patch attached. Andrew> This fixes two bugs: I'm going to split this into two patches, since the O(N^3) fix can be backpatched further than the operator token fix; the latter bug was introduced in 9.5 when operator precedences were corrected,

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-08-20 Thread David Rowley
On 3 August 2018 at 17:58, Amit Langote wrote: > On 2018/07/31 16:03, David Rowley wrote: >> Maybe we can do that as a follow-on patch. > > We probably could, but I think it would be a good idea get rid of *all* > redundant allocations due to tuple routing in one patch, if that's the > mission of

Re: Fix help option of contrib/oid2name

2018-08-20 Thread Tatsuro Yamada
On 2018/08/21 12:40, Michael Paquier wrote: On Tue, Aug 21, 2018 at 12:26:15PM +0900, Tatsuro Yamada wrote: BTW, can I add the patch to the Commitfest September? You should. Thanks, I'll do that. The patch includes improvements and bug fix as you know, So, I can divide the patch into 2

Re: Fix help option of contrib/oid2name

2018-08-20 Thread Michael Paquier
On Tue, Aug 21, 2018 at 12:26:15PM +0900, Tatsuro Yamada wrote: > BTW, can I add the patch to the Commitfest September? You should. > The patch includes improvements and bug fix as you know, So, I can divide the > patch into 2 patches for that. I am not really seeing any bug fix, but if you

Re: Fix help option of contrib/oid2name

2018-08-20 Thread Tatsuro Yamada
On 2018/08/20 17:38, Michael Paquier wrote: On Mon, Aug 20, 2018 at 03:51:07PM +0900, Tatsuro Yamada wrote: On 2018/08/20 13:54, Michael Paquier wrote: Therefore, "-P" is a manual bag. I investigated more using git log command and understood followings: 1. -P option was removed on 4192f2d85

Re: A typo in guc.c

2018-08-20 Thread Michael Paquier
On Tue, Aug 21, 2018 at 11:58:41AM +0900, Kyotaro HORIGUCHI wrote: > The "user" should be "use". > > As you(who?) know, this applies only 11 and dev. Thanks, applied. -- Michael signature.asc Description: PGP signature

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-08-20 Thread Thomas Munro
On Wed, Jul 25, 2018 at 7:27 PM, Daniel Gustafsson wrote: >> On 24 Jul 2018, at 22:57, Daniel Gustafsson wrote: >> >>> On 6 Jul 2018, at 02:18, Thomas Munro wrote: >>> >>> On Fri, Jun 15, 2018 at 9:56 AM, Daniel Gustafsson wrote: attached >>> >>> Hi Daniel, >>> >>> 6118 --select

A typo in guc.c

2018-08-20 Thread Kyotaro HORIGUCHI
Hello, I found a typo in guc.c. > {"enable_parallel_hash", PGC_USERSET, QUERY_TUNING_METHOD, > - gettext_noop("Enables the planner's user of parallel hash plans."), > + gettext_noop("Enables the planner's use of parallel hash plans."), The "user" should be "use". As you(who?)

Re: NLS handling fixes.

2018-08-20 Thread Kyotaro HORIGUCHI
At Mon, 20 Aug 2018 13:23:38 +0900, Michael Paquier wrote in <20180820042338.gh7...@paquier.xyz> > On Fri, Aug 10, 2018 at 04:50:55PM -0400, Tom Lane wrote: > > I would certainly *not* back-patch the GetConfigOptionByNum change, > > as that will be a user-visible behavioral change that people

Re: Slotification of partition tuple conversion

2018-08-20 Thread Amit Langote
On 2018/08/20 20:15, Amit Khandekar wrote: > On 17 August 2018 at 21:47, Amit Khandekar wrote: > >> Attached is a patch tup_convert.patch that does the conversion >> directly using input and output tuple slots. This patch is to be >> applied on an essential patch posted by Amit Langote in [1]

Re: Reopen logfile on SIGHUP

2018-08-20 Thread Michael Paquier
On Tue, Aug 21, 2018 at 09:26:54AM +0900, Kyotaro HORIGUCHI wrote: > I suspect that something bad can happen on Windows. [troll mode] More and even worse things than that could happen on Windows. [/troll mode] -- Michael signature.asc Description: PGP signature

Re: Problem while updating a foreign table pointing to a partitioned table on foreign server

2018-08-20 Thread Kyotaro HORIGUCHI
Fujita-san thank you for the comment. At Tue, 14 Aug 2018 20:49:02 +0900, Etsuro Fujita wrote in <5b72c1ae.8010...@lab.ntt.co.jp> > (2018/08/09 22:04), Etsuro Fujita wrote: > > (2018/08/08 17:30), Kyotaro HORIGUCHI wrote: > >> I choosed to expand tuple descriptor for junk column added to > >>

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-08-20 Thread Robert Haas
On Mon, Aug 20, 2018 at 4:21 PM, Alvaro Herrera wrote: > I wonder if this all stems from a misunderstanding of what I suggested > to David offlist. My suggestion was that the catalog scans would > continue to use the catalog MVCC snapshot, and that the relcache entries > would contain all the

Re: Improve behavior of concurrent ANALYZE/VACUUM

2018-08-20 Thread Michael Paquier
On Mon, Aug 20, 2018 at 08:57:00PM +, Bossart, Nathan wrote: > Sorry for the delay! I looked through the latest patch. Thanks a lot for the review! > I like the idea of emitting a separate WARNING for each for clarity on > what operations are being skipped. However, I think it could be a

Re: Reopen logfile on SIGHUP

2018-08-20 Thread Kyotaro HORIGUCHI
At Fri, 10 Aug 2018 15:33:26 +0300, Alexander Kuzmenkov wrote in <5142559a-82e6-b3e4-d6ed-8fd2d240c...@postgrespro.ru> > On 08/09/2018 10:33 AM, Kyotaro HORIGUCHI wrote: > > > > - Since I'm not sure unlink is signal-handler safe on all > >supported platforms, I moved unlink() call out of >

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-20 Thread Michael Paquier
On Mon, Aug 20, 2018 at 10:54:28AM -0300, Alvaro Herrera wrote: > Seems reasonable. Thanks Álvaro, pushed. -- Michael signature.asc Description: PGP signature

Re: Improve behavior of concurrent ANALYZE/VACUUM

2018-08-20 Thread Bossart, Nathan
Hi, Sorry for the delay! I looked through the latest patch. On 8/17/18, 1:43 AM, "Michael Paquier" wrote: > I have reworked the patch on this side, clarifying the use of the new > common API for the logs. One thing I am wondering about is what do we > want to do when VACUUM ANALYZE is used.

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-08-20 Thread Alvaro Herrera
On 2018-Aug-13, Robert Haas wrote: > I think this is a somewhat confused analysis. We don't use > SnapshotAny for catalog scans, and we never have. We used to use > SnapshotNow, and we now use a current MVCC snapshot. What you're > talking about, I think, is possibly using the transaction

Re: Getting NOT NULL constraint from pg_attribute

2018-08-20 Thread David G. Johnston
On Monday, August 20, 2018, Wu Ivy wrote: > > Thanks for the quick respond. > Why are SELECT query never marked nullable? For nullable columns, when I > call SPI_getvalue(), the result (in char*) is NULL. I don’t think I’m too > clear on the definition of *attnotnull*. Can you give me a example

Re: Getting NOT NULL constraint from pg_attribute

2018-08-20 Thread Wu Ivy
Hi tom, Thanks for the quick respond. Why are SELECT query never marked nullable? For nullable columns, when I call SPI_getvalue(), the result (in char*) is NULL. I don’t think I’m too clear on the definition of attnotnull. Can you give me a example in which the tupleTable is can be marked

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Chris Travers
For the record, here's the proof of concept code for the transaction manager which works off libpq connections. It is not ready yet by any means. But it is included for design discussion. If the previous patch gets in instead, that's fine, but figure it is worth including here for discussion

Re: A really subtle lexer bug

2018-08-20 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> select f(a =>-1); -- ERROR: column "a" does not exist Andrew> I guess the fix is to extend the existing special case code Andrew> that checks for one character left after removing trailing [+-] Andrew> and also check for the two-character

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Chris Travers
On Mon, Aug 20, 2018 at 4:41 PM Andres Freund wrote: > Hi, > > On 2018-08-20 16:28:01 +0200, Chris Travers wrote: > > 1. INSERTMETHOD=[insert|copy] option on foreign table. > > > > One significant limitation of the PostgreSQL FDW is that it does a > prepared > > statement insert on each row

Re: Truncation failure in autovacuum results in data corruption (duplicate keys)

2018-08-20 Thread Tom Lane
Alexander Korotkov writes: > On Wed, Apr 18, 2018 at 10:04 PM Tom Lane wrote: >> It's hard to see a way around this that isn't fairly catastrophic for >> performance :-(. But in any case it's wrapped up in order-of-operations >> issues. I've long since forgotten the details, but I seem to have

Re: Truncation failure in autovacuum results in data corruption (duplicate keys)

2018-08-20 Thread Alexander Korotkov
On Wed, Apr 18, 2018 at 10:04 PM Tom Lane wrote: > [ re-reads thread... ] The extra assumption you need in order to have > trouble is that the blocks in question are dirty in shared buffers and > have never been written to disk since their rows were deleted. Then > the situation is that the

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Andres Freund
Hi, On 2018-08-20 10:56:39 -0400, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2018-08-20 16:28:01 +0200, Chris Travers wrote: > > > 1. INSERTMETHOD=[insert|copy] option on foreign table. > > > > > > One significant limitation of the PostgreSQL FDW is that it does a

Re: libpq compression

2018-08-20 Thread Konstantin Knizhnik
On 13.08.2018 23:06, Andrew Dunstan wrote: On 08/13/2018 02:47 PM, Robert Haas wrote: On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: This thread appears to have gone quiet. What concerns me is that there appears to be substantial disagreement between the author and the reviewers.

Re: Truncation failure in autovacuum results in data corruption (duplicate keys)

2018-08-20 Thread Alexander Korotkov
On Wed, Apr 18, 2018 at 11:49 PM Tom Lane wrote: > I wrote: > > Relation truncation throws away the page image in memory without ever > > writing it to disk. Then, if the subsequent file truncate step fails, > > we have a problem, because anyone who goes looking for that page will > > fetch it

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2018-08-20 16:28:01 +0200, Chris Travers wrote: > > 1. INSERTMETHOD=[insert|copy] option on foreign table. > > > > One significant limitation of the PostgreSQL FDW is that it does a prepared > > statement insert on each row written

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Tom Lane
Chris Travers writes: > I am looking at trying to make two modifications to the PostgreSQL FDW and > would like feedback on this before I do. > 1. INSERTMETHOD=[insert|copy] option on foreign table. > One significant limitation of the PostgreSQL FDW is that it does a prepared > statement

Re: Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Andres Freund
Hi, On 2018-08-20 16:28:01 +0200, Chris Travers wrote: > 1. INSERTMETHOD=[insert|copy] option on foreign table. > > One significant limitation of the PostgreSQL FDW is that it does a prepared > statement insert on each row written which imposes a per-row latency. This > hits environments where

Two proposed modifications to the PostgreSQL FDW

2018-08-20 Thread Chris Travers
Hi all; I am looking at trying to make two modifications to the PostgreSQL FDW and would like feedback on this before I do. 1. INSERTMETHOD=[insert|copy] option on foreign table. One significant limitation of the PostgreSQL FDW is that it does a prepared statement insert on each row written

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-20 Thread Alvaro Herrera
On 2018-Aug-20, Michael Paquier wrote: > On Tue, Aug 14, 2018 at 06:53:32PM +0200, Michael Paquier wrote: > > I was thinking about adding "Even if it is not atomic" or such at the > > beginning of the paragraph, but at the end your phrasing sounds better > > to me. So I have hacked up the

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Tom Lane
Andres Freund writes: > On 2018-08-20 14:38:25 +0200, Peter Eisentraut wrote: >> Do you have an alternative in mind? > One is to just not do anything. I'm not sure I'm on board with the goal > of changing things to make DDL on system tables more palatable. FWIW, I agree with doing as little as

Re: remove ATTRIBUTE_FIXED_PART_SIZE

2018-08-20 Thread Tom Lane
Peter Eisentraut writes: >> I don't think the tuple packing issue has to do with the tuple >> descriptor code. The tuple descriptors already use allocations of size >> sizeof(FormData_pg_attribute) (CreateTemplateTupleDesc), just the memcpy >> and memset calls use (potentially) less. That might

Re: [FEATURE PATCH] pg_stat_statements with plans (v02)

2018-08-20 Thread Tom Lane
Andres Freund writes: > On 2018-08-20 15:21:07 +1200, Thomas Munro wrote: >> -Werror=vla in GCC, apparently. > How about detecting support for that in our configure script and > automatically using it? If we're uncomfortable with raising an error, > let's at least raise a warning? Yeah, we'd

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Andres Freund
On 2018-08-20 14:38:25 +0200, Peter Eisentraut wrote: > On 20/08/2018 12:48, Andres Freund wrote: > > On 2018-08-20 11:37:49 +0900, Michael Paquier wrote: > >> On Fri, Jul 13, 2018 at 11:05:10AM +0200, Peter Eisentraut wrote: > >>> After reviewing that thread, I think my patch would still be

Re: WaitForOlderSnapshots refactoring

2018-08-20 Thread Andres Freund
Hi, On 2018-08-20 14:35:34 +0200, Peter Eisentraut wrote: > The attached patch factors out the CREATE INDEX CONCURRENTLY code that > waits for transactions with older snapshots to finish into a new > function WaitForOlderSnapshots(). > This refactoring was part of a previously posted REINDEX

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Peter Eisentraut
On 20/08/2018 12:48, Andres Freund wrote: > On 2018-08-20 11:37:49 +0900, Michael Paquier wrote: >> On Fri, Jul 13, 2018 at 11:05:10AM +0200, Peter Eisentraut wrote: >>> After reviewing that thread, I think my patch would still be relevant. >>> Because the pending proposal is to not add TOAST

WaitForOlderSnapshots refactoring

2018-08-20 Thread Peter Eisentraut
The attached patch factors out the CREATE INDEX CONCURRENTLY code that waits for transactions with older snapshots to finish into a new function WaitForOlderSnapshots(). This refactoring was part of a previously posted REINDEX CONCURRENTLY patch. But this code is now also appearing as a

Re: TupleTableSlot abstraction

2018-08-20 Thread Andres Freund
On 2018-08-20 17:51:38 +0530, Ashutosh Bapat wrote: > > I also noticed an independent issue in your changes to > > ExecInitScanTupleSlot(): You can't assume that the plan belonging to the > > ScanState have a Scan node in their plan. Look e.g. at Material, Sort > > etc. So currently your scanrelid

Re: Conflict handling for COPY FROM

2018-08-20 Thread Robert Haas
On Sat, Aug 4, 2018 at 9:10 AM Surafel Temesgen wrote: > In order to prevent extreme condition the patch also add a new GUC > variable called copy_max_error_limit that control the amount of error to > swallow before start to error and new failed record file options for copy > to write a failed

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Michael Paquier
On Mon, Aug 20, 2018 at 12:43:20PM +0200, Peter Eisentraut wrote: > Tests would require enabling allow_system_table_mods, which is > PGC_POSTMASTER, so we couldn't do it inside the normal regression test > suite. I'm not sure setting up a whole new test suite for this is worth it. I forgot this

libpq host/hostaddr/conninfo inconsistencies

2018-08-20 Thread Fabien COELHO
Hello devs, While reviewing various patches by Tom which are focussing on libpq multi-host behavior, https://commitfest.postgresql.org/19/1749/ https://commitfest.postgresql.org/19/1752/ it occured to me that there are a few more problems with the documentation, the

Re: partitioning - changing a slot's descriptor is expensive

2018-08-20 Thread David Rowley
On 20 August 2018 at 23:21, Andres Freund wrote: > On 2018-08-20 16:40:17 +0530, Amit Khandekar wrote: >> On 20 August 2018 at 14:45, Amit Langote >> wrote: >> > By the way, I think it might be a good idea to >> > try to merge this patch with the effort in the following thread. >> > >> > *

Re: partitioning - changing a slot's descriptor is expensive

2018-08-20 Thread Andres Freund
Hi, On 2018-08-20 16:40:17 +0530, Amit Khandekar wrote: > On 20 August 2018 at 14:45, Amit Langote > wrote: > > By the way, I think it might be a good idea to > > try to merge this patch with the effort in the following thread. > > > > * Reduce partition tuple routing overheads * > >

Re: Slotification of partition tuple conversion

2018-08-20 Thread Amit Khandekar
On 17 August 2018 at 21:47, Amit Khandekar wrote: > Attached is a patch tup_convert.patch that does the conversion > directly using input and output tuple slots. This patch is to be > applied on an essential patch posted by Amit Langote in [1] that > arranges for dedicated per-partition slots

Re: partitioning - changing a slot's descriptor is expensive

2018-08-20 Thread Amit Khandekar
On 20 August 2018 at 14:45, Amit Langote wrote: > Thanks for the review. > > On 2018/08/17 15:00, Amit Khandekar wrote: >> Thanks for the patch. I did some review of the patch (needs a rebase). >> Below are my comments. >> >> @@ -1936,12 +1936,11 @@ ExecPartitionCheckEmitError(ResultRelInfo >>

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Andres Freund
On 2018-08-20 11:37:49 +0900, Michael Paquier wrote: > On Fri, Jul 13, 2018 at 11:05:10AM +0200, Peter Eisentraut wrote: > > After reviewing that thread, I think my patch would still be relevant. > > Because the pending proposal is to not add TOAST tables to some catalogs > > such as pg_attribute,

Re: remove ATTRIBUTE_FIXED_PART_SIZE

2018-08-20 Thread Andres Freund
On 2018-08-20 12:34:15 +0200, Peter Eisentraut wrote: > On 20/08/2018 12:32, Peter Eisentraut wrote: > > On 18/08/2018 23:05, Tom Lane wrote: > >> Possibly we need to be more careful than we are now about whether > >> there's padding at the end of the fixed-size fields ... but just > >> ripping

Re: ALTER TABLE on system catalogs

2018-08-20 Thread Peter Eisentraut
On 20/08/2018 04:37, Michael Paquier wrote: > For 0002, indeed the patch is still > seems relevant. The comment block at the beginning of > create_toast_table should be updated. Some tests would also be nice to > have. Tests would require enabling allow_system_table_mods, which is

Re: remove ATTRIBUTE_FIXED_PART_SIZE

2018-08-20 Thread Peter Eisentraut
On 20/08/2018 12:32, Peter Eisentraut wrote: > On 18/08/2018 23:05, Tom Lane wrote: >> Possibly we need to be more careful than we are now about whether >> there's padding at the end of the fixed-size fields ... but just >> ripping out the code that attempts to deal with that is hardly >> an

Re: remove ATTRIBUTE_FIXED_PART_SIZE

2018-08-20 Thread Peter Eisentraut
On 18/08/2018 23:05, Tom Lane wrote: > Possibly we need to be more careful than we are now about whether > there's padding at the end of the fixed-size fields ... but just > ripping out the code that attempts to deal with that is hardly > an improvement. I don't think the tuple packing issue has

Re: [FEATURE PATCH] pg_stat_statements with plans (v02)

2018-08-20 Thread Andres Freund
Hi, On 2018-08-20 15:21:07 +1200, Thomas Munro wrote: > On Mon, Aug 20, 2018 at 2:55 PM, Tom Lane wrote: > > Thomas Munro writes: > > ... but I'm less excited about this one. Seems like a great opportunity > > for unexpected stack overflows, and thence at least the chance for > > DOS-causing

Re: partitioning - changing a slot's descriptor is expensive

2018-08-20 Thread Amit Langote
Thanks for the review. On 2018/08/17 15:00, Amit Khandekar wrote: > Thanks for the patch. I did some review of the patch (needs a rebase). > Below are my comments. > > @@ -1936,12 +1936,11 @@ ExecPartitionCheckEmitError(ResultRelInfo > *resultRelInfo, > + /* Input slot might be of a partition,

Re: Fix help option of contrib/oid2name

2018-08-20 Thread Michael Paquier
On Mon, Aug 20, 2018 at 03:51:07PM +0900, Tatsuro Yamada wrote: > On 2018/08/20 13:54, Michael Paquier wrote: > Therefore, "-P" is a manual bag. I investigated more using git log command and > understood followings: > > 1. -P option was removed on 4192f2d85 > 2. -P option revived in only the

Calculate total_table_pages after set_base_rel_sizes()

2018-08-20 Thread David Rowley
I believe that we should be delaying the PlannerInfo's total_table_pages calculation until after constraint exclusion and partition pruning have taken place. Doing this calculation before we determine which relations we don't need to scan can lead to incorrectly applying random_page_cost to too

A really subtle lexer bug

2018-08-20 Thread Andrew Gierth
Currently in PG, the precedence of = and <> is supposed to be equal, and the precedence of unary - is very high. So, (true=1 between 1 and 1) parses as ((true)=(1 between 1 and 1)), and (true=-1 between 1 and 1) parses as ((true)=((-1) between 1 and 1)). All good so far. (true<>-1 between 1 and

Re: Fix help option of contrib/oid2name

2018-08-20 Thread Tatsuro Yamada
On 2018/08/20 13:54, Michael Paquier wrote: On Mon, Aug 20, 2018 at 12:30:29PM +0900, Tatsuro Yamada wrote: vacuumlo: Document - Add long options - Add environment section Let's keep things simple by not adding long options where it is not especially obvious, so I would

Re: Conflict handling for COPY FROM

2018-08-20 Thread Surafel Temesgen
Thanks for looking at it 1. It sounded like you added the copy_max_error_limit GUC as part of this patch to allow users to specify how many errors they want to swallow with this new functionality. The GUC didn't seem to be defined and we saw no mention of it in the code. My guess is this might