Re: Patch to document base64 encoding

2019-08-02 Thread Karl O. Pinc
On Tue, 30 Jul 2019 23:44:49 +0200 (CEST) Fabien COELHO wrote: > Personnaly, I'd be ok with having a separate "conversion function" > table, and also with Tom suggestion to have string functions with > "only simple string" functions, and if any binary appears it is moved > into the binary

Re: Patch to document base64 encoding

2019-08-02 Thread Karl O. Pinc
On Fri, 02 Aug 2019 10:44:43 -0400 Tom Lane wrote: > I don't really have a problem with > repeating the entries for other functions that exist in both > text and bytea variants, either. Ok. Thanks. I'll repeat entries then. Regards, Karl Free Software: "You don't pay back, you pay

Re: Memory-Bounded Hash Aggregation

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 08:11:19AM -0700, Jeff Davis wrote: On Fri, 2019-08-02 at 14:44 +0800, Adam Lee wrote: I'm late to the party. You are welcome to join any time! These two approaches both spill the input tuples, what if the skewed groups are not encountered before the hash table fills

Re: pglz performance

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: Hi, On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF. I don't understand why it's a good idea to improve the compression

Re: pglz performance

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote: > On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: > > Hi, > > > > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > > > We have some kind of "roadmap" of "extensible pglz". We plan to > > > provide implementation on

Re: pglz performance

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote: On 27.06.2019 21:33, Andrey Borodin wrote: 13 мая 2019 г., в 12:14, Michael Paquier написал(а): Decompression can matter a lot for mostly-read workloads and compression can become a bottleneck for heavy-insert loads, so

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 1:42 AM Julien Rouhaud wrote: > Trivial patch for that attached. Thanks! > The output is now like: > > [...] > Using an oid in the 8000- range is recommended. > For instance: 9427 > > (checking that the suggested random oid is not used yet.) I've taken your patch,

Re: pglz performance

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote: > Hmmm, I don't remember the details of those patches so I didn't realize > it allows incremental recompression. If that's possible, that would mean > existing systems can start using it. Which is good. That depends on what do you mean by

Optimize single tuple fetch from nbtree index

2019-08-02 Thread Floris Van Nee
Hi hackers, While I was reviewing some code in another patch, I stumbled upon a possible optimization in the btree index code in nbtsearch.c for queries using 'LIMIT 1'. I have written a small patch that implements this optimization, but I'd like some feedback on the design of the patch,

Re: Patch to document base64 encoding

2019-08-02 Thread Tom Lane
"Karl O. Pinc" writes: > But I'm not happy with putting any function that works with > bytea into the binary string section. This would mean moving, > say, length() out of the regular string section. There's a > lot of functions that work on both string and bytea inputs > and most (not all, see

Re: Patch to document base64 encoding

2019-08-02 Thread Chapman Flack
On 8/2/19 10:32 AM, Karl O. Pinc wrote: > But I'm not happy with putting any function that works with > bytea into the binary string section. This would mean moving, > say, length() out of the regular string section. I'm not sure why. The bytea section already has an entry for its length()

[PATCH] Improve performance of NOTIFY over many databases (v2)

2019-08-02 Thread Martijn van Oosterhout
Hoi hackers, Here is a reworked version of the previous patches. The original three patches have been collapsed into one as given the changes discussed it didn't make sense to keep them separate. There are now two patches (the third is just to help with testing): Patch 1: Tracks the listening

Re: pglz performance

2019-08-02 Thread Andrey Borodin
Thanks for looking into this! > 2 авг. 2019 г., в 19:43, Tomas Vondra > написал(а): > > On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote: >> >> It takes me some time to understand that your memcpy optimization is >> correct;) Seems that comments are not explanatory

Re: Index Skip Scan

2019-08-02 Thread Jesper Pedersen
Hi, On 8/2/19 8:14 AM, Dmitry Dolgov wrote: And this too (slightly rewritten:). We will soon post the new version of patch with updates about UniqueKey from Jesper. Yes. We decided to send this now, although there is still feedback from David that needs to be considered/acted on. The

Re: pglz performance

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote: Hi, On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote: > Hi, > > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > > We have some kind of "roadmap" of "extensible

Re: Add client connection check during the execution of the query

2019-08-02 Thread Konstantin Knizhnik
On 18.07.2019 6:19, Tatsuo Ishii wrote: I noticed that there are some confusions in the doc and code regarding what the new configuration parameter means. According to the doc: +Default value is zero - it disables connection +checks, so the backend will detect client

Re: pglz performance

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote: > We have some kind of "roadmap" of "extensible pglz". We plan to provide > implementation on Novembers CF. I don't understand why it's a good idea to improve the compression side of pglz. There's plenty other people that spent a lot of

Re: partition routing layering in nodeModifyTable.c

2019-08-02 Thread Andres Freund
Hi, On 2019-07-31 17:04:38 +0900, Amit Langote wrote: > I looked into trying to do the things I mentioned above and it seems > to me that revising BeginDirectModify()'s API to receive the > ResultRelInfo directly as Andres suggested might be the best way > forward. I've implemented that in the

Re: Memory-Bounded Hash Aggregation

2019-08-02 Thread Jeff Davis
On Fri, 2019-08-02 at 14:44 +0800, Adam Lee wrote: > I'm late to the party. You are welcome to join any time! > These two approaches both spill the input tuples, what if the skewed > groups are not encountered before the hash table fills up? The spill > files' size and disk I/O could be

Re: Recent failures in IsolationCheck deadlock-hard

2019-08-02 Thread Tom Lane
Thomas Munro writes: > There have been five failures on three animals like this, over the > past couple of months: Also worth noting is that anole failed its first try at the new deadlock-parallel isolation test:

Re: [PROPOSAL] Temporal query processing with range types

2019-08-02 Thread Ibrar Ahmed
Hi, I have rebased the patch and currently reviewing the patch on master (1e2fddfa33d3c7cc93ca3ee0f32852699bd3e012). On Mon, Jul 1, 2019 at 4:45 PM Thomas Munro wrote: > On Wed, Apr 3, 2019 at 2:12 AM Ibrar Ahmed wrote: > > I start looking at the patch, there is a couple of problems with

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Stephen Frost
Greetings, On Fri, Aug 2, 2019 at 19:36 Ian Barwick wrote: > On 8/3/19 8:24 AM, Andres Freund wrote: > > Hi, > > > > On 2019-08-03 08:22:29 +0900, Ian Barwick wrote: > >> What I came up with shoehorned a stripped-down version of the backend > >> config parser into fe_utils and provides a

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 3:52 PM Tom Lane wrote: > Better ... but I'm the world's second worst Perl programmer, > so I have little to say about whether it's idiomatic. Perhaps Michael can weigh in here? I'd rather hear a second opinion on v4 of the patch before proceeding. -- Peter Geoghegan

Re: POC: Cleaning up orphaned files using undo logs

2019-08-02 Thread Amit Kapila
On Mon, Jul 22, 2019 at 3:51 PM Dilip Kumar wrote: > > On Mon, Jul 22, 2019 at 2:21 PM Amit Kapila wrote: > > > > I have reviewed 0012-Infrastructure-to-execute-pending-undo-actions, > Please find my comment so far. .. > 4. > +void > +undoaction_redo(XLogReaderState *record) > +{ > + uint8 info

Re: Avoid full GIN index scan when possible

2019-08-02 Thread Nikita Glukhov
Attached 6th version of the patches. On 01.08.2019 22:28, Tom Lane wrote: Alexander Korotkov writes: On Thu, Aug 1, 2019 at 9:59 PM Tom Lane wrote: While I've not attempted to fix that here, I wonder whether we shouldn't fix it by just forcing forcedRecheck to true in any case where we

Re: pglz performance

2019-08-02 Thread Konstantin Knizhnik
On 02.08.2019 21:20, Andres Freund wrote: Another question is whether we'd actually want to include the code in core directly, or use system libraries (and if some packagers might decide to disable that, for whatever reason). I'd personally say we should have an included version, and a

Re: pglz performance

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Hi, On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote: Hmmm, I don't remember the details of those patches so I didn't realize it allows incremental recompression. If that's possible, that would mean existing systems can start using

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 18:38:46 -0400, Stephen Frost wrote: > On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > > > Tomas Vondra writes: > > > There seems to be a consensus that this this not a pg_basebackup issue > > > (i.e. duplicate values don't make the file invalid), and it should be > > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tom Lane
Andres Freund writes: > On 2019-08-02 18:47:07 -0400, Tom Lane wrote: >> I don't find that to be necessary or even desirable. Many (most?) of the >> situations where this would be important wouldn't have access to a running >> backend, and maybe not to any PG code at all --- what if your tool

Re: partition routing layering in nodeModifyTable.c

2019-08-02 Thread Andres Freund
Hi, On 2019-08-01 18:38:09 +0900, Etsuro Fujita wrote: > On Thu, Aug 1, 2019 at 10:33 AM Amit Langote wrote: > > If it's the approach of adding a resultRelation Index field to > > ForeignScan node, I tried and had to give up, realizing that we don't > > maintain ResultRelInfos in an array that

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 1:49 PM Tom Lane wrote: > Maybe s/make a/start with/ ? > Also, once people start doing this, it'd be unfriendly to suggest > 9099 if 9100 is already committed. There should be some attention > to *how many* consecutive free OIDs will be available if one starts > at the

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Isaac Morland
On Fri, 2 Aug 2019 at 16:49, Tom Lane wrote: > Peter Geoghegan writes: > > I've taken your patch, and changed the wording a bit. I think that > > it's worth being a bit more explicit. The attached revision produces > > output that looks like this: > > > Patches should use a more-or-less

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Tom Lane
Peter Geoghegan writes: > Attached is v3, which implements your suggestion, generating output > like the above. I haven't written a line of Perl in my life prior to > today, so basic code review would be helpful. The "if ($oid > $prev_oid + 2)" test seems unnecessary. It's certainly wrong to

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Tom Lane
Peter Geoghegan writes: > How about the attached? I've simply removed the "if ($oid > $prev_oid > + 2)" test. Better ... but I'm the world's second worst Perl programmer, so I have little to say about whether it's idiomatic. regards, tom lane

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tom Lane
I wrote: > Andres Freund writes: >> I think a commandline tool to perform the equivalent of ALTER SYSTEM on >> a shutdown cluster would be a great tool. > Perhaps, but ... >> Obviously this is widely out of scope for v12. > ... this. Although, there's always echo "alter system set work_mem =

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Ian Barwick
On 8/3/19 7:56 AM, Andres Freund wrote: Hi, On 2019-08-02 18:47:07 -0400, Tom Lane wrote: Stephen Frost writes: I disagree that this should only be addressed in alter system, as I’ve said before and as others have agreed with. Having one set of code that can be used to update parameters in

Re: Optimize single tuple fetch from nbtree index

2019-08-02 Thread Floris Van Nee
Hi Tom, Thanks for your quick reply! > Regardless of whether there's actually a LIMIT 1? That seems disastrous > for every other case than the narrow one where the optimization wins. > Because every other case is now going to have to touch the index page > twice. That's more CPU and about

Re: jsonb_plperl bug

2019-08-02 Thread Tom Lane
=?UTF-8?B?SXZhbiBQYW5jaGVua28=?= writes: > I have found a bug in jsonb_plperl extension. A possible fix is proposed > below. > ... > + /* SVt_PV without POK flag is also NULL */ > + if(SvTYPE(in) == SVt_PV) Ugh. Doesn't Perl provide some

Re: Remove HeapTuple and Buffer dependency for predicate locking functions

2019-08-02 Thread Ashwin Agrawal
On Wed, Jul 31, 2019 at 2:06 PM Andres Freund wrote: > Looking at the code as of master, we currently have: > Super awesome feedback and insights, thank you! - PredicateLockTuple() calls SubTransGetTopmostTransaction() to figure > out a whether the tuple has been locked by the current >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Stephen Frost
Greetings, On Fri, Aug 2, 2019 at 18:47 Tom Lane wrote: > Stephen Frost writes: > > On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > >>> The proposal seems to be to run through the .auto.conf file, remove any > >>> duplicates, and append the new entry at the end. That seems reasonable. > > >>

Re: using index or check in ALTER TABLE SET NOT NULL

2019-08-02 Thread Tom Lane
Tomas Vondra writes: > I think there's a consensus to change INFO to DEBUG1 in pg12, and then > maybe imlpement something like VERBOSE mode in the future. Objections? ISTM the consensus is "we'd rather reduce the verbosity, but we don't want to give up test coverage". So what's blocking this is

Re: Optimize single tuple fetch from nbtree index

2019-08-02 Thread Tom Lane
Floris Van Nee writes: > Every time the index scan is done, all tuples from the leaf page are > read in nbtsearch.c:_bt_readpage. The idea of this patch is to make an > exception for this *only* the first time amgettuple gets called. Regardless of whether there's actually a LIMIT 1? That seems

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tomas Vondra
Hi, This thread discusses an issue that's tracked as an open item for pg12, but it's been quiet for the last ~1 month. I think it's probably time to decide what to do with it. The thread is a bit long, so let me sum what the issue is and what options we have. The problem is that ALTER SYSTEM

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Stephen Frost
Greetings, On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > Tomas Vondra writes: > > There seems to be a consensus that this this not a pg_basebackup issue > > (i.e. duplicate values don't make the file invalid), and it should be > > handled in ALTER SYSTEM. > > Yeah. I doubt pg_basebackup is

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Ian Barwick
On 8/3/19 7:27 AM, Tom Lane wrote: Tomas Vondra writes: There seems to be a consensus that this this not a pg_basebackup issue (i.e. duplicate values don't make the file invalid), and it should be handled in ALTER SYSTEM. Yeah. I doubt pg_basebackup is the only actor that can create such

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tom Lane
Stephen Frost writes: > On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: >>> The proposal seems to be to run through the .auto.conf file, remove any >>> duplicates, and append the new entry at the end. That seems reasonable. >> +1 > I disagree that this should only be addressed in alter system, as

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 06:38:46PM -0400, Stephen Frost wrote: Greetings, On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: Tomas Vondra writes: > There seems to be a consensus that this this not a pg_basebackup issue > (i.e. duplicate values don't make the file invalid), and it should be >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Andres Freund
Hi, On 2019-08-03 08:22:29 +0900, Ian Barwick wrote: > What I came up with shoehorned a stripped-down version of the backend > config parser into fe_utils and provides a function to modify pg.auto.conf > in much the same way ALTER SYSTEM does, but with only the basic syntax > checking provided by

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 20:27:25 -0400, Stephen Frost wrote: > On Fri, Aug 2, 2019 at 18:47 Tom Lane wrote: > > Stephen Frost writes: > > > On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > > >>> The proposal seems to be to run through the .auto.conf file, remove any > > >>> duplicates, and append the

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 20:57:20 -0400, Stephen Frost wrote: > No, I’m saying that we already *have* a library and we can add a few > functions to it and if people want to leverage those functions then they > can write glue code to do so, just like was done with libpq. The argument > that “we shouldn’t

Re: pgbench - implement strict TPC-B benchmark

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 10:34:24 +0200, Fabien COELHO wrote: > > Hello Andres, > > Thanks a lot for these feedbacks and comments. > > > Using pgbench -Mprepared -n -c 8 -j 8 -S pgbench_100 -T 10 -r -P1 > > e.g. shows pgbench to use 189% CPU in my 4/8 core/thread laptop. That's > > a pretty

Re: using index or check in ALTER TABLE SET NOT NULL

2019-08-02 Thread Tomas Vondra
On Wed, Jun 12, 2019 at 08:34:57AM +1200, David Rowley wrote: On Tue, 11 Jun 2019 at 03:35, Sergei Kornilov wrote: > Does anyone think we shouldn't change the INFO message in ATTACH > PARTITION to a DEBUG1 in PG12? Seems no one wants to vote against this change. I'm concerned about two

Re: partition routing layering in nodeModifyTable.c

2019-08-02 Thread Etsuro Fujita
Hi, On Sat, Aug 3, 2019 at 3:01 AM Andres Freund wrote: > On 2019-07-31 17:04:38 +0900, Amit Langote wrote: > > I looked into trying to do the things I mentioned above and it seems > > to me that revising BeginDirectModify()'s API to receive the > > ResultRelInfo directly as Andres suggested

First draft of back-branch release notes is done

2019-08-02 Thread Tom Lane
See https://git.postgresql.org/pg/commitdiff/082c9f5f761ced18a6f014f2638096f6a8228164 Please send comments/corrections before Sunday. regards, tom lane

Re: partition routing layering in nodeModifyTable.c

2019-08-02 Thread Andres Freund
Hi, On 2019-08-03 05:20:35 +0900, Etsuro Fujita wrote: > On Sat, Aug 3, 2019 at 3:01 AM Andres Freund wrote: > > On 2019-07-31 17:04:38 +0900, Amit Langote wrote: > > > I looked into trying to do the things I mentioned above and it seems > > > to me that revising BeginDirectModify()'s API to

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 2:52 PM Isaac Morland wrote: > Noob question here: why not start with the next unused OID in the range, and > on the other hand reserve the range for sequentially-assigned values? The general idea is to avoid OID collisions while a patch is under development. Choosing a

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Ian Barwick
On 8/3/19 8:09 AM, Tom Lane wrote: I wrote: Andres Freund writes: I think a commandline tool to perform the equivalent of ALTER SYSTEM on a shutdown cluster would be a great tool. Perhaps, but ... Obviously this is widely out of scope for v12. ... this. Although, there's always

Re: Optimize single tuple fetch from nbtree index

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 1:43 PM Tom Lane wrote: > Meh. I think the odds that you got this 100% right are small, and the > odds that it would be maintainable are smaller. There's too much that > can happen if you're not holding any lock --- and there's a lot of active > work on btree indexes,

Re: Optimize single tuple fetch from nbtree index

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 5:34 PM Peter Geoghegan wrote: > I wonder if some variety of block nested loop join would be helpful > here. I'm not aware of any specific design that would help with > Floris' case, but the idea of reducing the number of scans required on > the inner side by buffering

Re: tableam vs. TOAST

2019-08-02 Thread Andres Freund
Hi, On 2019-08-01 12:23:42 -0400, Robert Haas wrote: > Roughly, on both master and with the patches, the first one takes > about 4.2 seconds, the second 7.5, and the third 1.2. The third one > is the fastest because it doesn't do any compression. Since it does > less irrelevant work than the

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Peter Geoghegan
On Fri, Aug 2, 2019 at 3:19 PM Tom Lane wrote: > The "if ($oid > $prev_oid + 2)" test seems unnecessary. > It's certainly wrong to keep iterating beyond the first > oid that's > $suggestion. Sorry. That was just carelessness on my part. (Being the world's worst Perl programmer is no excuse.)

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 18:47:07 -0400, Tom Lane wrote: > Stephen Frost writes: > > I disagree that this should only be addressed in alter system, as I’ve said > > before and as others have agreed with. Having one set of code that can be > > used to update parameters in the auto.conf and then have

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-02 Thread Andres Freund
Hi, On 2019-08-02 10:54:35 +0200, Julien Rouhaud wrote: > On Thu, Aug 1, 2019 at 11:05 PM Andres Freund wrote: > > > > I'm actually quite unconvinced that it's sensible to update the global > > value for nested queries. That'll mean e.g. the log_line_prefix and > > pg_stat_activity values are

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Ian Barwick
On 8/3/19 8:24 AM, Andres Freund wrote: Hi, On 2019-08-03 08:22:29 +0900, Ian Barwick wrote: What I came up with shoehorned a stripped-down version of the backend config parser into fe_utils and provides a function to modify pg.auto.conf in much the same way ALTER SYSTEM does, but with only

Re: SQL:2011 PERIODS vs Postgres Ranges?

2019-08-02 Thread Ibrar Ahmed
The patch does not work. postgres=# CREATE TABLE foo (id int,r int4range, valid_at tsrange, CONSTRAINT bar_pk PRIMARY KEY (r, valid_at WITHOUT OVERLAPS)); CREATE TABLE postgres=# CREATE TABLE bar (id int,r int4range, valid_at tsrange, CONSTRAINT bar_fk FOREIGN KEY (r, PERIOD valid_at)

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Julien Rouhaud
Le ven. 2 août 2019 à 20:12, Peter Geoghegan a écrit : > On Fri, Aug 2, 2019 at 1:42 AM Julien Rouhaud wrote: > > Trivial patch for that attached. > > Thanks! > > > The output is now like: > > > > [...] > > Using an oid in the 8000- range is recommended. > > For instance: 9427 > > > >

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Tom Lane
Peter Geoghegan writes: > I've taken your patch, and changed the wording a bit. I think that > it's worth being a bit more explicit. The attached revision produces > output that looks like this: > Patches should use a more-or-less consecutive range of OIDs. > Best practice is to make a random

jsonb_plperl bug

2019-08-02 Thread Ivan Panchenko
Hi, I have found a bug in jsonb_plperl extension. A possible fix is proposed below. jsonb_perl is the contrib module, which defines TRANSFORM functions for jsonb data type and PL/Perl procedural language. The bug can be reproduced as follows: CREATE EXTENSION plperl; CREATE EXTENSION 

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Tom Lane
Tomas Vondra writes: > There seems to be a consensus that this this not a pg_basebackup issue > (i.e. duplicate values don't make the file invalid), and it should be > handled in ALTER SYSTEM. Yeah. I doubt pg_basebackup is the only actor that can create such situations. > The proposal seems

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-02 Thread Stephen Frost
Greetings, On Fri, Aug 2, 2019 at 20:46 Andres Freund wrote: > Hi, > > On 2019-08-02 20:27:25 -0400, Stephen Frost wrote: > > On Fri, Aug 2, 2019 at 18:47 Tom Lane wrote: > > > Stephen Frost writes: > > > > On Fri, Aug 2, 2019 at 18:27 Tom Lane wrote: > > > >>> The proposal seems to be to

Re: block-level incremental backup

2019-08-02 Thread vignesh C
On Thu, Aug 1, 2019 at 5:06 PM Jeevan Chalke wrote: > > On Tue, Jul 30, 2019 at 9:39 AM Jeevan Chalke > wrote: >> >> I am almost done writing the patch for pg_combinebackup and will post soon. > > > Attached patch which implements the pg_combinebackup utility used to combine > full basebackup

Re: pglz performance

2019-08-02 Thread Konstantin Knizhnik
On 27.06.2019 21:33, Andrey Borodin wrote: 13 мая 2019 г., в 12:14, Michael Paquier написал(а): Decompression can matter a lot for mostly-read workloads and compression can become a bottleneck for heavy-insert loads, so improving compression or decompression should be two separate

Re: pgbench - implement strict TPC-B benchmark

2019-08-02 Thread Fabien COELHO
Hello Robert, All in all, pgbench overheads are small compared to postgres processing times and representative of a reasonably optimized client application. It's pretty easy to devise tests where pgbench is client-limited -- just try running it with threads = clients/4, sometimes even

Proposal: Clean up RangeTblEntry nodes after query preparation

2019-08-02 Thread Daniel Migowski
Hello, I currently have the problem that a simple prepared statement for a query like     select * from vw_report_invoice where id = $1 results in 33MB memory consumption on my side. It is a query that does about >20 joins over partially wide tables, but only a very small subset of columns

Re: complier warnings from ecpg tests

2019-08-02 Thread Sergei Kornilov
Hi >>  Thanks for the set of flags. So this comes from the use of -Og, and >>  the rest of the tree does not complain. The issue is that gcc >>  complains about the buffer not being large enough, but %d only uses up >>  to 2 characters so there is no overflow. In order to fix the issue it >>  is

Re: Memory-Bounded Hash Aggregation

2019-08-02 Thread Adam Lee
> High-level approaches: > > 1. When the in-memory hash table fills, keep existing entries in the > hash table, and spill the raw tuples for all new groups in a > partitioned fashion. When all input tuples are read, finalize groups > in memory and emit. Now that the in-memory hash table is

Re: Hash join explain is broken

2019-08-02 Thread Andres Freund
Hi, On 2019-07-02 10:50:02 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> Tom, any comments? Otherwise I'll go ahead, and commit after a round or > >> two of polishing. > > > Sorry for not getting to this sooner --- I'll try to look tomorrow. > > I took a look, and I think

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-02 Thread Kyotaro Horiguchi
Hello. At Fri, 2 Aug 2019 11:35:06 +1200, Thomas Munro wrote in > On Sat, Jul 27, 2019 at 6:26 PM Noah Misch wrote: > > On Thu, Jul 25, 2019 at 10:39:36AM +0900, Kyotaro Horiguchi wrote: > > > No substantial change have been made by this rebasing. > > > > Thanks. I'll likely review this on

Re: WIP: Data at rest encryption

2019-08-02 Thread Shawn Wang
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, failed Spec compliant: tested, failed Documentation:not tested Hi Antonin, It is very glad to see the new patch. I used the

Re: [proposal] de-TOAST'ing using a iterator

2019-08-02 Thread John Naylor
On Tue, Jul 30, 2019 at 8:20 PM Binguo Bao wrote: > > John Naylor 于2019年7月29日周一 上午11:49写道: >> >> 1). For every needle comparison, text_position_next_internal() >> calculates how much of the value is needed and passes that to >> detoast_iterate(), which then calculates if it has to do something

Re: Built-in connection pooler

2019-08-02 Thread Konstantin Knizhnik
On 02.08.2019 12:57, DEV_OPS wrote: Hello Konstantin would you please re-base this patch? I'm going to test it, and back port into PG10 stable and PG9 stable thank you very much Thank you. Rebased patch is attached. -- Konstantin Knizhnik Postgres Professional:

Re: Partial join

2019-08-02 Thread Richard Guo
On Thu, Aug 1, 2019 at 7:46 PM Arne Roland wrote: > Hello Richard, > > thanks for your quick reply. > > > > We need to fix this. > > > Do you have a better idea than just keeping the old quals - possibly just > the ones that get eliminated - in a separate data structure? Is the push > down of

Re: Fix typos

2019-08-02 Thread Sehrope Sarkuni
On Thu, Aug 1, 2019 at 10:18 PM Tom Lane wrote: > It's British vs. American spelling. For the most part, Postgres > follows American spelling, but there's the odd Briticism here and > there. Thanks for the explanation. I thought that might be the case but didn't find any other usages of

Re: Store FullTransactionId in TwoPhaseFileHeader/GlobalTransactionData

2019-08-02 Thread Thomas Munro
On Fri, Aug 2, 2019 at 1:38 AM vignesh C wrote: > On Thu, Aug 1, 2019 at 5:36 PM Thomas Munro wrote: > > On Thu, Aug 1, 2019 at 9:32 PM vignesh C wrote: > > > There is also one more comment which is yet to be concluded. The > > > comment discusses about changing subxids which are of

Re: Should we add xid_current() or a int8->xid cast?

2019-08-02 Thread Thomas Munro
On Thu, Jul 25, 2019 at 1:11 PM Thomas Munro wrote: > On Thu, Jul 25, 2019 at 12:42 PM Tom Lane wrote: > > Andres Freund writes: > > > On 2019-07-24 20:34:39 -0400, Tom Lane wrote: > > >> Yeah, I would absolutely NOT recommend that you open that can of worms > > >> right now. We have looked at

Re: Fix typos

2019-08-02 Thread Sehrope Sarkuni
On Fri, Aug 2, 2019 at 12:11 AM Michael Paquier wrote: > On Thu, Aug 01, 2019 at 11:01:59PM -0400, Alvaro Herrera wrote: > > I think slight variations don't really detract from the value of the > > product, and consider the odd variation a reminder of the diversity of > > the project. I don't

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-08-02 Thread Adrien Nayrat
On 8/1/19 12:04 PM, Tomas Vondra wrote: > On Thu, Aug 01, 2019 at 11:47:46AM +0200, Adrien Nayrat wrote: >> Hi, >> >> As we are at the end of this CF and there is still discussions about whether >> we >> should revert log_statement_sample_limit and log_statement_sample_rate, or >> try >> to fix

Re: [HACKERS] Cached plans and statement generalization

2019-08-02 Thread Daniel Migowski
Am 01.08.2019 um 18:56 schrieb Konstantin Knizhnik: I decided to implement your proposal. Much simple version of autoprepare patch is attached. At my computer I got the following results:  pgbench -M simple -S 22495 TPS  pgbench -M extended -S    47633 TPS  pgbench -M prepared -S   

Re: pgbench - implement strict TPC-B benchmark

2019-08-02 Thread Fabien COELHO
Hello Andres, Thanks a lot for these feedbacks and comments. Using pgbench -Mprepared -n -c 8 -j 8 -S pgbench_100 -T 10 -r -P1 e.g. shows pgbench to use 189% CPU in my 4/8 core/thread laptop. That's a pretty significant share. Fine, but what is the corresponding server load? 211%? 611%? And

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-02 Thread Julien Rouhaud
On Thu, Aug 1, 2019 at 11:05 PM Andres Freund wrote: > > I'm actually quite unconvinced that it's sensible to update the global > value for nested queries. That'll mean e.g. the log_line_prefix and > pg_stat_activity values are most of the time going to be bogus while > nested, because the

Re: [HACKERS] Cached plans and statement generalization

2019-08-02 Thread Konstantin Knizhnik
On 02.08.2019 11:25, Daniel Migowski wrote: Am 01.08.2019 um 18:56 schrieb Konstantin Knizhnik: I decided to implement your proposal. Much simple version of autoprepare patch is attached. At my computer I got the following results:  pgbench -M simple -S 22495 TPS  pgbench -M

Re: [HACKERS] Cached plans and statement generalization

2019-08-02 Thread Daniel Migowski
Am 02.08.2019 um 10:57 schrieb Konstantin Knizhnik: On 02.08.2019 11:25, Daniel Migowski wrote: I have two suggestions however: 1. Please allow to gather information about the autoprepared statements by returning them in pg_prepared_statements view. I would love to monitor usage of them as

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-02 Thread Julien Rouhaud
On Fri, Aug 2, 2019 at 6:21 AM Michael Paquier wrote: > > On Thu, Aug 01, 2019 at 06:59:06PM -0700, Peter Geoghegan wrote: > > Seems like I should propose a patch this time around. I don't do Perl, > > but I suppose I could manage something as trivial as this. > > Well, that new project policy is

A couple of random BF failures in kerberosCheck

2019-08-02 Thread Thomas Munro
Hello, https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=elver=2019-07-24%2003%3A22%3A17 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2019-08-02%2007%3A17%3A25 I wondered if this might be like the recently fixed problem with slapd not being ready to handle requests yet,

Re: Partial join

2019-08-02 Thread Richard Guo
On Thu, Aug 1, 2019 at 10:15 PM Tom Lane wrote: > Richard Guo writes: > > For the third query, a rough investigation shows that, the qual 'sl = > > 5' and 'sc.sl = sg.sl' will form an equivalence class and generate two > > implied equalities: 'sc.sl = 5' and 'sg.sl = 5', which can be pushed >

Re: Commitfest 2019-07, the first of five* for PostgreSQL 13

2019-08-02 Thread Fabien COELHO
I guess the CF app could show those kind of metrics, but having a written report from a human seems to be a good idea (I got it from Alvaro's blog[1]). The CF is now closed, and here are the final numbers: status | w1 | w2 | w3 | w4 | final

Recent failures in IsolationCheck deadlock-hard

2019-08-02 Thread Thomas Munro
Hi, There have been five failures on three animals like this, over the past couple of months: step s6a7: LOCK TABLE a7; step s7a8: LOCK TABLE a8; step s8a1: LOCK TABLE a1; -step s8a1: <... completed> step s7a8: <... completed> -error in steps s8a1 s7a8: ERROR: deadlock detected +step

Re: Store FullTransactionId in TwoPhaseFileHeader/GlobalTransactionData

2019-08-02 Thread Robert Haas
On Fri, Aug 2, 2019 at 6:37 AM Thomas Munro wrote: > Thanks. This looks pretty reasonable to me, and I don't think we need > to worry about the subxid list for now. Why not just do them all at once? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Index Skip Scan

2019-08-02 Thread Dmitry Dolgov
> On Thu, Jul 25, 2019 at 1:21 PM Kyotaro Horiguchi > wrote: > > I feel uncomfortable to look into indexinfo there. Couldnd't we > use indexskipprefix == -1 to signal !amcanskip from > create_index_path? Looks like it's not that straightforward to do this only in create_index_path, since to