Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
Dear Hackers, I've started working on something I've ended up calling "Super PathKeys". The idea here is to increase the likelihood of a Path with PathKeys being used for a purpose that requires a less strict sort order due to ordering being required from the return value of some precision loss

RE: [PROPOSAL]a new data type 'bytea' for ECPG

2018-10-30 Thread Tsunakawa, Takayuki
From: Michael Meskes [mailto:mes...@postgresql.org] > > bytea as a type of table definition may correspond to BLOB in the > > standard. > > Would we prefer to add a blob datatype then? > > > It seems that there is no defact and no product following to the > > standards. > > I wonder whether

Re: ToDo: show size of partitioned table

2018-10-30 Thread Amit Langote
Hi Mathias, Pavel, On 2018/08/17 12:26, Mathias Brossard wrote: > On Thu, Aug 16, 2018 at 12:46 AM Pavel Stehule >> >> This is question - maybe we can support older partitioning based on only >> inheritance - and the query can be more exact on PostgreSQL 10 and newer. >> >> Please, send any

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 4:48, Tom Lane wrote: > I was confused about why the memory leak in Bruno's example is so much > larger in HEAD than v11; spgbeginscan does allocate more stuff than > before, but only if numberOfOrderBys > 0, which surely doesn't apply for > the exclusion-check code path. Eventually

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 4:48, Tom Lane wrote: > I wrote: >> Alvaro Herrera writes: >>> How about modifying SysScanDescData to have a memory context member, >>> which is created by systable_beginscan and destroyed by endscan? > >> I think it would still have problems, in that it would affect in which >>

Re: [HACKERS] generated columns

2018-10-30 Thread Simon Riggs
On Wed, 27 Dec 2017 at 17:31, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 9/12/17 15:35, Jaime Casanova wrote: > > On 10 September 2017 at 00:08, Jaime Casanova > > wrote: > >> > >> During my own tests, though, i found some problems: > > Here is an updated patch that should

Re: ToDo: show size of partitioned table

2018-10-30 Thread Pavel Stehule
Hi út 30. 10. 2018 v 7:52 odesílatel Amit Langote < langote_amit...@lab.ntt.co.jp> napsal: > Hi Mathias, Pavel, > > On 2018/08/17 12:26, Mathias Brossard wrote: > > On Thu, Aug 16, 2018 at 12:46 AM Pavel Stehule > >> > >> This is question - maybe we can support older partitioning based on only

Re: Allow auto_explain to log to NOTICE

2018-10-30 Thread Daniel Gustafsson
> On 31 Jul 2018, at 14:23, Andrew Dunstan > wrote: >> I’m not sure it’s worth adding this much to the code just to be able to test >> it, but it seemed like a good excercise to write to have something to reason >> about. > > I think it probably is, buit I'm not very happy about the hack, so I

Re: INSTALL file

2018-10-30 Thread Andrew Dunstan
On 10/30/2018 06:14 AM, Andreas 'ads' Scherbaum wrote: On 30.10.18 04:11, Michael Paquier wrote: On Mon, Oct 29, 2018 at 01:01:47PM +0100, Andreas 'ads' Scherbaum wrote: That is not the first file people looking at. Especially not people looking at the GitHub copy:

Re: Online verification of checksums

2018-10-30 Thread Michael Banck
Hi Fabien, On Thu, Oct 25, 2018 at 10:16:03AM +0200, Fabien COELHO wrote: > >New version 5 attached. > > Patch does not seem to apply anymore. Thanks, rebased version attached. > Moreover, ISTM that some discussions about behavioral changes are not fully > settled. > > My current opinion is

Re: INSTALL file

2018-10-30 Thread Andreas 'ads' Scherbaum
On 30.10.18 11:49, Andrew Dunstan wrote: On 10/30/2018 06:14 AM, Andreas 'ads' Scherbaum wrote: On 30.10.18 04:11, Michael Paquier wrote: On Mon, Oct 29, 2018 at 01:01:47PM +0100, Andreas 'ads' Scherbaum wrote: That is not the first file people looking at. Especially not people looking at

Re: [HACKERS] generated columns

2018-10-30 Thread Sergei Kornilov
Hi I applied this patch on top 2fe42baf7c1ad96b5f9eb898161e258315298351 commit and found a bug while adding STORED column: postgres=# create table test(i int); CREATE TABLE postgres=# insert into test values (1),(2); INSERT 0 2 postgres=# alter table test add column gen_stored integer GENERATED

Re: Sequential UUID Generation

2018-10-30 Thread Uday Bhaskar V
Thanks Tomas! I will try. Regards, Uday On Tue, Oct 30, 2018 at 6:43 PM Tomas Vondra wrote: > I don't think PostgreSQL has anything like that at the moment. It would > not be difficult to tweak the UUID generator to generate sequential (or > monotonic) values, the tricky part seems to be

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On Tue, Oct 30, 2018 at 7:11 PM Amit Langote wrote: > > On 2018/10/30 4:48, Tom Lane wrote: > > I wrote: > >> Alvaro Herrera writes: > >>> How about modifying SysScanDescData to have a memory context member, > >>> which is created by systable_beginscan and destroyed by endscan? > > > >> I think

Re: Sequential UUID Generation

2018-10-30 Thread Tomas Vondra
I don't think PostgreSQL has anything like that at the moment. It would not be difficult to tweak the UUID generator to generate sequential (or monotonic) values, the tricky part seems to be durability requirements. One idea would be to simply store the value in (shared) memory, but that would

Re: PostgreSQL Limits and lack of documentation about them.

2018-10-30 Thread David Rowley
On 26 October 2018 at 11:40, Haribabu Kommi wrote: > On Fri, Oct 26, 2018 at 9:30 AM David Rowley > wrote: >> >> For a long time, we documented our table size, max columns, max column >> width limits, etc. in https://www.postgresql.org/about/ , but that >> information seems to have now been

Re: ToDo: show size of partitioned table

2018-10-30 Thread Michael Paquier
On Tue, Oct 30, 2018 at 09:24:01PM +0900, Amit Langote wrote: >> It is based on Mathias's patch. Although we can use >> pg_partition_tree on PostgreSQL, we still should to support >> PostgreSQL 10, 11 where this function is not available > > Ah, I forgot that psql will need to consider 10 and 11

Re: [HACKERS] generated columns

2018-10-30 Thread Erik Rijkers
On 2018-10-30 09:35, Peter Eisentraut wrote: [v5-0001-Generated-columns.patch ] Hi, I couldn't get this to apply to current head. I tried: patch --dry-run --ignore-whitespace -p 0 -F 5 < v5-0001-Generated-columns.patch and varied both -p and -F paramaters to no avail. Am I doing it

Re: ToDo: show size of partitioned table

2018-10-30 Thread Amit Langote
On Tue, Oct 30, 2018 at 8:04 PM Pavel Stehule wrote: > út 30. 10. 2018 v 7:52 odesílatel Amit Langote > napsal: >> The patch to add the pg_partition_tree() function was just committed: >> >> Add pg_partition_tree to display information about partitions >>

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread Simon Riggs
On Tue, 30 Oct 2018 at 07:58, David Rowley wrote: > I've started working on something I've ended up calling "Super > PathKeys". The idea here is to increase the likelihood of a Path with > PathKeys being used for a purpose that requires a less strict sort > order due to ordering being required

Re: shared-memory based stats collector

2018-10-30 Thread Tomas Vondra
On 10/05/2018 10:30 AM, Kyotaro HORIGUCHI wrote: > Hello. > > At Tue, 02 Oct 2018 16:06:51 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20181002.160651.117284090.horiguchi.kyot...@lab.ntt.co.jp> >> It doesn't work nor even compile since I failed to include some >> changes.

Re: pgbench doc fix

2018-10-30 Thread Tatsuo Ishii
Hi Fabien, > Ok, I understand that you mean that PQsendQueryParams uses an unamed > query internally to separate parsing & execution, which seems indeed > to be the case by looking at the libpq client-side code. > > However, if I'm not mistaken, the params version always sends and > possibly

Re: [HACKERS] generated columns

2018-10-30 Thread Sergei Kornilov
Hi > patch --dry-run --ignore-whitespace -p 0 -F 5 < > v5-0001-Generated-columns.patch > > and varied both -p and -F paramaters to no avail. Am I doing it wrong? I am able apply patch by command patch -p1 < v5-0001-Generated-columns.patch or by "git apply v5-0001-Generated-columns.patch", but

Re: COPY FROM WHEN condition

2018-10-30 Thread Surafel Temesgen
Hi, Thank you for looking at it . On Sun, Oct 28, 2018 at 7:19 PM Tomas Vondra wrote: > > 1) I think this deserves at least some regression tests. Plenty of tests > already use COPY, but there's no coverage for the new piece. So let's > add a new test suite, or maybe add a couple of tests into

Re: Continue work on changes to recovery.conf API

2018-10-30 Thread Sergei Kornilov
Hi I attached new version of this patch due merge conflict with pg_promote function. regards, Sergeidiff --git a/contrib/pg_standby/pg_standby.c b/contrib/pg_standby/pg_standby.c index ee1fbd7..946239c 100644 --- a/contrib/pg_standby/pg_standby.c +++ b/contrib/pg_standby/pg_standby.c @@ -611,7

Re: Installation instructions update (pg_ctl)

2018-10-30 Thread Andreas 'ads' Scherbaum
On 30.10.18 09:04, Michael Banck wrote: Hi, On Tue, Oct 30, 2018 at 12:08:49AM +0100, Andreas 'ads' Scherbaum wrote: The installation instructions (short version) are not consistent with the "initdb" output. The first one still uses "postgres -D", even mentions "check initdb output", but

PSA: rr recorder/debugger works well with Postgres + Linux

2018-10-30 Thread Peter Geoghegan
Heikki mentioned the rr recorder/debugger during his recent pgConf.EU talk. It was recommended as a general debugging tool. I can now second that recommendation. See https://rr-project.org for general background information. It's a framework that extends gdb, so it's largely compatible with

Re: Constraint documentation

2018-10-30 Thread David G. Johnston
The product name, when used in the documentation, is "PostgreSQL" with appropriate html elements surrounding it. Some parts that look or read oddly to me: "you may expect troubles" Use - if possible - (commas, not hypens, are customary here) "does not currently" - drop "currently", it doesn't and

Re: Online verification of checksums

2018-10-30 Thread Fabien COELHO
Hallo Michael, Patch v6 applies cleanly, compiles, local make check is ok. My current opinion is that when offline some errors are not admissible, whereas the same errors are admissible when online because they may be due to the ongoing database processing, so the behavior should not be

Re: replication_slots usability issue

2018-10-30 Thread Andres Freund
On 2018-10-30 11:51:09 +0900, Michael Paquier wrote: > On Mon, Oct 29, 2018 at 12:13:04PM -0700, Andres Freund wrote: > > I don't think this quite is the problem. ISTM the issue is rather that > > StartupReplicationSlots() *needs* to check whether wal_level > minimal, > > and doesn't. So you can

Re: replication_slots usability issue

2018-10-30 Thread Joshua D. Drake
On 10/30/18 10:52 AM, Andres Freund wrote: On 2018-10-30 11:51:09 +0900, Michael Paquier wrote: On Mon, Oct 29, 2018 at 12:13:04PM -0700, Andres Freund wrote: I don't think this quite is the problem. ISTM the issue is rather that StartupReplicationSlots() *needs* to check whether wal_level >

Re: [HACKERS] generated columns

2018-10-30 Thread Peter Eisentraut
On 30/10/2018 15:19, Erik Rijkers wrote: > On 2018-10-30 09:35, Peter Eisentraut wrote: > >> [v5-0001-Generated-columns.patch ] > > Hi, > > I couldn't get this to apply to current head. > > I tried: > > patch --dry-run --ignore-whitespace -p 0 -F 5 < > v5-0001-Generated-columns.patch > >

Re: Sequential UUID Generation

2018-10-30 Thread Adam Brusselback
I would be very interested in a extension which generated sequential uuids. My entire db is key'd with uuids, and I have measured some index bloat related specifically to random uuid generation. Thanks for bringing this up.

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-30 Thread Stephen Frost
Greetings, * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: > At Wed, 24 Oct 2018 14:31:37 +0900, Michael Paquier > wrote in <20181024053137.gl1...@paquier.xyz> > > On Sun, Oct 21, 2018 at 08:56:32PM -0400, Stephen Frost wrote: > > > All of this pie-in-the-sky about what pluggable

Re: Getting fancy errors when accessing information_schema on 10.5

2018-10-30 Thread Tom Lane
Axel Rau writes: >> Am 30.10.2018 um 16:04 schrieb Tom Lane : >> That is ... odd. Is it possible that you have cpu_operator_cost set >> to zero, or some very tiny number? > Yes: > cpu_index_tuple_cost = 0.01 > cpu_operator_cost = 0 Ah, well that explains why the clauses are seen as being the

Re: Getting fancy errors when accessing information_schema on 10.5

2018-10-30 Thread Axel Rau
> Am 30.10.2018 um 16:29 schrieb Tom Lane : > > Axel Rau writes: >>> Am 30.10.2018 um 16:04 schrieb Tom Lane : >>> That is ... odd. Is it possible that you have cpu_operator_cost set >>> to zero, or some very tiny number? > >> Yes: > >> cpu_index_tuple_cost = 0.01 >> cpu_operator_cost = 0

Function like "pg_trigger_depth" for Event Triggers

2018-10-30 Thread Fabrízio de Royes Mello
Hi all, There are some reason to don't have a similar function to return how many levels deep into an event trigger like we have using "pg_trigger_depth"?? Maybe one called "pg_event_trigger_depth"?? Regards, -- Fabrízio de Royes Mello Timbira - http://www.timbira.com.br/

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-30 Thread David Steele
On 10/30/18 11:59 AM, Stephen Frost wrote: > > * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: >> >> So I'm +1 for the Michael's current patch as (I think) we can't >> make visible or large changes. >> >> That said, I agree with Stephen's concern on the point we could >> omit

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread Tom Lane
David Rowley writes: > I've started working on something I've ended up calling "Super > PathKeys". The idea here is to increase the likelihood of a Path with > PathKeys being used for a purpose that requires a less strict sort > order due to ordering being required from the return value of some

Re: Lambda expressions (was Re: BUG #15471)

2018-10-30 Thread Tom Lane
Andres Freund writes: > On 2018-10-30 15:04:55 -0400, Tom Lane wrote: >> The core idea that I'm working on is to invent a new node type >> LambdaExpr that evaluates an expression and substitutes it as a >> Param into another expression, notationally sort of like >> LET($n := expression1 IN ...

Re: Lambda expressions (was Re: BUG #15471)

2018-10-30 Thread Andres Freund
On 2018-10-30 16:23:37 -0400, Tom Lane wrote: > Andres Freund writes: > > How did you deal with the fact that we might extract subset of the LET() > > into e.g. a RestrictionInfo (and then e.g. an IndexPath), but another > > part would be e.g. evaluated as part of a qual? > > Well, a Lambda

Re: replication_slots usability issue

2018-10-30 Thread Andres Freund
On 2018-10-30 11:02:04 -0700, Joshua D. Drake wrote: > On 10/30/18 10:52 AM, Andres Freund wrote: > > On 2018-10-30 11:51:09 +0900, Michael Paquier wrote: > > > On Mon, Oct 29, 2018 at 12:13:04PM -0700, Andres Freund wrote: > > > > I don't think this quite is the problem. ISTM the issue is rather

FDW Parallel Append

2018-10-30 Thread Sanyo Moura
Hi hackers, I am trying to improve my xdr_fdw (a foreign data wrapper that scan file systems that keep big data compacted) to scan partitions in parallel. I have set "IsForeignScanParallelSafe" to true and added path with "add_partial_path". My "GetForeignPaths" looks like code below: path =

Re: Lambda expressions (was Re: BUG #15471)

2018-10-30 Thread Tom Lane
Andres Freund writes: > On 2018-10-30 16:23:37 -0400, Tom Lane wrote: >> Well, a Lambda expression is not something that can be optimized away >> (unless perhaps you can get rid of the need for any of its output Params) >> so I don't see how any of its subexpressions would ever wind up split out

Re: Lambda expressions (was Re: BUG #15471)

2018-10-30 Thread Andres Freund
Hi, On 2018-10-30 16:54:45 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-10-30 16:23:37 -0400, Tom Lane wrote: > >> Well, a Lambda expression is not something that can be optimized away > >> (unless perhaps you can get rid of the need for any of its output Params) > >> so I don't

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-10-30 Thread Daniel Gustafsson
> On 11 Oct 2018, at 03:29, Michael Paquier wrote: Hi!, Thanks for reviewing this patch, and sorry for having been slow lately. > On Wed, Oct 10, 2018 at 02:20:53PM +0200, Daniel Gustafsson wrote: >>> On 9 Oct 2018, at 07:38, Michael Paquier wrote: >>> In order to make a test with non-ASCII

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
On 31 October 2018 at 08:52, Tom Lane wrote: > David Rowley writes: >> I've started working on something I've ended up calling "Super >> PathKeys". The idea here is to increase the likelihood of a Path with >> PathKeys being used for a purpose that requires a less strict sort >> order due to

Re: [HACKERS] logical decoding of two-phase transactions

2018-10-30 Thread Tomas Vondra
Hi Nikhil, Any progress on the issues discussed in the last couple of messages? That is: 1) removing of the sleep() from tests 2) changes to systable_getnext() wrt. TransactionIdIsInProgress() 3) adding asserts / checks to codepaths not going through systable_* 4) (not) adding this as a

Re: Ordered Partitioned Table Scans

2018-10-30 Thread David Rowley
On 31 October 2018 at 12:24, Julien Rouhaud wrote: > On Mon, Oct 29, 2018 at 1:44 AM David Rowley > wrote: >> >> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> > I just had a look at your patch. I see that you implemented only a >> > subset of the possible optimizations (only the case

Re: Ordered Partitioned Table Scans

2018-10-30 Thread Julien Rouhaud
On Mon, Oct 29, 2018 at 1:44 AM David Rowley wrote: > > On 28 October 2018 at 03:49, Julien Rouhaud wrote: > > I just had a look at your patch. I see that you implemented only a > > subset of the possible optimizations (only the case for range > > partitionoing without subpartitions). This has

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
On 31 October 2018 at 14:23, Tomas Vondra wrote: > The other thing likely affecting this is locale / collation. Probably > not for date_trunc, but certainly for things like substr()/trim(), > mentioned by Simon upthread. > > In some languages the rules are pretty complex, and there's no chance >

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread Tomas Vondra
Hi, On 10/30/2018 11:41 PM, David Rowley wrote: > On 31 October 2018 at 08:52, Tom Lane wrote: >> David Rowley writes: >>> I've started working on something I've ended up calling "Super >>> PathKeys". The idea here is to increase the likelihood of a Path with >>> PathKeys being used for a

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 21:27, Amit Langote wrote: > On Tue, Oct 30, 2018 at 7:11 PM Amit Langote >> I've tried to fix that with the attached patches. >> >> 0001 adds the ability for the callers of index_beginscan to specify a >> memory context. index_beginscan_internals switches to that context before >>

Re: FDW Parallel Append

2018-10-30 Thread Amit Langote
Hi, On 2018/10/31 3:25, Sanyo Moura wrote: > Hi hackers, > > I am trying to improve my xdr_fdw (a foreign data wrapper that scan file > systems that keep big data compacted) to scan partitions in parallel. > > But when I execute or analyze I get an error: > > EXPLAIN ANALYZE SELECT * FROM

Re: replication_slots usability issue

2018-10-30 Thread Michael Paquier
On Tue, Oct 30, 2018 at 10:52:54AM -0700, Andres Freund wrote: > On 2018-10-30 11:51:09 +0900, Michael Paquier wrote: >> Er... At the same time, shouldn't RestoreSlotFromDisk() *not* use PANIC >> if more slots are found in pg_replslot than max_replication_slots can >> handle. A FATAL is fine at

Re: ToDo: show size of partitioned table

2018-10-30 Thread Amit Langote
On 2018/10/30 20:03, Pavel Stehule wrote: > út 30. 10. 2018 v 7:52 odesílatel Amit Langote < > langote_amit...@lab.ntt.co.jp> napsal: >> Could one of you please revise the patch to use that function to produce >> the output of \dP+? >> > > here it is. > > It is based on Mathias's patch. Although

RE: libpq debug log

2018-10-30 Thread Iwata, Aya
Hi, I create a first libpq trace log patch. In this patch, - All message that PQtrace() gets are output to the libpq trace log file (I maybe select more effective message in the future patch) - Trace log output style is changed slightly from previously proposed This patch not include

RE: [PROPOSAL]a new data type 'bytea' for ECPG

2018-10-30 Thread Matsumura, Ryo
Hi Michael > > In Pro*C, the data should be represented as hex format C string. > > Just to clarify, there is no special datatype for binary data? I apology for lack of research again. Since it's a little difficult to answer, I explain by samples. The following works. unsigned char

Re: [HACKERS] Block level parallel vacuum

2018-10-30 Thread Masahiko Sawada
On Tue, Aug 14, 2018 at 9:31 AM Masahiko Sawada wrote: > > On Thu, Nov 30, 2017 at 11:09 AM, Michael Paquier > wrote: > > On Tue, Oct 24, 2017 at 5:54 AM, Masahiko Sawada > > wrote: > >> Yeah, I was thinking the commit is relevant with this issue but as > >> Amit mentioned this error is

Re: pgbench doc fix

2018-10-30 Thread Fabien COELHO
Hello Tatsuo-san, pgbench doc (and some comments in pgbench.c) regarding "-M prepared" option is not quite correct. [...] Actually "extended" mode uses prepared statements too. Ok, I understand that you mean that PQsendQueryParams uses an unamed query internally to separate parsing &

Re: Installation instructions update (pg_ctl)

2018-10-30 Thread Michael Banck
Hi, On Tue, Oct 30, 2018 at 12:08:49AM +0100, Andreas 'ads' Scherbaum wrote: > The installation instructions (short version) are not consistent with the > "initdb" output. The first one still uses "postgres -D", even mentions > "check initdb output", but "initdb" emits "pg_ctl" commands. > > The

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-30 Thread Krzysztof Nienartowicz
Thanks for both clarifications! I skimmed through the commits related to Inserts with partitioning since 10 and indeed - while not impossible it seems like quite some work to merge them into PG 10 codebase. We might consider preparing the patch in-house as otherwise PG 10 based partitioning is a