[HACKERS] pg_upgrade vs user created range type extension

2016-09-22 Thread Andrew Dunstan
I have just encountered an apparent bug in pg_upgrade (or possibly pg_dump). To recreate, install the cranges extension - which can be obtained via "git clone https://bitbucket.org/adunstan/pg-closed-ranges.git; - for both 9.4 and 9.5. Create a fresh 9.4 instance, create a database and in

Re: [HACKERS] pg_upgrade vs user created range type extension

2016-09-22 Thread Tom Lane
Andrew Dunstan writes: > I have just encountered an apparent bug in pg_upgrade (or possibly pg_dump). Hmm, it sort of looks like pg_dump believes it should dump the range's constructor function in binary-upgrade mode, while the backend is creating the constructor function

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-22 Thread Robert Haas
On Thu, Sep 22, 2016 at 7:44 PM, Tomas Vondra wrote: > I don't dare to suggest rejecting the patch, but I don't see how we could > commit any of the patches at this point. So perhaps "returned with feedback" > and resubmitting in the next CF (along with analysis of

Re: [HACKERS] 9.6 TAP tests and extensions

2016-09-22 Thread Craig Ringer
On 23 September 2016 at 00:32, Tom Lane wrote: > Craig Ringer writes: >> On 13 September 2016 at 22:02, Tom Lane wrote: >>> Without taking a position on the merits of this patch per se, I'd like >>> to say that I find the argument

Re: [HACKERS] Why postgres take RowExclusiveLock on all partition

2016-09-22 Thread Bruce Momjian
On Fri, Sep 16, 2016 at 09:56:39PM +0530, Sachin Kotwal wrote: > Hi Tom, > > What I understood from this https://www.postgresql.org/docs/9.5/static/ > explicit-locking.html#TABLE-LOCK-COMPATIBILITY > is : > > The RowExclusiveLock conflicts with queries want SHARE, SHARE ROW EXCLUSIVE, >

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

2016-09-22 Thread Peter Eisentraut
This is looking pretty good. More comments on this patch set: 0001: Keep the file order alphabetical in Mkvcbuild.pm. Needs nls.mk updates for file move (in initdb and pg_basebackup directories). 0002: durable_rename needs close(fd) in non-error path (compare backend code). Changing from

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Robert Haas
On Thu, Sep 22, 2016 at 7:10 PM, Thomas Munro wrote: > Interesting. OK, I agree that it'd be useful to show that we're > waiting because there's nothing happening, or waiting because the user > asked us to sleep, or waiting on IO, or waiting for an IPC response >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-22 Thread Tomas Vondra
On 09/23/2016 03:20 AM, Robert Haas wrote: On Thu, Sep 22, 2016 at 7:44 PM, Tomas Vondra wrote: I don't dare to suggest rejecting the patch, but I don't see how we could commit any of the patches at this point. So perhaps "returned with feedback" and resubmitting

Re: [HACKERS] pg_upgrade vs user created range type extension

2016-09-22 Thread Andrew Dunstan
On 09/22/2016 07:33 PM, Tom Lane wrote: Andrew Dunstan writes: I have just encountered an apparent bug in pg_upgrade (or possibly pg_dump). Hmm, it sort of looks like pg_dump believes it should dump the range's constructor function in binary-upgrade mode, while the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-22 Thread Tomas Vondra
On 09/21/2016 08:04 AM, Amit Kapila wrote: On Wed, Sep 21, 2016 at 3:48 AM, Tomas Vondra wrote: ... I'll repeat the test on the 4-socket machine with a newer kernel, but that's probably the last benchmark I'll do for this patch for now. Attached are results

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Thomas Munro
On Fri, Sep 23, 2016 at 1:49 AM, Robert Haas wrote: > On Wed, Sep 21, 2016 at 5:49 PM, Thomas Munro > wrote: >>> Moreover, it's pretty confusing that we have this general concept of >>> wait events in pg_stat_activity, and then here the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-22 Thread Amit Kapila
On Fri, Sep 23, 2016 at 5:14 AM, Tomas Vondra wrote: > On 09/21/2016 08:04 AM, Amit Kapila wrote: >> > > (c) Although it's not visible in the results, 4.5.5 almost perfectly > eliminated the fluctuations in the results. For example when 3.2.80 produced > this results

Re: [HACKERS] Showing parallel status in \df+

2016-09-22 Thread Rushabh Lathia
On Thu, Sep 22, 2016 at 10:04 PM, Tom Lane wrote: > Rushabh Lathia writes: > > I agree with the argument in this thread, having "Source code" as part > > of \df+ is bit annoying, specifically when output involve some really > > big PL language

Re: [HACKERS] Typo in libpq-int.h

2016-09-22 Thread Heikki Linnakangas
On 09/22/2016 04:35 PM, Daniel Gustafsson wrote: Ran into a typo in libpq-int.h while reading/hacking: - char *gsslib; /* What GSS librart to use ("gssapi" or + char *gsslib; /* What GSS library to use ("gssapi” or Patch

Re: [HACKERS] pageinspect: Hash index support

2016-09-22 Thread Amit Kapila
On Fri, Sep 23, 2016 at 9:40 AM, Peter Eisentraut wrote: > On 9/21/16 9:30 AM, Jesper Pedersen wrote: >> Attached is v5, which add basic page verification. > > There are still some things that I found that appear to follow the old > style (btree) conventions

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-09-22 Thread Amit Kapila
On Fri, Sep 23, 2016 at 7:17 AM, Tomas Vondra wrote: > On 09/23/2016 03:20 AM, Robert Haas wrote: >> >> On Thu, Sep 22, 2016 at 7:44 PM, Tomas Vondra >> wrote: >>> >>> I don't dare to suggest rejecting the patch, but I don't see how >>>

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Amit Kapila
On Fri, Sep 23, 2016 at 7:02 AM, Robert Haas wrote: > On Thu, Sep 22, 2016 at 7:10 PM, Thomas Munro > wrote: > >> I was thinking about suggesting a category "Replication" to cover the >> waits for client IO relating to replication, as opposed

Re: [HACKERS] Parallel sec scan in plpgsql

2016-09-22 Thread Amit Kapila
On Thu, Sep 22, 2016 at 7:32 PM, Robert Haas wrote: > On Thu, Sep 22, 2016 at 8:36 AM, Amit Kapila wrote: >> I think for certain cases like into clause, the rows passed will be >> equal to actual number of rows, otherwise it will generate error.

Re: [HACKERS] [RFC] Should we fix postmaster to avoid slow shutdown?

2016-09-22 Thread Tsunakawa, Takayuki
> From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > On Tue, Sep 20, 2016 at 2:20 AM, Tsunakawa, Takayuki > wrote: > > There's no apparent evidence to indicate the cause, but I could guess > > a few

Re: [HACKERS] pageinspect: Hash index support

2016-09-22 Thread Peter Eisentraut
On 9/21/16 9:30 AM, Jesper Pedersen wrote: > Attached is v5, which add basic page verification. There are still some things that I found that appear to follow the old style (btree) conventions instead the new style (brin, gin) conventions. - Rename hash_metap to hash_metapage_info. - Don't use

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Amit Kapila
On Thu, Sep 22, 2016 at 7:19 PM, Robert Haas wrote: > On Wed, Sep 21, 2016 at 5:49 PM, Thomas Munro > wrote: > > So, I tried to classify these. Here's what I came up with. > > Activity: ArchiverMain, AutoVacuumMain, BgWriterMain, >

Re: [HACKERS] Stopping logical replication protocol

2016-09-22 Thread Craig Ringer
On 19 September 2016 at 07:12, Vladimir Gordiychuk wrote: > New patch in attach. 0001 and 0002 without changes. > 0003 - patch contain improvements for pg_recvloginca, now pg_recvloginca > after receive SIGINT will send CopyDone package to postgresql and wait from > server

Re: [HACKERS] pg_ctl promote wait

2016-09-22 Thread Michael Paquier
On Fri, Sep 23, 2016 at 12:16 AM, Masahiko Sawada wrote: > On Thu, Sep 22, 2016 at 1:25 AM, Peter Eisentraut > wrote: >> On 8/11/16 9:28 AM, Michael Paquier wrote: >> Committed with that adjustment. Thanks... > Commit e7010ce seems to

[HACKERS] [PATCH] pgpassfile connection option

2016-09-22 Thread Julian Markwort
Hello psql-hackers! We thought it would be advantageous to be able to specify a 'custom' pgpassfile within the connection string along the lines of the existing parameters sslkey and sslcert. Which is exactly what this very compact patch does. The patch is minimally invasive - when no

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-22 Thread Heikki Linnakangas
On 09/22/2016 03:40 AM, Claudio Freire wrote: On Tue, Sep 20, 2016 at 3:34 PM, Claudio Freire wrote: The results seem all over the map. Some regressions seem significant (both in the amount of performance lost and their significance, since all 4 runs show a similar

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
For list partitions, the ListInfo stores the index maps for values i.e. the index of the partition to which the value belongs. Those indexes are same as the indexes in partition OIDs array and come from the catalogs. In case a user creates two partitioned tables with exactly same lists for

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-22 Thread Claudio Freire
On Thu, Sep 22, 2016 at 4:17 AM, Heikki Linnakangas wrote: > On 09/22/2016 03:40 AM, Claudio Freire wrote: >> >> On Tue, Sep 20, 2016 at 3:34 PM, Claudio Freire >> wrote: >>> >>> The results seem all over the map. Some regressions seem significant >>>

Re: [HACKERS] PL/Python adding support for multi-dimensional arrays

2016-09-22 Thread Pavel Stehule
Hi 2016-09-21 19:53 GMT+02:00 Dave Cramer : > > On 18 September 2016 at 09:27, Dave Cramer wrote: > >> >> On 10 August 2016 at 01:53, Pavel Stehule >> wrote: >> >>> Hi >>> >>> 2016-08-03 13:54 GMT+02:00 Alexey Grishchenko

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat wrote: > For list partitions, the ListInfo stores the index maps for values > i.e. the index of the partition to which the value belongs. Those > indexes are same as the indexes in partition OIDs array and come from

Re: [HACKERS] Confusing docs about GetForeignUpperPaths in fdwhandler.sgml

2016-09-22 Thread Rushabh Lathia
On Fri, Sep 2, 2016 at 5:00 PM, Etsuro Fujita wrote: > Hi Robert, > > Thanks for the comments! > > On 2016/09/02 11:55, Robert Haas wrote: > >> On Mon, Aug 1, 2016 at 5:44 PM, Etsuro Fujita >> wrote: >> >>> I noticed that the following

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2016-09-22 Thread Rajkumar Raghuwanshi
On Tue, Sep 20, 2016 at 4:26 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > PFA patch which takes care of some of the TODOs mentioned in my > previous mail. The patch is based on the set of patches supporting > declarative partitioning by Amit Langoted posted on 26th August. I

Re: [HACKERS] README of hash index

2016-09-22 Thread Amit Kapila
On Fri, Sep 16, 2016 at 9:56 PM, Jeff Janes wrote: > On Fri, Sep 16, 2016 at 4:20 AM, Amit Kapila > wrote: >> >> Currently README of hash module contain algorithms written in below form. >> >> The insertion algorithm is rather similar: >> >> pin

Re: [HACKERS] Better tracking of free space during SP-GiST index build

2016-09-22 Thread Tom Lane
Tomas Vondra writes: >> On 08/25/2016 01:45 AM, Tom Lane wrote: >>> I'll put this in the commitfest queue. It could use review from >>> someone with the time and motivation to do performance >>> testing/tuning. > I've been toying with this patch a bit today,

Re: [HACKERS] Exclude schema during pg_restore

2016-09-22 Thread Michael Banck
Hi, On Tue, Sep 20, 2016 at 08:59:37PM -0400, Peter Eisentraut wrote: > On 9/19/16 3:23 PM, Michael Banck wrote: > > Version 2 attached. > > Committed, thanks. Thanks! > I added the new option to the help output in pg_restore. Oh, sorry I missed that. Michael -- Michael Banck

Re: [HACKERS] Use of SizeOfIptrData - is that obsolete?

2016-09-22 Thread Tom Lane
Pavan Deolasee writes: > On Tue, Sep 20, 2016 at 8:34 PM, Tom Lane wrote: >> Realistically, because struct HeapTupleHeaderData contains a field of >> type ItemPointerData, it's probably silly to imagine that we can save >> anything if the compiler

Re: [HACKERS] Possibly too stringent Assert() in b-tree code

2016-09-22 Thread Tom Lane
Robert Haas writes: > On Mon, Sep 19, 2016 at 7:07 AM, Amit Kapila wrote: >> I think you have a valid point. It seems we don't need to write WAL >> for reuse page (aka don't call _bt_log_reuse_page()), if the page is >> new, as the only purpose of

Re: [HACKERS] gratuitous casting away const

2016-09-22 Thread Mark Dilger
> On Sep 22, 2016, at 9:14 AM, Tom Lane wrote: > > I'd call this kind of a wash, I guess. I'd be more excited about it if > the change allowed removal of an actual cast-away-of-constness somewhere. > > I suppose it's a bit of a chicken and egg situation, in that the lack >

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-22 Thread Kuntal Ghosh
On Wed, Sep 21, 2016 at 6:53 PM, Fabien COELHO wrote: > In front of the tps line. Well, the performance displayed could also be > improved... On my dual core SSD laptop I just got: > > sh> ./pgbench -c 10 -t 1000 > starting vacuum...end. > transaction type: > scaling

Re: [HACKERS] File system operations.

2016-09-22 Thread Yury Zhuravlev
On четверг, 15 сентября 2016 г. 19:09:16 MSK, Tom Lane wrote: Robert Haas writes: On Thu, Sep 15, 2016 at 11:01 AM, Anastasia Lubennikova wrote: What do you think about moving stuff from pg_upgrade/file.c to storage/file/ to allow reuse of

Re: [HACKERS] Parallel sec scan in plpgsql

2016-09-22 Thread Amit Kapila
On Tue, Sep 20, 2016 at 8:31 PM, Robert Haas wrote: > On Tue, Sep 20, 2016 at 9:24 AM, Amit Kapila wrote: >> I think here point is that for any case where there is count of rows >> to be selected, we disable parallelism. There are many genuine

[HACKERS] Executor's internal ParamExecData Params versus EvalPlanQual

2016-09-22 Thread Tom Lane
I looked into the misbehavior reported in bug #14328, https://www.postgresql.org/message-id/20160919160136.1348.55...@wrigleys.postgresql.org What is happening is that during the EvalPlanQual recheck to see whether the updated row still satisfies the SELECT's quals, we run ExecInitCteScan and

Re: [HACKERS] [PATCH] get_home_path: use HOME

2016-09-22 Thread Daniel Verite
Tom Lane wrote: > If we take this patch, what's to stop someone from complaining that we > broke *their* badly-designed system that abuses the HOME variable? POSIX warns against doing that, listing HOME in the variables that should be left to their intended usage:

Re: [HACKERS] gratuitous casting away const

2016-09-22 Thread Tom Lane
Mark Dilger writes: >> On Sep 20, 2016, at 1:06 PM, Tom Lane wrote: >> ... that seems to be discarding type information in order to add >> "const"; does not seem like a net benefit from here. > The following seems somewhere in between, with

Re: [HACKERS] [PATCH] pgpassfile connection option

2016-09-22 Thread Andrew Dunstan
On 09/22/2016 10:44 AM, Julian Markwort wrote: Hello psql-hackers! We thought it would be advantageous to be able to specify a 'custom' pgpassfile within the connection string along the lines of the existing parameters sslkey and sslcert. Which is exactly what this very compact patch

Re: [HACKERS] ICU integration

2016-09-22 Thread Alvaro Herrera
Petr Jelinek wrote: > > I'm not sure how well it will work to replace all the bits of LIKE and > > regular expressions with ICU API calls. One problem is that ICU likes > > to do case folding as a whole string, not by character. I need to do > > more research about that. > > Can't we use the

Re: [HACKERS] [PATCH] pgpassfile connection option

2016-09-22 Thread Julian Markwort
I haven't really thought about this as I had been asked to make this work as an additional option to the connection parameters... Now that I've looked at it - there is really only the benefit of saving the step of setting the PGPASSFILE environment variable. However, there might be cases in

Re: [HACKERS] ICU integration

2016-09-22 Thread Tom Lane
Alvaro Herrera writes: > Petr Jelinek wrote: >>> I'm not sure how well it will work to replace all the bits of LIKE and >>> regular expressions with ICU API calls. One problem is that ICU likes >>> to do case folding as a whole string, not by character. I need to do

Re: [HACKERS] pg_ctl promote wait

2016-09-22 Thread Masahiko Sawada
On Thu, Sep 22, 2016 at 1:25 AM, Peter Eisentraut wrote: > On 8/11/16 9:28 AM, Michael Paquier wrote: >> I have looked at them and the changes are looking fine for me. So I >> have switched the patch as ready for committer, aka you. >> >> Just a nit: >> +

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-22 Thread Heikki Linnakangas
On 08/02/2016 01:18 AM, Peter Geoghegan wrote: No merging in parallel -- Currently, merging worker *output* runs may only occur in the leader process. In other words, we always keep n worker processes busy with scanning-and-sorting (and maybe some merging), but then all

Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Windows service is not starting so there’s message in log: FATAL: "could not create shared memory segment “Global/PostgreSQL.851401618”: Permiss

2016-09-22 Thread Tom Lane
Robert Haas writes: > On Tue, Sep 20, 2016 at 12:53 PM, Tom Lane wrote: >> I'd be the first to agree that this point is inadequately documented >> in the code, but PostmasterRandom should be reserved for its existing >> security-related uses, not

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-22 Thread Robert Haas
On Thu, Sep 22, 2016 at 3:51 PM, Heikki Linnakangas wrote: > It'd be good if you could overlap the final merges in the workers with the > merge in the leader. ISTM it would be quite straightforward to replace the > final tape of each worker with a shared memory queue, so that the

Re: [HACKERS] Parallel sec scan in plpgsql

2016-09-22 Thread Robert Haas
On Thu, Sep 22, 2016 at 8:36 AM, Amit Kapila wrote: > I think for certain cases like into clause, the rows passed will be > equal to actual number of rows, otherwise it will generate error. So > we can pass that information in executor layer. Another kind of cases >

Re: [HACKERS] Executor's internal ParamExecData Params versus EvalPlanQual

2016-09-22 Thread Tom Lane
I wrote: > The reason this happens is that EvalPlanQualBegin copies down all the > es_param_exec_vals values from the outer query into the EPQ execution > state. That's OK for most uses of es_param_exec_vals values, but not > at all for cteParam Params, which are used as internal communication >

[HACKERS] Typo in libpq-int.h

2016-09-22 Thread Daniel Gustafsson
Ran into a typo in libpq-int.h while reading/hacking: - char *gsslib; /* What GSS librart to use ("gssapi" or + char *gsslib; /* What GSS library to use ("gssapi” or Patch attached. cheers ./daniel typo-libpq-int.diff

Re: [HACKERS] 9.6 TAP tests and extensions

2016-09-22 Thread Tom Lane
Craig Ringer writes: > On 13 September 2016 at 22:02, Tom Lane wrote: >> Without taking a position on the merits of this patch per se, I'd like >> to say that I find the argument for back-patching into 9.6 and not >> further than that to be pretty

Re: [HACKERS] Showing parallel status in \df+

2016-09-22 Thread Tom Lane
Rushabh Lathia writes: > I agree with the argument in this thread, having "Source code" as part > of \df+ is bit annoying, specifically when output involve some really > big PL language functions. Having is separate does make \df+ output more > readable. So I would vote

Re: [HACKERS] Hash Indexes

2016-09-22 Thread AP
On Wed, Sep 21, 2016 at 08:44:15PM +0100, Geoff Winkless wrote: > On 21 September 2016 at 13:29, Robert Haas wrote: > > I'd be curious what benefits people expect to get. > > An edge case I came across the other day was a unique index on a large > string: postgresql popped

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Robert Haas
On Thu, Sep 22, 2016 at 1:52 AM, Michael Paquier wrote: > Then comes the interesting bits: I don't think that it is a good idea > to associate only one event for a waiting point in the system views. > Say that at a waiting point WaitLatch is called with >

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Robert Haas
On Wed, Sep 21, 2016 at 5:49 PM, Thomas Munro wrote: >> Moreover, it's pretty confusing that we have this general concept of >> wait events in pg_stat_activity, and then here the specific type of >> wait event we're waiting for is the ... wait event kind. Uh, what?

Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Windows service is not starting so there’s message in log: FATAL: "could not create shared memory segment “Global/Po

2016-09-22 Thread Tom Lane
Amit Kapila writes: > On Tue, Sep 20, 2016 at 10:33 PM, Tom Lane wrote: >> ISTM both the previous coding and this version can fail for no good >> reason, that is what if GetLastError() happens to return one of these >> error codes as a result of some