Re: [HACKERS] Parallel Seq Scan

2015-09-17 Thread Amit Kapila
On Thu, Sep 17, 2015 at 1:40 AM, Robert Haas wrote: > > On Thu, Sep 10, 2015 at 12:12 AM, Amit Kapila wrote: > >> 2. I think it's probably a good idea - at least for now, and maybe > >> forever - to avoid nesting parallel plans inside of other

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Shulgin, Oleksandr
On Wed, Sep 16, 2015 at 8:07 PM, Pavel Stehule wrote: > > 2015-09-16 16:31 GMT+02:00 Shulgin, Oleksandr < > oleksandr.shul...@zalando.de>: > >> >> I've added the timeout parameter to the pg_cmdstatus call, and more >> importantly to the sending side of the queue, so that

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 11:55 GMT+02:00 Shulgin, Oleksandr : > On Wed, Sep 16, 2015 at 8:07 PM, Pavel Stehule > wrote: > >> >> 2015-09-16 16:31 GMT+02:00 Shulgin, Oleksandr < >> oleksandr.shul...@zalando.de>: >> >>> >>> I've added the timeout parameter

Re: [HACKERS] a funnel by any other name

2015-09-17 Thread Nicolas Barbier
2015-09-17 Robert Haas : > 1. Exchange Bushy > 2. Exchange Inter-Operator (this is what's currently implemented) > 3. Exchange Replicate > 4. Exchange Merge > 5. Interchange > 1. ? > 2. Gather > 3. Broadcast (sorta) > 4. Gather Merge > 5. Redistribute > 1. Parallel Child

[HACKERS] LW_SHARED_MASK macro

2015-09-17 Thread Alexander Korotkov
Hackers, while exploring lwlock.c I found following macro to be strange. #define LW_SHARED_MASK ((uint32)(1 << 23)) This is macro is used to extract number of shared locks from state. ereport(LOG, (errhidestmt(true), errhidecontext(true), errmsg("%d: %s(%s): excl %u shared %u haswaiters %u

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Robert Haas
On Tue, Sep 15, 2015 at 5:00 AM, Shulgin, Oleksandr wrote: > Please see attached for implementation of this approach. The most > surprising thing is that it actually works :) It's cool to see these interesting ideas for using some of the code I've been working on

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Robert Haas
On Wed, Sep 16, 2015 at 10:31 AM, Shulgin, Oleksandr wrote: > I've also decided we really ought to suppress any possible ERROR level > messages generated during the course of processing the status request in > order not to prevent the originally running transaction

Re: [HACKERS] [patch] Proposal for \rotate in psql

2015-09-17 Thread Pavel Stehule
2015-09-08 22:55 GMT+02:00 Daniel Verite : > Pavel Stehule wrote: > > > rotate ~ sounds like transpose matrix, what is not true in this case. > > The various definitions that I can see, such as > http://dictionary.reference.com/browse/rotate > make no mention of

Re: [HACKERS] Parallel Seq Scan

2015-09-17 Thread Amit Kapila
On Tue, Sep 15, 2015 at 8:34 AM, Haribabu Kommi wrote: > > > > > I reviewed the parallel_seqscan_funnel_v17.patch and following are my comments. > I will continue my review with the parallel_seqscan_partialseqscan_v17.patch. > Thanks for the review. > + if

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-17 Thread Kyotaro HORIGUCHI
Hi, sorry in advance for hardly readable long descriptions.. At Mon, 14 Sep 2015 13:27:47 +0200, Tomas Vondra wrote in <55f6af33.8000...@2ndquadrant.com> > I don't think this is particularly related to the patch, because some > of the anomalies can be observed even

Re: [HACKERS] Parallel Seq Scan

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 2:54 AM, Amit Kapila wrote: > As per my understanding, what you have done there will not prohibit such > cases. > > +* For now, we don't try to use parallel mode if we're running inside > +* a parallel worker. We might eventually be able

[HACKERS] planstate_tree_walker

2015-09-17 Thread Robert Haas
In my reviews of Amit's parallel sequential scan patches yesterday, I complained that he was using planstate_tree_walker incorrectly, but I failed to notice that this was because he'd defined planstate_tree_walker incorrectly. This morning I figured that out. :-) Here is a patch that *just*

Re: [HACKERS] a funnel by any other name

2015-09-17 Thread Amit Kapila
On Thu, Sep 17, 2015 at 8:09 AM, Robert Haas wrote: > > Or, yet another option, we could combine the similar operators under > one umbrella while keeping the things that are more different as > separate nodes: > > 1, 2. Exchange (or Gather or Funnel) > 3, 5. Distribute (or

Re: [HACKERS] tsvector work with citext

2015-09-17 Thread Teodor Sigaev
However, it does seem like this function is not implementing its specification. Why isn't it just "IsBinaryCoercible(typid, TEXTOID)"? Oversight, I suppose. is_text_type() was introduced by commit 635aaab278afc1af972a4b6a55ff632ab763505d Author: Tom Lane Date: Tue Apr 8

Re: [HACKERS] honour TEMP_CONFIG in pg_upgrade tests

2015-09-17 Thread Andrew Dunstan
On 09/17/2015 09:42 AM, Tom Lane wrote: Andrew Dunstan writes: I propose to have the pg_upgrade test honour TEMP_CONFIG as pg_regress (or its Makefile) does, by applying the patch below. There might be opther places this should also be done, but this would be a start.

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 10:28 AM, Pavel Stehule wrote: > Please, can you explain what is wrong on using of shm_mq? It works pretty > well now. > > There can be a contra argument, why don't use shm_mq, because it does data > exchange between processes and this patch

Re: [HACKERS] [PATCH] Microvacuum for gist.

2015-09-17 Thread Teodor Sigaev
But I don't understand this comment: + /* +* While we delete only one tuple at once we could mix calls +* PageIndexTupleDelete() here and PageIndexMultiDelete() in +* gistRedoPageUpdateRecord() +*/ Does this mean:

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Shulgin, Oleksandr
On Thu, Sep 17, 2015 at 2:14 PM, Robert Haas wrote: > On Wed, Sep 16, 2015 at 10:31 AM, Shulgin, Oleksandr > wrote: > > I've also decided we really ought to suppress any possible ERROR level > > messages generated during the course of

Re: [HACKERS] creating extension including dependencies

2015-09-17 Thread Jeff Janes
On Tue, Sep 15, 2015 at 8:44 PM, Petr Jelinek wrote: > On 2015-09-08 04:06, Michael Paquier wrote: > >> On Tue, Sep 8, 2015 at 10:44 AM, Michael Paquier >> wrote: >> >>> >>> Attached are as well changes for the documentation that I spotted in >>>

Re: [HACKERS] [PATCH] Microvacuum for gist.

2015-09-17 Thread Jeff Janes
On Wed, Sep 16, 2015 at 8:36 AM, Teodor Sigaev wrote: > But It seems to me that it would be better to rewrite all mentions of >> TupleDelete to MultiDelete in gist code. >> > > Sure. Patch is attached, and it changes WAL format, so be carefull with > testing. > Please, have a

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-17 Thread Tomas Vondra
Hello Horiguchi-san, On 09/17/2015 12:45 PM, Kyotaro HORIGUCHI wrote: After all, what should be done to properly ignore useless conditions would be, 1. Make create_index_paths() to make the list of restrict clauses which are implied by the index predicate of the index in focus.

Re: [HACKERS] pg_resetxlog sentences

2015-09-17 Thread Alvaro Herrera
Fujii Masao wrote: > On Thu, Sep 17, 2015 at 5:00 AM, Alvaro Herrera > wrote: > > Robert Haas wrote: > >> On Tue, Sep 15, 2015 at 9:52 PM, Euler Taveira > >> wrote: > >> > While updating translations, I came across those almost similar > >> >

Re: [HACKERS] planstate_tree_walker

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 9:33 AM, Tom Lane wrote: > Robert Haas writes: >> Here is a patch that *just* introduces planstate_tree_walker and which >> is hopefully correct. I stole the logic from ExplainPreScanNode, >> which I also refactored to use the

[HACKERS] honour TEMP_CONFIG in pg_upgrade tests

2015-09-17 Thread Andrew Dunstan
I propose to have the pg_upgrade test honour TEMP_CONFIG as pg_regress (or its Makefile) does, by applying the patch below. There might be opther places this should also be done, but this would be a start. The motivation is to see if it stops the errors we often see on the somewhat slow

Re: [HACKERS] Improving test coverage of extensions with pg_dump

2015-09-17 Thread Michael Paquier
On Thu, Sep 17, 2015 at 7:24 AM, Andres Freund wrote: > On 2015-09-16 22:18:55 -0700, Michael Paquier wrote: >> Problem is similar with --column-inserts, --inserts and COPY. We could >> use --exclude-table like in the patch attached when taking the dump >> from source database but that's grotty,

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 16:46 GMT+02:00 Robert Haas : > On Thu, Sep 17, 2015 at 10:28 AM, Pavel Stehule > wrote: > > Please, can you explain what is wrong on using of shm_mq? It works pretty > > well now. > > > > There can be a contra argument, why don't use

Re: [HACKERS] planstate_tree_walker

2015-09-17 Thread Tom Lane
Robert Haas writes: > Here is a patch that *just* introduces planstate_tree_walker and which > is hopefully correct. I stole the logic from ExplainPreScanNode, > which I also refactored to use the new walker instead of duplicating > the logic. It seems a little odd to

Re: [HACKERS] Improving test coverage of extensions with pg_dump

2015-09-17 Thread Andres Freund
On 2015-09-16 22:18:55 -0700, Michael Paquier wrote: > So, here we go. Cool. > I have found something quite interesting when playing with the patch > attached: dump does not guarantee the column ordering across databases > for some inherited tables, see that example from the main regression >

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 16:06 GMT+02:00 Shulgin, Oleksandr : > On Thu, Sep 17, 2015 at 12:06 PM, Pavel Stehule > wrote: > >> >>> That won't work really well with something like I use to do when testing >>> this patch, namely: >>> >>> postgres=# select

Re: [HACKERS] honour TEMP_CONFIG in pg_upgrade tests

2015-09-17 Thread Tom Lane
Andrew Dunstan writes: > I propose to have the pg_upgrade test honour TEMP_CONFIG as pg_regress > (or its Makefile) does, by applying the patch below. There might be > opther places this should also be done, but this would be a start. No objection to the concept, but I

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 14:06 GMT+02:00 Robert Haas : > On Tue, Sep 15, 2015 at 5:00 AM, Shulgin, Oleksandr > wrote: > > Please see attached for implementation of this approach. The most > > surprising thing is that it actually works :) > > It's cool to

Re: [HACKERS] LW_SHARED_MASK macro

2015-09-17 Thread Andres Freund
Hi, On 2015-09-17 14:35:20 +0300, Alexander Korotkov wrote: > while exploring lwlock.c I found following macro to be strange. > > #define LW_SHARED_MASK ((uint32)(1 << 23)) > > This is macro is used to extract number of shared locks from state. > > ereport(LOG, > (errhidestmt(true), >

Re: [HACKERS] tsvector work with citext

2015-09-17 Thread Teodor Sigaev
Oversight, I suppose. is_text_type() was introduced by commit 635aaab278afc1af972a4b6a55ff632ab763505d Author: Tom Lane Date: Tue Apr 8 18:20:29 2008 + Fix tsvector_update_trigger() to be domain-friendly: it needs to allow all the columns it works with to be

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Shulgin, Oleksandr
On Thu, Sep 17, 2015 at 12:06 PM, Pavel Stehule wrote: > >> That won't work really well with something like I use to do when testing >> this patch, namely: >> >> postgres=# select pid, array(select pg_cmdstatus(pid, 1, 10)) from >> pg_stat_activity where

[HACKERS] Postgres releases scheduled for first week of October

2015-09-17 Thread Tom Lane
It's been over three months since our last set of back-branch releases. It also seems to be about time for a beta release of 9.5. The Postgres release team has accordingly decided to wrap 9.5beta1 as well as all active branches on Monday Oct 5, for public announcement Thursday Oct 8. Please note

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Shulgin, Oleksandr
On Thu, Sep 17, 2015 at 5:16 PM, Pavel Stehule wrote: > 2015-09-17 16:46 GMT+02:00 Robert Haas : > >> >> Second, using a shm_mq manipulates the state of the process latch. I >> don't think you can make the assumption that it's safe to reset the >>

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Tom Lane
Robert Haas writes: > To meet these needs, what I propose to do is have > set_plan_references() assign an integer ID to every plan node that is > unique across the entire plan tree. OK. > It would be nice if we could assign the integer IDs with no gaps, and > with the IDs

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Robert Haas
Thanks much for the quick reply. On Thu, Sep 17, 2015 at 2:23 PM, Tom Lane wrote: >> It would be nice if we could assign the integer IDs with no gaps, and >> with the IDs of a node's children immediately following that of their >> parent. > > I do not think this should be a

Re: [HACKERS] [patch] Proposal for \rotate in psql

2015-09-17 Thread Pavel Stehule
Hi 2015-09-16 11:35 GMT+02:00 Daniel Verite : > Hi, > > This is the 2nd iteration of this patch, for comments and review. > > > > my comments: 1. I don't understand why you are use two methods for sorting columns (qsort, and query with ORDER BY) 2. Data column are

Re: [HACKERS] Improving test coverage of extensions with pg_dump

2015-09-17 Thread Michael Paquier
On Thu, Sep 17, 2015 at 9:47 AM, Michael Paquier wrote: > COPY or INSERT include the column list in dumps, so that's not really > an issue here, what is potentially a problem (looking at that freshly) > is --inserts with data-only dumps though. So yes we had better fix it > and perhaps consider a

[HACKERS] numbering plan nodes

2015-09-17 Thread Robert Haas
Hi, One of the problems which the remaining parallel query patches need to solve is to correlate Plan or PlanState nodes across multiple backends. This need arises in a couple of cases. First, if a group of workers are all executing the same plan nodes, and es_instrument != 0, then we'll

Re: [HACKERS] pg_resetxlog sentences

2015-09-17 Thread Euler Taveira
On 17-09-2015 00:25, Fujii Masao wrote: One relevant question is; why doesn't pg_controldata report newestCommitTs? I thought about it while looking at the code but forgot to ask. AFAICS it is an oversight. See attached patch. -- Euler Taveira Timbira -

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 11:16 AM, Pavel Stehule wrote: > Is there some risk if we take too much large DSM segment? And what is max > size of DSM segment? When we use shm_mq, we don't need to solve limits. I can't really think you are going to have a problem. How much

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Tom Lane
Robert Haas writes: > On Thu, Sep 17, 2015 at 2:23 PM, Tom Lane wrote: >> I can't imagine I'd be doing anything that would break the simple case >> of "give every node a distinct ID". If you are building in weird >> assumptions about traversal order,

Re: [HACKERS] some pg_rewind usability issues

2015-09-17 Thread Michael Paquier
On Wed, Sep 16, 2015 at 9:46 PM, Michael Paquier wrote: > That's something that we discussed in this CF's patch to ease the > handling of timeline switches when rewinding a node, I wouldn't have > any objection to get that backpatched to 9.5 though (the > DB_SHUTDOWNED_IN_RECOVERY part I mean).

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Simon Riggs
On 17 September 2015 at 13:14, Robert Haas wrote: > My main concern with this design is how future-proof it is. > Passing array offsets sounds brittle to me. It would screw up any attempts to post-process the plans. Later enhancements seem certain to break that scheme.

Re: [HACKERS] tsvector work with citext

2015-09-17 Thread David E. Wheeler
On Sep 17, 2015, at 6:17 AM, Teodor Sigaev wrote: > I'm wrong, in this commit it was just renamed. It was originally coded by me. > But it's still oversight. Fixable? Thanks, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Kouhei Kaigai
> One of the problems which the remaining parallel query patches need to > solve is to correlate Plan or PlanState nodes across multiple > backends. This need arises in a couple of cases. First, if a group > of workers are all executing the same plan nodes, and es_instrument != > 0, then we'll

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-17 Thread Kyotaro HORIGUCHI
Hello, At Thu, 17 Sep 2015 17:40:27 +0200, Tomas Vondra wrote in <55fadeeb.4000...@2ndquadrant.com> > Yes, this seems sane. I've been poking at this a bit too, and I came > to the same plan in general, except that I think it's better to build > list of clauses that

Re: [HACKERS] [COMMITTERS] pgsql: Fix an O(N^2) problem in foreign key references.

2015-09-17 Thread Jan Wieck
On 09/15/2015 12:02 PM, Jan Wieck wrote: On 09/15/2015 11:54 AM, Tom Lane wrote: Jan Wieck writes: Would it be appropriate to use (Un)RegisterXactCallback() in case of detecting excessive sequential scanning of that cache and remove all invalid entries from it at that time?

Re: [HACKERS] cache type info in json_agg and friends

2015-09-17 Thread Andrew Dunstan
On 09/14/2015 03:41 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: Currently, json_agg, jsonb_agg, json_object_agg and jsonb_object_agg do type classification on their arguments on each call to the transition function. This is quite unnecessary, as the argument types won't change. This

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 4:22 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Sep 17, 2015 at 2:23 PM, Tom Lane wrote: >>> I can't imagine I'd be doing anything that would break the simple case >>> of "give every node a distinct

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 5:19 PM, Simon Riggs wrote: > Passing array offsets sounds brittle to me. How would this case be any different from other places in the planner where we already do exactly that? For example, RTIs. > It would screw up any attempts to post-process

Re: [HACKERS] creating extension including dependencies

2015-09-17 Thread Petr Jelinek
On 2015-09-17 17:31, Jeff Janes wrote: If I fail to specify CASCADE and get an ERROR, I think there should be a HINT which suggests the use of CASCADE. create extension earthdistance ; ERROR: required extension "cube" is not installed (no hint) There is a HINT on the reverse operation:

Re: [HACKERS] extend pgbench expressions with functions

2015-09-17 Thread Kyotaro HORIGUCHI
Hello, At Tue, 15 Sep 2015 19:26:51 +0200 (CEST), Fabien COELHO wrote in > > 1.evalInt and evalDouble are mutually calling on single expr > > node. > > Indeed. > > > It looks simple and seems to work but could easily broken > > to

Re: [HACKERS] numbering plan nodes

2015-09-17 Thread Robert Haas
On Thu, Sep 17, 2015 at 9:01 PM, Kouhei Kaigai wrote: > I entirely agree with the idea of plan-node identifier, however, > uncertain whether the node-id shall represent physical location on > the dynamic shared memory segment, because > (1) Relatively smaller number of node

Re: [HACKERS] vacuumdb sentence

2015-09-17 Thread Amit Kapila
On Fri, Sep 18, 2015 at 8:30 AM, Euler Taveira wrote: > > Hi, > > Is there a reason to quote "jobs" at this sentence? > > 190 fprintf(stderr, _("%s: number of parallel \"jobs\" must be at least 1\n"), > progname); > > AFAICS "jobs" is neither an identifier nor an option to

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 16:37 GMT+02:00 Pavel Stehule : > > > 2015-09-17 16:06 GMT+02:00 Shulgin, Oleksandr < > oleksandr.shul...@zalando.de>: > >> On Thu, Sep 17, 2015 at 12:06 PM, Pavel Stehule >> wrote: >> >>> That won't work really well with

Re: [HACKERS] vacuumdb sentence

2015-09-17 Thread Euler Taveira
On 18-09-2015 00:59, Amit Kapila wrote: On Fri, Sep 18, 2015 at 8:30 AM, Euler Taveira > wrote: > > Hi, > > Is there a reason to quote "jobs" at this sentence? > > 190 fprintf(stderr, _("%s: number of parallel \"jobs\" must be at least

[HACKERS] vacuumdb sentence

2015-09-17 Thread Euler Taveira
Hi, Is there a reason to quote "jobs" at this sentence? 190 fprintf(stderr, _("%s: number of parallel \"jobs\" must be at least 1\n"), progname); AFAICS "jobs" is neither an identifier nor an option to justify the quotation. Also, another message a few lines above (correctly) does not

Re: [HACKERS] Parallel Seq Scan

2015-09-17 Thread Amit Kapila
On Thu, Sep 17, 2015 at 6:58 AM, Robert Haas wrote: > > On Thu, Sep 3, 2015 at 6:21 AM, Amit Kapila wrote: > > [ new patches ] > > + pscan = shm_toc_lookup(node->ss.ps.toc, PARALLEL_KEY_SCAN); > > This is total nonsense. You can't

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-17 Thread Pavel Stehule
2015-09-17 22:13 GMT+02:00 Robert Haas : > On Thu, Sep 17, 2015 at 11:16 AM, Pavel Stehule > wrote: > > Is there some risk if we take too much large DSM segment? And what is max > > size of DSM segment? When we use shm_mq, we don't need to solve

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-09-17 Thread Fujii Masao
On Sat, Sep 5, 2015 at 7:48 PM, Petr Jelinek wrote: > On 2015-09-02 16:14, Fujii Masao wrote: >> >> On Wed, Aug 5, 2015 at 2:16 AM, Robert Haas wrote: >>> >>> On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao >>> wrote: