Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-08 Thread Jaime Casanova
On Wed, May 7, 2014 at 10:52 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Thu, May 8, 2014 at 5:30 AM, Jaime Casanova ja...@2ndquadrant.com wrote: Hi, This patch implements $subject only when ANALYZE and VERBOSE are on. I made it that way because for years nobody seemed interested in

Re: [HACKERS] [COMMITTERS] pgsql: Clean up jsonb code.

2014-05-08 Thread Heikki Linnakangas
On 05/08/2014 02:25 AM, Peter Geoghegan wrote: findJsonbValueFromSuperHeader()'s lowbound argument previously served to establish a low bound for searching when searching for multiple keys (so the second and subsequent user-supplied key could skip much of the object). Got that. In the case

Re: [HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Heikki Linnakangas
On 05/08/2014 08:01 AM, Michael Paquier wrote: Hi all, Since commit a692ee5, code compilation on windows is full of warnings caused by the re-definitions of popen and pclose: In file included from ../../../src/include/c.h:1028:0, from ../../../src/include/postgres.h:47,

Re: [HACKERS] Ignore files in src/interfaces/libpq generated by windows builds

2014-05-08 Thread Heikki Linnakangas
On 05/08/2014 08:48 AM, Michael Paquier wrote: Hi all, While doing some builds with mingw and 9.4, I noticed that a couple of files generated by build are not ignored in the source tree. Those two files are system.c and win32setlocale.c in src/interfaces/libpq. Please find attached a patch

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 04:33, Kouhei Kaigai kai...@ak.jp.nec.com wrote: From your description, my understanding is that you would like to stream data from 2 standard tables to the GPU, then perform a join on the GPU itself. I have been told that is not likely to be useful because of the data

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 03:36, Stephen Frost sfr...@snowman.net wrote: Given that I count 4-5 beneficial use cases for this index-like lookaside, it seems worth investing time in. I'm all for making use of MatViews and GPUs, but there's more than one way to get there and look-asides feels like pushing

Re: [HACKERS] [bug fix] pg_ctl always uses the same event source

2014-05-08 Thread MauMau
From: Amit Kapila amit.kapil...@gmail.com Only one minor suggestion: +Name of the event source for pg_ctl to use for event log. The +default is PostgreSQL. From this description, it is not clear when the event log will be used in pg_ctl. For example, if user uses -e option

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 12:19 AM, Craig Ringer cr...@2ndquadrant.com wrote: In terms of ugliness, would you be happier using a pg_extern instead of extern, where we: #ifdef WIN32 #define pg_extern extern PGDLLIMPORT #else #define pg_extern extern #endif ? I personally would not be

Re: [HACKERS] pg_shmem_allocations view

2014-05-08 Thread Robert Haas
On Wed, May 7, 2014 at 5:54 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-05-07 17:48:15 -0400, Robert Haas wrote: On Tue, May 6, 2014 at 6:09 PM, Andres Freund and...@2ndquadrant.com wrote: I guess I'd vote for ditching the allocated column completely and outputting the memory

Re: [HACKERS] PGDLLEXPORTing all GUCs?

2014-05-08 Thread Andres Freund
On 2014-05-08 07:56:46 -0400, Robert Haas wrote: On Thu, May 8, 2014 at 12:19 AM, Craig Ringer cr...@2ndquadrant.com wrote: In terms of ugliness, would you be happier using a pg_extern instead of extern, where we: #ifdef WIN32 #define pg_extern extern PGDLLIMPORT #else #define

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Robert Haas
On Wed, May 7, 2014 at 4:01 AM, Simon Riggs si...@2ndquadrant.com wrote: Agreed. My proposal is that if the planner allows the lookaside to an FDW then we pass the query for full execution on the FDW. That means that the scan, aggregate and join could take place via the FDW. i.e. Custom Plan

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: On 8 May 2014 03:36, Stephen Frost sfr...@snowman.net wrote: I'm all for making use of MatViews and GPUs, but there's more than one way to get there and look-asides feels like pushing the decision, unnecessarily, on to the user. I'm not sure I

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: As Stephen notes downthread, Tom has already expressed opposition to this idea on other threads, and I tend to agree with him, at least to some degree. I think the drive to use foreign data wrappers for PGStrom, CitusDB, and other things that

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 13:48, Stephen Frost sfr...@snowman.net wrote: We have multiple use cases where an alternate data structure could be used to speed up queries. I don't view on-GPU memory as being an alternate *permanent* data store. As I've said, others have expressed an interest in placing

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-08 Thread Fujii Masao
On Wed, May 7, 2014 at 4:57 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, May 6, 2014 at 7:04 PM, Christoph Berg c...@df7cb.de wrote: Hi, if you split configuration and data by setting data_directory, postgresql.auto.conf is writen to the data directory (/var/lib/postgresql/9.4/main

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 04:33, Kouhei Kaigai kai...@ak.jp.nec.com wrote: In case when it replaced join relations by ForeignScan, it will be almost same as expected ForeignScan with join-pushed down. Unlike usual table scan, it does not have actual relation definition on catalog, and its result

Re: [HACKERS] pg_shmem_allocations view

2014-05-08 Thread Andres Freund
On 2014-05-08 07:58:34 -0400, Robert Haas wrote: On Wed, May 7, 2014 at 5:54 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. Not sure what you're ACKing here ;). The idea of giving the unallocated memory a NULL key. Ok. A new version of the patches implementing that are attached.

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: On 8 May 2014 13:48, Stephen Frost sfr...@snowman.net wrote: I don't view on-GPU memory as being an alternate *permanent* data store. As I've said, others have expressed an interest in placing specific data on specific external resources that we

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Wed, May 7, 2014 at 4:01 AM, Simon Riggs si...@2ndquadrant.com wrote: Agreed. My proposal is that if the planner allows the lookaside to an FDW then we pass the query for full execution on the FDW. That means that the scan, aggregate and join could take place via the FDW. i.e. Custom

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: From here, it looks exactly like pushing a join into an FDW. If we had that, we wouldn't need Custom Scan at all. I may be mistaken and there is a critical difference. Local sub-plans doesn't sound like a big difference. Erm. I'm not sure that

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 14:32, Stephen Frost sfr...@snowman.net wrote: The API you've outlined requires users to specify on a per-relation basis what join types are valid. No, it doesn't. I've not said or implied that at any point. If you keep telling me what I mean, rather than asking, we won't get

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Bruce Momjian
On Tue, May 6, 2014 at 06:20:53PM -0400, Tom Lane wrote: David E. Wheeler da...@justatheory.com writes: On May 6, 2014, at 2:20 PM, Bruce Momjian br...@momjian.us wrote: Well, then, we only have a few days to come up with a name. What are the options? We have no proposals as yet.

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
Simon, Perhaps you've changed your proposal wrt LOOKASIDES's and I've missed it somewhere in the thread, but this is what I was referring to with my concerns regarding per-relation definition of 'LOOKASIDES': * Simon Riggs (si...@2ndquadrant.com) wrote: Roughly, I'm thinking of this...

Re: [HACKERS] New pg_lsn type doesn't have hash/btree opclasses

2014-05-08 Thread Andres Freund
Hi, On 2014-05-06 23:55:04 -0300, Fabrízio de Royes Mello wrote: If helps, I added some regression tests to the lastest patch. I think we should apply this patch now. It's much more sensible with the opclasses present and we don't win anything by waiting for 9.5. Greetings, Andres Freund --

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 14:40, Stephen Frost sfr...@snowman.net wrote: Allow me to re-state your suggestion here: An FDW is loaded which provides hook for join push-down (whatever those end up being). A query is run which joins *local* table A to *local* table B. Standard heaps, standard indexes,

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 14:49, Stephen Frost sfr...@snowman.net wrote: Your downthread comments on 'CREATE MATERIALIZED VIEW' are in the same vein, though there I agree that we need it per-relation as there are other trade-offs to consider (storage costs of the matview, cost to maintain the matview,

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Tue, May 6, 2014 at 06:20:53PM -0400, Tom Lane wrote: I wonder why there's not an option to store keys and values separately, but as hashes not as the original strings, so that indexability of everything could be guaranteed. Or a variant of that

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: On 8 May 2014 14:40, Stephen Frost sfr...@snowman.net wrote: Allow me to re-state your suggestion here: An FDW is loaded which provides hook for join push-down (whatever those end up being). A query is run which joins *local* table A to

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Bruce Momjian
On Thu, May 8, 2014 at 10:16:54AM -0400, Tom Lane wrote: Can we hash just the values, not the keys, in jsonb_ops, and hash the combo in jsonb_hash_ops. That would give us key-only lookups without a recheck. No, because there's nothing in JSON limiting the length of keys, any more than

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Andres Freund
On 2014-05-08 10:34:04 -0400, Bruce Momjian wrote: On Thu, May 8, 2014 at 10:16:54AM -0400, Tom Lane wrote: Can we hash just the values, not the keys, in jsonb_ops, and hash the combo in jsonb_hash_ops. That would give us key-only lookups without a recheck. No, because there's

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Bruce Momjian
On Thu, May 8, 2014 at 04:37:05PM +0200, Andres Freund wrote: On 2014-05-08 10:34:04 -0400, Bruce Momjian wrote: On Thu, May 8, 2014 at 10:16:54AM -0400, Tom Lane wrote: Can we hash just the values, not the keys, in jsonb_ops, and hash the combo in jsonb_hash_ops. That would give us

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-08 Thread Amit Kapila
On Thu, May 8, 2014 at 6:51 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, May 7, 2014 at 4:57 PM, Amit Kapila amit.kapil...@gmail.com wrote: This problem occurs because we don't have the value of data_directory set in postgresql.conf by the time we want to parse .auto.conf file during

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-08 Thread Stephen Frost
* Amit Kapila (amit.kapil...@gmail.com) wrote: On Thu, May 8, 2014 at 6:51 PM, Fujii Masao masao.fu...@gmail.com wrote: Should we just ignore such problematic setting in postgresql.auto.conf with warning message? Another way could be that we detect the same during server start (while

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-08 Thread Andres Freund
On 2014-05-08 22:21:43 +0900, Fujii Masao wrote: On Wed, May 7, 2014 at 4:57 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, May 6, 2014 at 7:04 PM, Christoph Berg c...@df7cb.de wrote: Hi, if you split configuration and data by setting data_directory, postgresql.auto.conf is

[HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Robert Haas
According to the documentation for PQputCopyEnd: The result is 1 if the termination data was sent, zero if it was not sent because the attempt would block (this case is only possible if the connection is in nonblocking mode), or -1 if an error occurred. (Use PQerrorMessage to retrieve

Re: [HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 05/08/2014 08:01 AM, Michael Paquier wrote: Since commit a692ee5, code compilation on windows is full of warnings caused by the re-definitions of popen and pclose: Hmm. Does the MinGW version of popen() and system() do the quoting for

Re: [HACKERS] Compilation errors with mingw build caused by undefined optreset

2014-05-08 Thread Tom Lane
Michael Paquier michael.paqu...@gmail.com writes: Since commit 60ff2fd introducing the centralizated getopt-related things in a global header file, build on Windows with mingw is failing Hm, buildfarm member narwhal doesn't seem to be having any such problem. (It's got its own issues, but not

Re: [HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 11:19 AM, Tom Lane wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: On 05/08/2014 08:01 AM, Michael Paquier wrote: Since commit a692ee5, code compilation on windows is full of warnings caused by the re-definitions of popen and pclose: Hmm. Does the MinGW version of

Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 2:31 AM, Jaime Casanova ja...@2ndquadrant.com wrote: On Wed, May 7, 2014 at 10:52 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Thu, May 8, 2014 at 5:30 AM, Jaime Casanova ja...@2ndquadrant.com wrote: Hi, This patch implements $subject only when ANALYZE and VERBOSE

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 15:25, Stephen Frost sfr...@snowman.net wrote: * Simon Riggs (si...@2ndquadrant.com) wrote: On 8 May 2014 14:40, Stephen Frost sfr...@snowman.net wrote: Allow me to re-state your suggestion here: An FDW is loaded which provides hook for join push-down (whatever those end up

Re: [HACKERS] Compilation errors with mingw build caused by undefined optreset

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 11:30 AM, Tom Lane wrote: Michael Paquier michael.paqu...@gmail.com writes: Since commit 60ff2fd introducing the centralizated getopt-related things in a global header file, build on Windows with mingw is failing Hm, buildfarm member narwhal doesn't seem to be having any such

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 7 May 2014 02:05, Kouhei Kaigai kai...@ak.jp.nec.com wrote: (1) DDL support and system catalog Simon suggested that DDL command should be supported to track custom- plan providers being installed, and to avoid nonsense hook calls if it is an obvious case that custom-plan provider can

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: According to the documentation for PQputCopyEnd: The result is 1 if the termination data was sent, zero if it was not sent because the attempt would block (this case is only possible if the connection is in nonblocking mode), or -1 if an error

Re: [HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I'm pretty sure we need this on Mingw - this SYSTEMQUOTE stuff dates back well before 8.3, IIRC, which is when we first got full MSVC support. I tried googling for some info on this, and got a number of hits suggesting that mingw didn't emulate popen

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-08 Thread Christoph Berg
Re: Andres Freund 2014-05-08 20140508145901.gb1...@awork2.anarazel.de Maybe this is nitpicking, but what happens when postgresql.auto.conf also includes the setting of data_directory? This is possible because we can set data_directory via ALTER SYSTEM now. Should we just ignore such

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Tomas Vondra
On 6.5.2014 23:01, Tomas Vondra wrote: On 6.5.2014 22:24, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: I recall there was a call for more animals with CLOBBER_CACHE_ALWAYS some time ago, so I went and enabled that on all three animals. Let's see how long that will take. I see there

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 12:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: According to the documentation for PQputCopyEnd: The result is 1 if the termination data was sent, zero if it was not sent because the attempt would block (this case is only possible

[HACKERS] A couple logical decoding fixes/patches

2014-05-08 Thread Andres Freund
Hi, Patch 01: Fix a couple of embarassing typos. Most of them harmless, but one isn't and can lead to crashing or decoding wrong data. Patch 02: Don't crash with an Assert() failure if wal_level=logical but max_replication_slots=0. Patch 03: Add valgrind suppression for writing out padding

[HACKERS] [PATCH] Fix harmless access to uninitialized memory in ri_triggers.c.

2014-05-08 Thread andres
From: Andres Freund and...@anarazel.de When cache invalidations arrive while ri_LoadConstraintInfo() is busy filling a new cache entry, InvalidateConstraintCacheCallBack() compares the - not yet initialized - oidHashValue field with the to-be-invalidated hash value. To fix check whether the entry

Re: [HACKERS] Recursive ReceiveSharedInvalidMessages not safe

2014-05-08 Thread Andres Freund
On 2014-05-05 15:41:22 -0400, Tom Lane wrote: After far, far too much confused head scratching, code reading, random elog()s et al I found out that this is just because of a deficiency in valgrind's undefinedness tracking. [...] Unfortunately this cannot precisely be caught by valgrind's

Re: [HACKERS] Recursive ReceiveSharedInvalidMessages not safe

2014-05-08 Thread Andres Freund
On 2014-05-05 18:50:59 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-05-05 15:41:22 -0400, Tom Lane wrote: Looks all right to me. Yeah, the right shift might have undefined high-order bits, but we don't care because we're storing the result into an int16.

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: What I'm now thinking I need to do is something like this: 1. If PQputCopyEnd returns -1, error. 2. while ((rc = PQflush(conn)) != 0) { if (rc 0) { error; } else { wait for socket to become read-ready or write-ready; } } 3. while (PQisBusy(conn)) {

Re: [HACKERS] popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 12:14 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: I'm pretty sure we need this on Mingw - this SYSTEMQUOTE stuff dates back well before 8.3, IIRC, which is when we first got full MSVC support. I tried googling for some info on this, and got a number of hits

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 12:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: What I'm now thinking I need to do is something like this: 1. If PQputCopyEnd returns -1, error. 2. while ((rc = PQflush(conn)) != 0) { if (rc 0) { error; } else { wait for socket

Re: [HACKERS] 9.4 release notes

2014-05-08 Thread MauMau
From: Bruce Momjian br...@momjian.us I have completed the initial version of the 9.4 release notes. You can view them here: http://www.postgresql.org/docs/devel/static/release-9-4.html Feedback expected and welcomed. I expect to be modifying this until we release 9.4 final. I have marked

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: OK. It still seems to me that there's a doc fix needed here, if nothing else. The documentation for PQputCopyEnd() clearly implies that if you get a return value of 1, the message is sent, and that's just not true. That's fair.

Re: [HACKERS] [WIP] showing index maintenance on EXPLAIN

2014-05-08 Thread Jaime Casanova
On Thu, May 8, 2014 at 10:41 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 8, 2014 at 2:31 AM, Jaime Casanova ja...@2ndquadrant.com wrote: On Wed, May 7, 2014 at 10:52 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Thu, May 8, 2014 at 5:30 AM, Jaime Casanova ja...@2ndquadrant.com

Re: [HACKERS] 9.4 release notes

2014-05-08 Thread Peter Geoghegan
On Thu, May 8, 2014 at 10:50 AM, MauMau maumau...@gmail.com wrote: Could you add the following item, client-only installation on Windows, if it's appropriate for release note? I think that it is appropriate. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Alvaro Herrera
Tomas Vondra wrote: H, with CLOBBER_CACHE_ALWAYS + CLOBBER_FREED_MEMORY the tests take ~20h on a single branch/animal. With a single locale (e.g. C) it would take ~4h, but we're testing a bunch of additional czech/slovak locales. The tests are running in sequence (magpie-treepie-fulmar)

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 1:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: OK. It still seems to me that there's a doc fix needed here, if nothing else. The documentation for PQputCopyEnd() clearly implies that if you get a return value of 1, the message is

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: It would seem normal and natural to have * pg_joinam catalog table for join methods with a join method API Which would include some way of defining which operators/datatypes we consider this for, so if PostGIS people come up with some fancy GIS

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 3:10 PM, Stephen Frost sfr...@snowman.net wrote: * Simon Riggs (si...@2ndquadrant.com) wrote: It would seem normal and natural to have * pg_joinam catalog table for join methods with a join method API Which would include some way of defining which operators/datatypes we

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 12:21 PM, Tomas Vondra wrote: On 6.5.2014 23:01, Tomas Vondra wrote: On 6.5.2014 22:24, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: I recall there was a call for more animals with CLOBBER_CACHE_ALWAYS some time ago, so I went and enabled that on all three animals.

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 20:40, Robert Haas robertmh...@gmail.com wrote: For my money, we'd be better off getting some kind of basic custom scan node functionality committed first, even if the cases where you can actually inject them into real plans are highly restricted. Then, we could later work on

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Alvaro Herrera
Andrew Dunstan wrote: Here is what I do on my FreeBSD VM. I have 2 animals, nightjar and friarbird. They have the same buildroot. friarbird is set up to build with CLOBBER_CACHE_ALWAYS, building just HEAD and just testing C locale; nightjar builds all branches we are interested in and tests

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 20:10, Stephen Frost sfr...@snowman.net wrote: * A USERSET mechanism to allow users to turn it off for testing or otherwise, at user, database level If we re-implement our existing components through this (eat our own dogfood as it were), I'm not sure that we'd be able to have

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I'm pretty skeptical about this whole line of inquiry. We've only got three kinds of joins, and each one of them has quite a bit of bespoke logic, and all of this code is pretty performance-sensitive on large join nests. If there's a way to make this

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 04:09 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: Here is what I do on my FreeBSD VM. I have 2 animals, nightjar and friarbird. They have the same buildroot. friarbird is set up to build with CLOBBER_CACHE_ALWAYS, building just HEAD and just testing C locale; nightjar builds

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 8 May 2014 20:40, Robert Haas robertmh...@gmail.com wrote: For my money, we'd be better off getting some kind of basic custom scan node functionality committed first, even if the cases where you can actually inject them into real plans are highly

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: We only consider merge joins if the join uses operators with oprcanmerge=true. We only consider hash joins if the join uses operators with oprcanhash=true So it seems reasonable to have a way to define/declare what is possible and what is not. But my

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 04:54 PM, Andrew Dunstan wrote: Why? This was actually discussed when I set this up and Tom opined that a once a day run with CLOBBER_CACHE_ALWAYS was plenty. It takes about 4 /12 hours. The rest of the time nightjar runs. friarbird runs a bit after midnight US East Coast

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, May 8, 2014 at 1:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: OK. It still seems to me that there's a doc fix needed here, if nothing else. The documentation for PQputCopyEnd() clearly implies that if

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Simon Riggs
On 8 May 2014 21:55, Tom Lane t...@sss.pgh.pa.us wrote: So I'm not real sure how we move forward. Maybe something to brainstorm about in Ottawa. I'm just about to go on away for a week, so that's probably the best place to leave (me out of) the discussion until Ottawa. I've requested some

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Alvaro Herrera
Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I wasn't objecting, merely an observation. Note that Tomas mentioned he's okay with running 4 builds at once. My main point here,

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Peter Geoghegan
On Thu, May 8, 2014 at 6:34 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Umm... I'm now missing the direction towards my goal. What approach is the best way to glue PostgreSQL and PGStrom? I haven't really paid any attention to PGStrom. Perhaps it's just that I missed it, but I would find it

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Andrew Dunstan
On 05/08/2014 05:21 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I wasn't objecting, merely an observation. Note that Tomas mentioned he's okay with

Re: [HACKERS] default opclass for jsonb (was Re: Call for GIST/GIN/SP-GIST opclass documentation)

2014-05-08 Thread Tom Lane
I wrote: I think the idea of hashing only keys/values that are too long is a reasonable compromise. I've not finished coding it (because I keep getting distracted by other problems in the code :-() but it does not look to be very difficult. I'm envisioning the cutoff as being something like

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-05-08 Thread David G Johnston
Tom Lane-2 wrote Robert Haas lt; robertmhaas@ gt; writes: On Thu, May 8, 2014 at 1:51 PM, Tom Lane lt; tgl@.pa gt; wrote: Robert Haas lt; robertmhaas@ gt; writes: OK. It still seems to me that there's a doc fix needed here, if nothing else. The documentation for PQputCopyEnd()

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Tomas Vondra
On 8.5.2014 23:48, Andrew Dunstan wrote: On 05/08/2014 05:21 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I wasn't objecting, merely an observation.

Re: [HACKERS] Issue with GRANT/COMMENT ON FUNCTION with default

2014-05-08 Thread Jim Nasby
On 5/5/14, 3:22 PM, Alvaro Herrera wrote: Jim Nasby wrote: Prior to default parameters on functions, GRANT and COMMENT accepted full parameter syntax. IE: GRANT EXECUTE ON test(t text) TO public as opposed to regprocedure, which only accepts the data types ( test(text), not test(t

Re: [HACKERS] Compilation errors with mingw build caused by undefined optreset

2014-05-08 Thread Michael Paquier
On Fri, May 9, 2014 at 12:55 AM, Andrew Dunstan and...@dunslane.net wrote: On 05/08/2014 11:30 AM, Tom Lane wrote: Michael Paquier michael.paqu...@gmail.com writes: Since commit 60ff2fd introducing the centralizated getopt-related things in a global header file, build on Windows with mingw is

Re: [HACKERS] Compilation errors with mingw build caused by undefined optreset

2014-05-08 Thread Michael Paquier
On Fri, May 9, 2014 at 12:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com writes: Since commit 60ff2fd introducing the centralizated getopt-related things in a global header file, build on Windows with mingw is failing Hm, buildfarm member narwhal doesn't

[HACKERS] Re: popen and pclose redefinitions causing many warning in Windows build

2014-05-08 Thread Noah Misch
On Thu, May 08, 2014 at 12:14:44PM -0400, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: I'm pretty sure we need this on Mingw - this SYSTEMQUOTE stuff dates back well before 8.3, IIRC, which is when we first got full MSVC support. I tried googling for some info on this, and

Re: [HACKERS] 9.4 release notes

2014-05-08 Thread Michael Paquier
On Fri, May 9, 2014 at 2:50 AM, MauMau maumau...@gmail.com wrote: From: Bruce Momjian br...@momjian.us I have completed the initial version of the 9.4 release notes. You can view them here: http://www.postgresql.org/docs/devel/static/release-9-4.html Feedback expected and welcomed. I

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 4:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: I thought that the executor side of his patch wasn't in bad shape. The real problems were in the planner, and indeed largely in the backend part of the planner where there's a lot of hard-wired logic for fixing up low-level

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
I also think that there are really two separate problems here: getting the executor to call a custom scan node when it shows up in the plan tree; and figuring out how to get it into the plan tree in the first place. I'm not sure we've properly separated those problems, and I'm not sure

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Thu, May 8, 2014 at 4:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: I thought that the executor side of his patch wasn't in bad shape. The real problems were in the planner, and indeed largely in the backend part of the planner where there's a lot of hard-wired logic for fixing up

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Thu, May 8, 2014 at 6:34 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Umm... I'm now missing the direction towards my goal. What approach is the best way to glue PostgreSQL and PGStrom? I haven't really paid any attention to PGStrom. Perhaps it's just that I missed it, but I would

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: On 8 May 2014 20:40, Robert Haas robertmh...@gmail.com wrote: For my money, we'd be better off getting some kind of basic custom scan node functionality committed first, even if the cases where you can actually inject them into real plans are

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
So it seems reasonable to have a way to define/declare what is possible and what is not. But my take is that adding a new column to pg_operator for every CustomJoin node is probably out of the question, hence my suggestion to list the operators we know it can work with. It does seem

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: On Thu, May 8, 2014 at 6:34 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Umm... I'm now missing the direction towards my goal. What approach is the best way to glue PostgreSQL and PGStrom? I haven't really paid any attention to PGStrom. Perhaps

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: Well, I consider that somewhat good news, because I think it would be rather nice if we could get by with solving one problem at a time, and if the executor part is close to being well-solved, excellent. Sadly, I'm afraid the news really isn't all

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I initially intended to allow extensions to add their custom-path based on their arbitrary decision, because the core backend cannot have expectation towards the behavior of custom-plan. However, of course, the custom-path that replaces built-in

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I initially intended to allow extensions to add their custom-path based on their arbitrary decision, because the core backend cannot have expectation towards the behavior of custom-plan. However, of course, the custom-path that replaces

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: GPU has workloads likes and dislikes. It is a reasonable idea to list up operators (or something else) that have advantage to run on custom-path. For example, numeric calculation on fixed-length variables has greate advantage on GPU, but locale

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I didn't ask this before but it's been on my mind for a while- how will this work for custom data types, ala the 'geometry' type from PostGIS? There's user-provided code that we have to execute to check equality for those, but they're not

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I didn't ask this before but it's been on my mind for a while- how will this work for custom data types, ala the 'geometry' type from PostGIS? There's user-provided code that we have to execute to check equality for those, but they're not

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Peter Geoghegan
On Thu, May 8, 2014 at 7:13 PM, Stephen Frost sfr...@snowman.net wrote: Of course, things will change when we are able to parallelize joins across multiple CPUs ourselves.. In a way, the PGStrom approach gets to cheat us today, since it can parallelize the work where core can't and that ends

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Stephen Frost
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: So the extension will need to be aware of all custom data types and then installed *after* all other extensions are installed? That doesn't strike me as workable... I'm not certain why do you think an extension will need to support all the

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Robert Haas
On Thu, May 8, 2014 at 10:16 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: Well, I consider that somewhat good news, because I think it would be rather nice if we could get by with solving one problem at a time, and if the executor part is close to

  1   2   >