Re: [HACKERS] Fixing pg_basebackup with tablespaces found in $PGDATA

2014-01-05 Thread Craig Ringer
On 01/02/2014 06:53 AM, Dimitri Fontaine wrote: As much as I've seen people frown upon $subject, it still happens in the wild I met a new case of it a couple of weeks ago, so I can certainly confirm that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] [PATCH] Support for pg_stat_archiver view

2014-01-05 Thread Magnus Hagander
On Sat, Jan 4, 2014 at 2:01 PM, Gabriele Bartolini gabriele.bartol...@2ndquadrant.it wrote: Il 04/01/14 13:25, Magnus Hagander ha scritto: With those two, I think it would make much sense to have a view like this. Ok, I will prepare version 2 with those. Oh, and you need to change

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Craig Ringer
On 01/04/2014 01:06 AM, Claudio Freire wrote: You're forgetting ORM workloads. I'm impressed that you've come up with an area where ORMs are beneficial ;-) JDBC also has a statement batching interface. Right now PgJDBC just unwraps the batch and runs each query individually. Any async-support

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Craig Ringer
On 01/04/2014 01:22 AM, Merlin Moncure wrote: Long term, I'd rather see an optimized 'ORM flush' assemble the data into a structured data set (perhaps a JSON document) and pass it to some receiving routine that decomposed it into records. The same is true on the input side. I'd much rather be

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Greg Stark
On Fri, Jan 3, 2014 at 3:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think Florian has a good point there, and the reason is this: what you are talking about will be of exactly zero use to applications that want to see the results of one query before launching the next. There are techniques

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Florian Weimer
On 01/05/2014 03:11 PM, Greg Stark wrote: On Fri, Jan 3, 2014 at 3:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think Florian has a good point there, and the reason is this: what you are talking about will be of exactly zero use to applications that want to see the results of one query before

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread Florian Weimer
On 01/04/2014 04:39 PM, Martijn van Oosterhout wrote: Why switch between COPY commands, why could you not do it in one? For example: COPY table1(col1, col2, ...), table2(col1, col2, ...) FROM STDIN WITH (tableoids); tableoid1tabcol1tabcol2... tableoid2tab... ... \. My originally idea

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread Robert Haas
On Sat, Jan 4, 2014 at 3:27 PM, knizhnik knizh...@garret.ru wrote: 1. I want IMCS to work with PostgreSQL versions not supporting DSM (dynamic shared memory), like 9.2, 9.3.1,... Yeah. If it's loaded at postmaster start time, then it can work with any version. On 9.4+, you could possibly make

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread james
On 05/01/2014 16:50, Robert Haas wrote: But on Windows, segments are*automatically* destroyed*by the operating system* when the last process unmaps them, so it's not quite so clear to me how we can allow it there. The main shared memory segment is no problem because the postmaster always has

[HACKERS] dynamic shared memory and locks

2014-01-05 Thread Robert Haas
One of the things that you might want to do with dynamic shared memory is store a lock in it. In fact, my bet is that almost everything that uses dynamic shared memory will want to do precisely that, because, of course, it's dynamic *shared* memory, which means that it is concurrently accessed by

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread Robert Haas
On Sun, Jan 5, 2014 at 12:34 PM, james ja...@mansionfamily.plus.com wrote: On 05/01/2014 16:50, Robert Haas wrote: But on Windows, segments are *automatically* destroyed *by the operating system* when the last process unmaps them, so it's not quite so clear to me how we can allow it there.

Re: [HACKERS] dynamic shared memory and locks

2014-01-05 Thread Andres Freund
On 2014-01-05 12:56:05 -0500, Robert Haas wrote: Right now, storing spinlocks in dynamic shared memory *almost* works, but there are problems with --disable-spinlocks. In that configuration, we use semaphores to simulate spinlocks. Every time someone calls SpinLockInit(), it's going to

Re: [HACKERS] [PATCH] Support for pg_stat_archiver view

2014-01-05 Thread Gabriele Bartolini
Il 05/01/14 13:52, Magnus Hagander ha scritto: Yes, that's what I'm talking about. And just increment it by 1. Done. I am attaching version 2 of the patch, which now implements only one function (pg_stat_get_archiver()) and adds: * failed attempts * WAL of the last failed attempt * time of the

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread knizhnik
From my point of view it is not a big problem that it is not possible to place LWLock in DSM. I can allocate LWLocks in standard way - using RequestAddinLWLocks and use them for synchronization. Concerning support of huge pages - actually I do not think that it should involve something more

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread james
On 05/01/2014 18:02, Robert Haas wrote: On Sun, Jan 5, 2014 at 12:34 PM, jamesja...@mansionfamily.plus.com wrote: On 05/01/2014 16:50, Robert Haas wrote: But on Windows, segments are*automatically* destroyed*by the operating system* when the last process unmaps them, so it's not quite so

Re: [HACKERS] dynamic shared memory and locks

2014-01-05 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: For what it's worth, my vote is currently for #2. I can't think of many interesting to do with dynamic shared memory without having at least spinlocks, so I don't think we'd be losing much. #1 seems needlessly unfriendly, #3 seems like a lot of work

Re: [HACKERS] [PATCH] Support for pg_stat_archiver view

2014-01-05 Thread Fabrizio Mello
Enviado via iPhone Em 05/01/2014, às 16:27, Gabriele Bartolini gabriele.bartol...@2ndquadrant.it escreveu: Il 05/01/14 13:52, Magnus Hagander ha scritto: Yes, that's what I'm talking about. And just increment it by 1. Done. I am attaching version 2 of the patch, which now implements only

Re: [HACKERS] dynamic shared memory and locks

2014-01-05 Thread Andres Freund
On 2014-01-05 14:06:52 -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: For what it's worth, my vote is currently for #2. I can't think of many interesting to do with dynamic shared memory without having at least spinlocks, so I don't think we'd be losing much. #1 seems

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Robert Haas
On Sat, Jan 4, 2014 at 1:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would suggest addressing Robert's concern about lack of error checking by refusing to allow a custom reloption to be set unless the relevant extension is loaded and checks it. Unlike the postgresql.conf problem, I don't see

Re: [HACKERS] In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier

2014-01-05 Thread Mark Dilger
I am building a regression test system for replication and came across this email thread.  I have gotten pretty far into my implementation, but would be happy to make modifications if folks have improvements to suggest.  If the community likes my design, or a modified version based on your

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 2:22 PM, Andres Freund wrote: It would only force serialization for transactions that modify tables covered by the assert, that doesn't seem to bad. Anything covered by an assert shoulnd't be modified frequently, otherwise you'll run into major performance problems. I think that

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 2:45 PM, Andres Freund wrote: Well, the patch's syntax seems to only allow to directly specify a SQL query to check - we could iterate over the querytree to gather all related tables and reject any function we do not understand. Creating an assertion scans the query tree and

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Peter Eisentraut
On 12/18/13, 4:04 PM, Andrew Dunstan wrote: Umm, that's really a major limitation in utility. We need to come up with a better answer than this, which would essentially hobble the facility. We don't have any facility to run a trigger on just any command, it needs to be triggered by a table.

Re: [HACKERS] more psprintf() use

2014-01-05 Thread Peter Eisentraut
On 1/2/14, 9:28 AM, Tom Lane wrote: Heikki is right and you are wrong. There is an ancient supposition that datatype output functions, in particular, always return palloc'd strings. I recently got rid of the pfree's in the main output path, cf commit

Re: [HACKERS] more psprintf() use

2014-01-05 Thread Peter Eisentraut
On 1/2/14, 2:12 PM, Alvaro Herrera wrote: Peter Eisentraut wrote: psprintf() in place of hardcoded palloc(N) + sprintf() and the like. +values[j++] = psprintf(%d, stat.blkno); +values[j++] = psprintf(%c, stat.type); +values[j++] = psprintf(%d, stat.live_items); +

Re: [HACKERS] GiST support for inet datatypes

2014-01-05 Thread Emre Hasegeli
2013-12-17 Emre Hasegeli e...@hasegeli.com: Query planner never chooses to use the index for the operators which the index is particularly useful because selectivity estimation functions are missing. I am planning to work on them. Attached patch adds selectivity estimation functions for the

Re: [HACKERS] more psprintf() use

2014-01-05 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On 1/2/14, 9:28 AM, Tom Lane wrote: Heikki is right and you are wrong. There is an ancient supposition that datatype output functions, in particular, always return palloc'd strings. I recently got rid of the pfree's in the main output path, cf commit

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Jan 4, 2014 at 1:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would suggest addressing Robert's concern about lack of error checking by refusing to allow a custom reloption to be set unless the relevant extension is loaded and checks it. Unlike

Re: [HACKERS] [PATCH] SQL assertions prototype

2014-01-05 Thread Andres Freund
Peter Eisentraut pete...@gmx.net schrieb: On 12/18/13, 2:22 PM, Andres Freund wrote: It would only force serialization for transactions that modify tables covered by the assert, that doesn't seem to bad. Anything covered by an assert shoulnd't be modified frequently, otherwise you'll run into

[HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier

2014-01-05 Thread Greg Stark
-- greg On 5 Jan 2014 14:54, Mark Dilger markdil...@yahoo.com wrote: I am building a regression test system for replication and came across this email thread. I have gotten pretty far into my implementation, but would be happy to make modifications if folks have improvements to suggest. If

Re: [HACKERS] In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier

2014-01-05 Thread Michael Paquier
On Mon, Jan 6, 2014 at 4:51 AM, Mark Dilger markdil...@yahoo.com wrote: I am building a regression test system for replication and came across this email thread. I have gotten pretty far into my implementation, but would be happy to make modifications if folks have improvements to suggest.

Re: [HACKERS] dynamic shared memory and locks

2014-01-05 Thread Robert Haas
On Sun, Jan 5, 2014 at 2:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: I seem to recall that there was some good reason for keeping all the LWLocks in an array, back when the facility was first designed. I'm too lazy to research the point right now, but you might want to go back and look at the

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Robert Haas
On Sun, Jan 5, 2014 at 3:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jan 4, 2014 at 1:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would suggest addressing Robert's concern about lack of error checking by refusing to allow a custom reloption to be

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread Robert Haas
On Sun, Jan 5, 2014 at 1:28 PM, knizhnik knizh...@garret.ru wrote: From my point of view it is not a big problem that it is not possible to place LWLock in DSM. I can allocate LWLocks in standard way - using RequestAddinLWLocks and use them for synchronization. Sure, well, that works fine if

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread Robert Haas
On Sun, Jan 5, 2014 at 1:44 PM, james ja...@mansionfamily.plus.com wrote: I'm intrigued - how are the handles shared between children that are peers in the current scheme? Some handle transfer must already be in place. That's up to the application. After calling dsm_create(), you call

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Fabrízio de Royes Mello
On Mon, Jan 6, 2014 at 1:08 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jan 5, 2014 at 3:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jan 4, 2014 at 1:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would suggest addressing Robert's

[HACKERS] Compiling extensions on Windows

2014-01-05 Thread Craig Ringer
Hi all Out of personal interest (in pain and suffering) I was recently looking into how to compile extensions out-of-tree on Windows using Visual Studio (i.e. no PGXS). It looks like the conventional answer to this is Do a source build of PG, compile your ext in-tree in contrib/, and hope the

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Jan 5, 2014 at 3:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: pg_dump creates extensions before tables, no? So what dump/restore hazard? Creating the extension doesn't guarantee that the shared library will always be loaded. No, but unless the

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-05 Thread Tom Lane
=?ISO-8859-1?Q?Fabr=EDzio_de_Royes_Mello?= fabriziome...@gmail.com writes: You are correct. pg_dump export reloptions using WITH clause of CREATE TABLE statement. I.e.: CREATE TABLE foo ( ) WITH (autovacuum_enabled=false, bdr.do_replicate=false); So if this statement checks for 'bdr'

Re: [HACKERS] Compiling extensions on Windows

2014-01-05 Thread Andrew Dunstan
On 01/05/2014 10:32 PM, Craig Ringer wrote: Hi all Out of personal interest (in pain and suffering) I was recently looking into how to compile extensions out-of-tree on Windows using Visual Studio (i.e. no PGXS). It looks like the conventional answer to this is Do a source build of PG,

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread Amit Kapila
On Sun, Jan 5, 2014 at 11:04 PM, james ja...@mansionfamily.plus.com wrote: On 05/01/2014 16:50, Robert Haas wrote: But on Windows, segments are *automatically* destroyed *by the operating system* when the last process unmaps them, so it's not quite so clear to me how we can allow it there.

Re: [HACKERS] RFC: Async query processing

2014-01-05 Thread David Fetter
On Fri, Jan 03, 2014 at 03:06:11PM -0200, Claudio Freire wrote: On Fri, Jan 3, 2014 at 12:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Claudio Freire klaussfre...@gmail.com writes: On Fri, Jan 3, 2014 at 10:22 AM, Florian Weimer fwei...@redhat.com wrote: Loading data into the database isn't