Re: [HACKERS] Add pg_settings.pending_restart column

2015-02-15 Thread David G Johnston
Peter Eisentraut-2 wrote So here is a patch for that. It adds a column pending_restart to pg_settings that is true when the configuration file contains a changed setting that requires a restart. We already had the logic to detect such changes, for producing the log entry. I have also set it

Re: [HACKERS] KNN-GiST with recheck

2015-02-15 Thread Alexander Korotkov
On Thu, Jan 8, 2015 at 1:12 AM, Alexander Korotkov aekorot...@gmail.com wrote: On Tue, Dec 16, 2014 at 4:37 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Patch attached. It should be applied on top of my pairing heap patch at

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread David G Johnston
happy times wrote Sure, we can utilize the runtime parameter default_transaction_read_only, however, it does not restrict user from changing transaction attribute to non-readonly mode, so is not safe. ISTM that implementing a means to make this setting only super-user changeable would be a

Re: [HACKERS] forward vs backward slashes in msvc build code

2015-02-15 Thread Michael Paquier
On Sun, Feb 15, 2015 at 12:51 PM, Peter Eisentraut pete...@gmx.net wrote: I understand that on Windows, you can use forward slashes in path names interchangeably with backward slashes (except possibly in the shell). I have developed the attached patch to change the msvc build code to use

Re: [HACKERS] Logical Replication Helpers WIP for discussion

2015-02-15 Thread Petr Jelinek
On 13/02/15 14:04, Petr Jelinek wrote: On 13/02/15 08:48, Michael Paquier wrote: Looking at this patch, I don't see what we actually gain much here except a decoder plugin that speaks a special protocol for a special background worker that has not been presented yet. What actually is the value

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: A quick test suggests that initializing the hash value to ~0 rather than 0 has a curious effect: the number of batches still explodes, but the performance does not suffer the same way. (I think because almost all the batches end up empty.) I

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-02-15 Thread Kouhei Kaigai
The attached patch is a rebased version of join replacement with foreign-/custom-scan. Here is no feature updates at this moment but SGML documentation is added (according to Michael's comment). This infrastructure allows foreign-data-wrapper and custom-scan- provider to add alternative scan

[HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
Hi, 8b38a538c0aa5a432dacd90f10805dc667a3d1a0 changed things so that all columns are checked for NOT NULL ness. Which is fine in general, but it currently does make it impossible to have a varlena column (except OID/INT2 vector...) as a key column in syscache. Even if the column is factually NOT

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Andrew Dunstan
On 02/15/2015 11:47 AM, Sehrope Sarkuni wrote: For jsonb_indent, how about having it match up closer to the JavaScript JSON.stringify(value, replacer, space)[1]? That way a user can specify the indentation level and optionally filter the fields they'd like to output. In JS, the replacer

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2015-02-15 12:31:10 -0500, Tom Lane wrote: Where are you thinking of sticking that exactly, and will pgindent do something sane with it? Hm, I was thinking about /* extversion should never be null, but the others can be. */ text

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
On 2015-02-15 12:11:52 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: So, how about providing bootstrap infrastructure for marking columns as NOT NULL? We've not desperately needed it up to now, but if you can think of a clean representation, go for it. I'd want to

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I was thinking of adding BKI_FORCENOTNULL which would be specified on the attributes you want it. The FORCE in there representing that the default choice is overwritten. Where are you thinking of sticking that exactly, and will pgindent do something

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-15 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: What this patch does is add a GUC call old_snapshot_threshold. It defaults to -1, which leaves behavior matching unpatched code. Above that it allows tuples to be vacuumed

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Sehrope Sarkuni
For jsonb_indent, how about having it match up closer to the JavaScript JSON.stringify(value, replacer, space)[1]? That way a user can specify the indentation level and optionally filter the fields they'd like to output. In JS, the replacer parameter can be either a JS function or an array of

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: So, how about providing bootstrap infrastructure for marking columns as NOT NULL? We've not desperately needed it up to now, but if you can think of a clean representation, go for it. I'd want to preserve the property that all columns accessible via

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
On 2015-02-15 12:31:10 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I was thinking of adding BKI_FORCENOTNULL which would be specified on the attributes you want it. The FORCE in there representing that the default choice is overwritten. Where are you thinking of

Re: [HACKERS] gcc5: initdb produces gigabytes of _fsm files

2015-02-15 Thread Tom Lane
Christoph Berg c...@df7cb.de writes: gcc5 is lurking in Debian experimental, and it's breaking initdb. FYI, this is now fixed in Red Hat's rawhide version: https://bugzilla.redhat.com/show_bug.cgi?id=1190978 Don't know what the update process is like for Debian's copy, but maybe you could

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Florian Pflug
On Feb15, 2015, at 10:13 , David G Johnston david.g.johns...@gmail.com wrote: happy times wrote Sure, we can utilize the runtime parameter default_transaction_read_only, however, it does not restrict user from changing transaction attribute to non-readonly mode, so is not safe. ISTM that

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Andres Freund
On 2015-02-14 17:28:38 -0600, Jim Nasby wrote: Throw an error in AssignTransactionId/GetNewTransactionId? I see 4 calls to Get*TransactionId in logical replication, though arguably if we're fixing that we should look at doing something special for Slony and the likes. I don't think there are

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Magnus Hagander
On Sat, Feb 14, 2015 at 6:42 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Feb 14, 2015 at 7:29 AM, Magnus Hagander mag...@hagander.net wrote: Ok, I've pushed an attempt at doing this. For each mailthread, you can now create annotations. Each annotation is connected to a mail in

Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-15 Thread Tom Lane
Here's an updated version of the patch I sent before. Notable changes: * I switched over to calling deserialized objects expanded objects, and the default representation is now called flat or flattened instead of reserialized. Per suggestion from Robert. * I got rid of the bit about detecting

Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-15 Thread Robert Haas
On Sun, Feb 15, 2015 at 6:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: I'm going to stick this into the commitfest even though it's not really close to being committable; I see some other people doing likewise with their pet patches ;-). What it could particularly do with some reviewing help on

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: A quick test suggests that initializing the hash value to ~0 rather than 0 has a curious effect: the number of batches still explodes, but the performance does not suffer the same way. (I think because almost all the batches end up empty.) I

Re: [HACKERS] Replication identifiers, take 4

2015-02-15 Thread Andres Freund
Hi, Here's my next attept attempt at producing something we can agree upon. The major change that might achieve that is that I've now provided a separate method to store the origin_id of a node. I've made it conditional on !REPLICATION_IDENTIFIER_REUSE_PADDING, to show both paths. That new

Re: [HACKERS] Replication identifiers, take 4

2015-02-15 Thread Andres Freund
On 2015-02-16 01:21:55 +0100, Andres Freund wrote: Here's my next attept attempt at producing something we can agree upon. The major change that might achieve that is that I've now provided a separate method to store the origin_id of a node. I've made it conditional on

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE} 2.0

2015-02-15 Thread Peter Geoghegan
On Sat, Feb 14, 2015 at 2:06 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Thanks for taking a look at it. That's somewhat cleaned up in the attached patchseries - V2.2. This has been rebased to repair the minor bit-rot pointed out by Thom. I don't really have the energy to review

Re: [HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread Peter Eisentraut
On 2/15/15 6:31 PM, David Steele wrote: I had a problem installing the doc tools following the directions for OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. I'm running OSX Yosemite. I got it to work by changing the command from: sudo port install docbook-dsssl

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Eisentraut
On 2/14/15 7:30 AM, Magnus Hagander wrote: On Mon, Feb 9, 2015 at 4:56 PM, Robert Haas robertmh...@gmail.com Can we make it smarter, so that the kinds of things people produce intending for them to be patches are thought by the CF app to be patches? Doing it wouldn't be too

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Peter Eisentraut
On 2/14/15 7:24 PM, Tom Lane wrote: Another possibility that would be attractive for replication-related use-cases would be nothing that generates WAL thank you very much. This would be useful, as it essentially simulates a hot standby. -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Tomas Vondra
Hi, On 16.2.2015 00:50, Andrew Gierth wrote: Tom == Tom Lane t...@sss.pgh.pa.us writes: I've now tried the attached patch to correct the bucketsize estimates, and it does indeed stop the planner from considering the offending path (in this case it just does the join the other way round).

[HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
Hi! While playing with the memory context internals some time ago, I've been wondering if there are better ways to allocate memory from the kernel - either tweaking libc somehow, or maybe interacting with kernel directly. I mostly forgot about that topic, but after the local conference last week

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Geoghegan
On Sun, Feb 15, 2015 at 8:36 AM, Magnus Hagander mag...@hagander.net wrote: Hmm. This kind of looks like the right idea, but it's hard to use, because all you've got to work with is the subject of the message (which is the same for all), the time it was sent, and the author. In the old

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 20:56, Heikki Linnakangas wrote: On 02/15/2015 08:57 PM, Tomas Vondra wrote: One of the wilder ideas (I mentined beer was involved!) was a memory allocator based on mmap [2], bypassing the libc malloc implementation altogether. mmap() has some nice features (e.g. no issues with

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Andres Freund
On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: On 15.2.2015 20:56, Heikki Linnakangas wrote: On 02/15/2015 08:57 PM, Tomas Vondra wrote: One of the wilder ideas (I mentined beer was involved!) was a memory allocator based on mmap [2], bypassing the libc malloc implementation

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 21:13, Andres Freund wrote: On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: malloc() does that only for allocations over MAP_THRESHOLD, which is 128kB by default. Vast majority of blocks we allocate are = 8kB, so mmap() almost never happens. The problem is that mmap() is, to

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-15 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes: On 2/15/15 10:36 AM, Tom Lane wrote: I wonder if we couldn't achieve largely the same positive effects through adding a simple transaction-level timeout option. A common use-case is long-running reports hitting relatively stable data in a database

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: On 15.2.2015 20:56, Heikki Linnakangas wrote: glibc's malloc() also uses mmap() for larger allocations. Precisely because those allocations can then be handed back to the OS. I don't think we'd want

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-15 Thread Jim Nasby
On 2/15/15 10:36 AM, Tom Lane wrote: Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: What this patch does is add a GUC call old_snapshot_threshold. It defaults to -1, which leaves behavior matching unpatched code. Above

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Heikki Linnakangas
On 02/15/2015 08:57 PM, Tomas Vondra wrote: One of the wilder ideas (I mentined beer was involved!) was a memory allocator based on mmap [2], bypassing the libc malloc implementation altogether. mmap() has some nice features (e.g. no issues with returning memory back to the kernel, which may be

[HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
This came up today on IRC, though I suspect the general problem has been seen before: create table m3 (id uuid, status integer); create table q3 (id uuid); insert into m3 select uuid_generate_v4(), floor(random() * 4)::integer from generate_series(1,100); insert into q3 select id

Re: [HACKERS] assessing parallel-safety

2015-02-15 Thread Noah Misch
It's important for parallelism to work under the extended query protocol, and that's nontrivial. exec_parse_message() sets cursorOptions. exec_bind_message() runs the planner. We don't know if a parallel plan is okay before seeing max_rows in exec_execute_message(). On Sun, Feb 15, 2015 at

[HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread David Steele
I had a problem installing the doc tools following the directions for OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. I'm running OSX Yosemite. I got it to work by changing the command from: sudo port install docbook-dsssl docbook-sgml-4.2 docbook-xml-4.2 docbook-xsl

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 21:38, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: On 15.2.2015 20:56, Heikki Linnakangas wrote: glibc's malloc() also uses mmap() for larger allocations. Precisely because those allocations can then be handed

Re: [HACKERS] gcc5: initdb produces gigabytes of _fsm files

2015-02-15 Thread Christoph Berg
Re: Tom Lane 2015-02-15 21030.1424022...@sss.pgh.pa.us Christoph Berg c...@df7cb.de writes: gcc5 is lurking in Debian experimental, and it's breaking initdb. FYI, this is now fixed in Red Hat's rawhide version: https://bugzilla.redhat.com/show_bug.cgi?id=1190978 Don't know what the

Re: [HACKERS] Selectivity estimation for inet operators

2015-02-15 Thread Emre Hasegeli
New version of the patch attached with the optimization to break the loop before looking at all of the histogram values. I can reduce join selectivity estimation runtime by reducing the values of the left hand side or both of the sides, if there is interest. Even if the above aspects of the

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Andrew Dunstan
On 02/14/2015 10:06 PM, Andrew Dunstan wrote: Attached is a patch to provide a number of very useful facilities to jsonb that people have asked for. These are based on work by Dmitry Dolgov in his jsonbx extension, but I take responsibility for any bugs. The facilities are: new

[HACKERS] Rationalizing the API for array_ref, array_set, and friends

2015-02-15 Thread Tom Lane
The four functions array_ref, array_set, array_get_slice, array_set_slice have traditionally declared their array inputs and results as being of type ArrayType *. This is a lie, and has been since Berkeley days, because they actually also support fixed-length array types such as name and point.

Re: [HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread David Steele
On 2/15/15 7:50 PM, Peter Eisentraut wrote: On 2/15/15 6:31 PM, David Steele wrote: I had a problem installing the doc tools following the directions for OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. I'm running OSX Yosemite. I got it to work by changing the

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE} 2.0

2015-02-15 Thread Peter Geoghegan
On Fri, Feb 13, 2015 at 7:22 PM, Jim Nasby jim.na...@bluetreble.com wrote: In patch 1, sepgsql is also affected by this commit. Note that this commit necessitates an initdb, since stored ACLs are broken. Doesn't that warrant bumping catversion? Yes, but traditionally that is left until the

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
Tomas == Tomas Vondra tomas.von...@2ndquadrant.com writes: Tomas Improving the estimates is always good, but it's not going to Tomas fix the case of non-NULL values (it shouldn't be all that Tomas difficult to create such examples with a value whose hash starts Tomas with a bunch of zeroes).

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Geoghegan
On Sun, Feb 15, 2015 at 4:59 PM, Peter Eisentraut pete...@gmx.net wrote: I think the old system where the patch submitter declared, this message contains my patch, is the only one that will work. I tend to agree. That being said, calling out latest attachments is also useful (or highlighting

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2015-02-15 Thread Etsuro Fujita
On 2014/09/13 0:13, Tom Lane wrote: Albe Laurenz laurenz.a...@wien.gv.at writes: Tom Lane wrote: I'm not sure offhand what the new plan tree ought to look like. We could just generate a ForeignScan node, but that seems like rather a misnomer. Is it worth inventing a new ForeignUpdate node

Re: [HACKERS] Logical Replication Helpers WIP for discussion

2015-02-15 Thread Michael Paquier
On Sun, Feb 15, 2015 at 11:25 PM, Petr Jelinek p...@2ndquadrant.com wrote: On 13/02/15 14:04, Petr Jelinek wrote: On 13/02/15 08:48, Michael Paquier wrote: Looking at this patch, I don't see what we actually gain much here except a decoder plugin that speaks a special protocol for a

Re: [HACKERS] Commit fest 2015-12 enters money time

2015-02-15 Thread Michael Paquier
On Fri, Feb 13, 2015 at 10:06 AM, Michael Paquier michael.paqu...@gmail.com wrote: In order to move on to the next CF, I am going to go through all the remaining patches and update their status accordingly. And sorry for slacking a bit regarding that. If you wish to complain, of course feel

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Kouhei Kaigai
Hanada-san, Your patch mixtures enhancement of custom-/foreign-scan interface and enhancement of contrib/postgres_fdw... Probably, it is a careless mis- operation. Please make your patch as differences from my infrastructure portion. Also, I noticed this Join pushdown support for foreign tables

Re: [HACKERS] Allow snapshot too old error, to prevent bloat

2015-02-15 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Jim Nasby jim.na...@bluetreble.com writes: On 2/15/15 10:36 AM, Tom Lane wrote: I wonder if we couldn't achieve largely the same positive effects through adding a simple transaction-level timeout option. We suggested this to our customer and got out of the

Re: [HACKERS] pg_dump gets attributes from tables in extensions

2015-02-15 Thread Michael Paquier
On Mon, Feb 16, 2015 at 4:45 PM, Michael Paquier michael.paqu...@gmail.comwrote: but not dumping any tables that are part of ext_member. Excuse the typo = s/but/by/. -- Michael

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Shigeru Hanada
Hi I've revised the patch based on Kaigai-san's custom/foreign join patch posted in the thread below. http://www.postgresql.org/message-id/9a28c8860f777e439aa12e8aea7694f80108c...@bpxm15gp.gisp.nec.co.jp Basically not changed from the version in the last CF, but as Robert commented before,

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Shigeru Hanada
Kaigai-san, Oops. I rebased the patch onto your v4 custom/foreign join patch. But as you mentioned off-list, I found a flaw about inappropriate change about NestPath still remains in the patch... I might have made my dev branch into unexpected state. I'll check it soon. 2015-02-16 13:13

[HACKERS] pg_dump gets attributes from tables in extensions

2015-02-15 Thread Michael Paquier
Hi all, While looking at the patch to fix pg_dump with extensions containing tables referencing each other, I got surprised by the fact that getTableAttrs tries to dump table attributes even for tables that are part of an extension. Is that normal? Attached is a patch that I think makes things