Re: [HACKERS] better atomics - v0.5

2014-07-11 Thread Amit Kapila
On Sat, Jul 12, 2014 at 1:26 AM, Martijn van Oosterhout wrote: > On Thu, Jul 10, 2014 at 08:46:55AM +0530, Amit Kapila wrote: > > As per my understanding of the general theory around barriers, > > read and write are defined to avoid reordering due to compiler and > > full memory barriers are defin

Re: [HACKERS] things I learned from working on memory allocation

2014-07-11 Thread Amit Kapila
On Fri, Jul 11, 2014 at 11:15 PM, Robert Haas wrote: > On Thu, Jul 10, 2014 at 1:05 AM, Amit Kapila wrote: > > If there is an noticeable impact, then do you think having separate > > file/infrastructure for parallel sort can help, basically non-parallel > > and parallel sort will have some common

Re: [HACKERS] Crash on backend exit w/ OpenLDAP [2.4.24, 2.4.31]

2014-07-11 Thread Noah Misch
On Thu, Jun 19, 2014 at 01:01:34PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Thu, Jun 12, 2014 at 05:02:19PM -0400, Noah Misch wrote: > >> You can cause the at-exit crash by building PostgreSQL against OpenLDAP > >> 2.4.31, connecting with LDAP authentication, and issuing "LOAD 'dblink'".

Re: [HACKERS] Allow multi-byte characters as escape in SIMILAR TO and SUBSTRING

2014-07-11 Thread Jeff Davis
On Fri, 2014-07-11 at 11:51 -0400, Tom Lane wrote: > Jeff Davis writes: > > Attached is a small patch to $SUBJECT. > > In master, only single-byte characters are allowed as an escape. Of > > course, with the patch it must still be a single character, but it may > > be multi-byte. > > I'm concerne

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-07-11 Thread Robert Haas
On Thu, Jul 10, 2014 at 2:52 AM, Tomonari Katsumata wrote: > Several couple weeks ago, I posted a mail to pgsql-doc. > http://www.postgresql.org/message-id/53992ff8.2060...@po.ntts.co.jp > > In this thread, I concluded that it's better to > round up the value which is less than its unit. > Current

Re: [HACKERS] better atomics - v0.5

2014-07-11 Thread Martijn van Oosterhout
On Thu, Jul 10, 2014 at 08:46:55AM +0530, Amit Kapila wrote: > As per my understanding of the general theory around barriers, > read and write are defined to avoid reordering due to compiler and > full memory barriers are defined to avoid reordering due to processors. > There are some processors th

[HACKERS] Over-optimization in ExecEvalWholeRowVar

2014-07-11 Thread Tom Lane
This example in the regression database is a simplified version of a bug I was shown off-list: regression=# select ( select q from ( select 1,2,3 where f1>0 union all select 4,5,6.0 where f1<=0 ) q ) from int4_tbl; ERROR: record type has not been registered The reason for the problem is that

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Fri, Jul 11, 2014 at 3:47 PM, Greg Stark wrote: > On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire > wrote: >> Marking as read-only is ok, or emitting a NOTICE so that if anyone >> changes those parameters that change the shape of the index, they know >> it needs a rebuild would be OK too. Bot

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Greg Stark
On Fri, Jul 11, 2014 at 6:00 PM, Claudio Freire wrote: > Marking as read-only is ok, or emitting a NOTICE so that if anyone > changes those parameters that change the shape of the index, they know > it needs a rebuild would be OK too. Both mechanisms work for me. We don't actually have any of the

Re: [HACKERS] RLS Design

2014-07-11 Thread Stephen Frost
Robert, On Friday, July 11, 2014, Robert Haas wrote: > On Fri, Jul 11, 2014 at 4:55 AM, Stephen Frost > wrote: > > My feeling at the moment is that having them be per-table makes sense and > > we'd still have flexibility to change later if we had some compelling > reason > > to do so. > > I don

Re: [HACKERS] things I learned from working on memory allocation

2014-07-11 Thread Robert Haas
On Thu, Jul 10, 2014 at 1:05 AM, Amit Kapila wrote: > On Tue, Jul 8, 2014 at 1:27 AM, Robert Haas wrote: >> 6. In general, I'm worried that it's going to be hard to keep the >> overhead of parallel sort from leaking into the non-parallel case. >> With the no-allocator approach, every place that u

Re: [HACKERS] RLS Design

2014-07-11 Thread Robert Haas
On Fri, Jul 11, 2014 at 4:55 AM, Stephen Frost wrote: > On Thursday, July 10, 2014, Robert Haas wrote: >> On Wed, Jul 9, 2014 at 2:13 AM, Stephen Frost wrote: >> > Yes, this would be possible (and is nearly identical to the original >> > patch, except that this includes per-role considerations),

Re: [HACKERS] Supporting Windows SChannel as OpenSSL replacement

2014-07-11 Thread Alvaro Herrera
Heikki Linnakangas wrote: > I did again the refactoring you did back in 2006, patch attached. > One thing I did differently: I moved the raw, non-encrypted, > read/write functions to separate functions: pqsecure_raw_read and > pqsecure_raw_write. Those functions encapsulate the SIGPIPE > handling.

Re: [HACKERS] tweaking NTUP_PER_BUCKET

2014-07-11 Thread Tomas Vondra
On 10.7.2014 21:33, Tomas Vondra wrote: > On 9.7.2014 16:07, Robert Haas wrote: >> On Tue, Jul 8, 2014 at 5:16 PM, Tomas Vondra wrote: >>> Thinking about this a bit more, do we really need to build the hash >>> table on the first pass? Why not to do this: >>> >>> (1) batching >>> - read the tu

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Alvaro Herrera
Sawada Masahiko wrote: > As you said, if line number reached UINT_MAX then I think that this > case is too strange. > I think INT_MAX is enough for line number. My point is not whether 2 billion is a better number than 4 billion as a maximum value. My point is that wraparound of signed int is, I

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Sawada Masahiko
On Fri, Jul 11, 2014 at 11:01 PM, Alvaro Herrera wrote: > Jeevan Chalke wrote: > >> On Fri, Jul 11, 2014 at 3:13 PM, Sawada Masahiko >> wrote: > >> > And the line number should be switched to 1 when line number has >> > reached to INT_MAX? >> >> Yes, when it goes beyond INT_MAX, wrap around to 1.

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Claudio Freire
On Thu, Jul 10, 2014 at 4:20 PM, Alvaro Herrera wrote: > Claudio Freire wrote: >> On Wed, Jul 9, 2014 at 6:16 PM, Alvaro Herrera >> wrote: >> > Another thing I noticed is that version 8 of the patch blindly believed >> > the "pages_per_range" declared in catalogs. This meant that if somebody >>

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread David Johnston
On Fri, Jul 11, 2014 at 12:12 PM, Michael Banck wrote: > On Fri, Jul 11, 2014 at 11:07:21AM -0400, Tom Lane wrote: > > David G Johnston writes: > > > Benedikt Grundmann wrote > > >> That is it possible to tell the planner that index is off limits > > >> i.e. > > >> don't ever generate a plan usi

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Michael Banck
On Fri, Jul 11, 2014 at 11:07:21AM -0400, Tom Lane wrote: > David G Johnston writes: > > Benedikt Grundmann wrote > >> That is it possible to tell the planner that index is off limits > >> i.e. > >> don't ever generate a plan using it? > > > Catalog hacking could work but not recommended (nor do

Re: [HACKERS] Allow multi-byte characters as escape in SIMILAR TO and SUBSTRING

2014-07-11 Thread Tom Lane
Jeff Davis writes: > Attached is a small patch to $SUBJECT. > In master, only single-byte characters are allowed as an escape. Of > course, with the patch it must still be a single character, but it may > be multi-byte. I'm concerned about the performance cost of this patch. Have you done any me

Re: [HACKERS] Allow multi-byte characters as escape in SIMILAR TO and SUBSTRING

2014-07-11 Thread Jeff Davis
On Fri, 2014-07-11 at 14:41 +0900, Fujii Masao wrote: > Could you add the patch into next CF? Sure. The patch is so small I was thinking about committing it in a few days (assuming no complaints), but I'm in no hurry. > The patch doesn't contain the change of the document. But I think that > it's

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Tom Lane
Andres Freund writes: > On 2014-07-11 11:20:08 -0400, Tom Lane wrote: >> If you're talking about SnapshotNow hazards, I think the risk would be >> minimal, and probably no worse than cases that the system will cause >> by itself. > Yes, SnapshotNow. I could reproduce it causing 'spurious' HOT upd

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Andres Freund
On 2014-07-11 11:20:08 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-07-11 11:07:21 -0400, Tom Lane wrote: > >> Hacking pg_index.indisvalid could work, given a reasonably recent PG. > >> I would not try it in production until I'd tested it ;-) > > > Works, but IIRC can cause problems

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Tom Lane
Andres Freund writes: > On 2014-07-11 11:07:21 -0400, Tom Lane wrote: >> Hacking pg_index.indisvalid could work, given a reasonably recent PG. >> I would not try it in production until I'd tested it ;-) > Works, but IIRC can cause problems at least < 9.4 because concurrent > cache builds might mi

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Andres Freund
On 2014-07-11 11:07:21 -0400, Tom Lane wrote: > David G Johnston writes: > > Benedikt Grundmann wrote > >> That is it possible to tell the planner that index is off limits i.e. > >> don't ever generate a plan using it? > > > Catalog hacking could work but not recommended (nor do I know the proper

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Tom Lane
David G Johnston writes: > Benedikt Grundmann wrote >> That is it possible to tell the planner that index is off limits i.e. >> don't ever generate a plan using it? > Catalog hacking could work but not recommended (nor do I know the proper > commands and limitations). Do you need the database/ta

Re: [HACKERS] Allowing NOT IN to use ANTI joins

2014-07-11 Thread Tom Lane
David Rowley writes: > On Fri, Jul 11, 2014 at 1:11 PM, Tom Lane wrote: >> Hm ... actually, there might be a better answer: what about transforming >> WHERE (x,y) NOT IN (SELECT provably-not-null-values FROM ...) >> to >> WHERE AND x IS NOT NULL AND y IS NOT NULL > I think this is the way to go

Re: [HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread David G Johnston
Benedikt Grundmann wrote > That is it possible to tell the planner that index is off limits i.e. > don't > ever generate a plan using it? > > Rationale: Schema changes on big tables. I might have convinced myself / > strong beliefs that for all queries that I need to be fast the planner > does >

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Alvaro Herrera
Jeevan Chalke wrote: > On Fri, Jul 11, 2014 at 3:13 PM, Sawada Masahiko > wrote: > > And the line number should be switched to 1 when line number has > > reached to INT_MAX? > > Yes, when it goes beyond INT_MAX, wrap around to 1. > > BTW, I wonder, can't we simply use unsigned int instead? Th

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-11 Thread Bruce Momjian
On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > > Uh, why does this need to be in ALTER TABLE? Can't this be part of > > table creation done by pg_dump? > > Uh, I think you need to read the thread. We have to delay the toast > creation part so we don't use an oid that will later

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-11 Thread Bruce Momjian
On Fri, Jul 11, 2014 at 12:18:40AM -0400, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > > > > I have thought some more on this. I thought I would need to open > > > pg_class in C and do complex backend stuff, but I now realize I

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Alvaro Herrera
Fujii Masao wrote: > On Thu, Jul 10, 2014 at 6:16 AM, Alvaro Herrera > wrote: > > Here's a new version of this patch, which is more generic the original > > versions, and similar to what you describe. > > I've not read the discussion so far at all, but I found the problem > when I played with th

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-07-11 Thread Rahila Syed
Thank you for review. >So, you're compressing backup blocks one by one. I wonder if that's the >right idea and if we shouldn't instead compress all of them in one run to >increase the compression ratio. The idea behind compressing blocks one by one was to keep the code as much similar to the origi

Re: [HACKERS] Allowing NOT IN to use ANTI joins

2014-07-11 Thread David Rowley
On Fri, Jul 11, 2014 at 1:11 PM, Tom Lane wrote: > I wrote: > > We could no doubt fix this by also insisting that the left-side vars > > be provably not null, but that's going to make the patch even slower > > and even less often applicable. I'm feeling discouraged about whether > > this is wort

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Jeevan Chalke
Hi, On Fri, Jul 11, 2014 at 3:13 PM, Sawada Masahiko wrote: > > > > > > To my understating cleanly, you means that line number is not changed > when newline has reached to INT_MAX, is incorrect? > As per my thinking yes. > And the line number should be switched to 1 when line number has > re

Re: [HACKERS] Missing autocomplete for CREATE DATABASE

2014-07-11 Thread Magnus Hagander
On Fri, Jul 11, 2014 at 12:01 AM, Vik Fearing wrote: > On 07/10/2014 09:32 PM, Magnus Hagander wrote: >> It seems psql is missing autocomplete entries for LC_COLLATE and >> LC_CTYPE for the CREATE DATABASE command. Attached patch adds that. >> >> I doubt this is important enough to backpatch - tho

Re: [HACKERS] tweaking NTUP_PER_BUCKET

2014-07-11 Thread Simon Riggs
On 11 July 2014 10:23, Tomas Vondra wrote: > On 11 Červenec 2014, 9:27, Simon Riggs wrote: >> On 9 July 2014 18:54, Tomas Vondra wrote: >> >>> (1) size the buckets for NTUP_PER_BUCKET=1 (and use whatever number >>> of batches this requires) >> >> If we start off by assuming NTUP_PER_BUCKET =

Re: [HACKERS] No exact/lossy block information in EXPLAIN ANALYZE for a bitmap heap scan

2014-07-11 Thread Fujii Masao
On Fri, Jul 11, 2014 at 5:45 PM, Etsuro Fujita wrote: > I've noticed that EXPLAIN ANALYZE shows no information on exact/lossy > blocks for a bitmap heap scan when both the numbers of exact/lossy pages > retrieved by the node are zero. Such an example is shown below. I > think it would be better

[HACKERS] [PATCH] Fix search_path default value separator.

2014-07-11 Thread Christoph Martin
Hi I noticed a minor inconsistency with the search_path separator used in the default configuration. The schemas of any search_path set using `SET search_path TO...` are separated by ", " (comma, space), while the default value is only separated by "," (comma). The attached patch against master

Re: [HACKERS] Allowing join removals for more join types

2014-07-11 Thread David Rowley
On Wed, Jul 9, 2014 at 12:59 PM, Tom Lane wrote: > David Rowley writes: > > On 9 July 2014 09:27, Tom Lane wrote: > >> On review it looks like analyzejoins.c would possibly benefit from an > >> earlier fast-path check as well. > > > Do you mean for non-subqueries? There already is a check to se

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Sawada Masahiko
On Fri, Jul 11, 2014 at 4:23 PM, Jeevan Chalke wrote: > Hi, > > A. > > However, this introduced new bug. As I told, when editor number of lines > reaches INT_MAX it starts giving negative number. You tried overcoming this > issue by adding "< 0" check. But I guess you again fumbled in setting that

Re: [HACKERS] Securing "make check" (CVE-2014-0067)

2014-07-11 Thread Christoph Berg
Re: To Bruce Momjian 2014-07-11 <20140711093923.ga3...@msg.df7cb.de> > Re: Bruce Momjian 2014-07-08 <20140708202114.gd9...@momjian.us> > > > > > I believe pg_upgrade itself still needs a fix. While it's not a > > > > > security problem to put the socket in $CWD while upgrading (it is > > > > > usin

Re: [HACKERS] Securing "make check" (CVE-2014-0067)

2014-07-11 Thread Christoph Berg
Re: Bruce Momjian 2014-07-08 <20140708202114.gd9...@momjian.us> > > > > I believe pg_upgrade itself still needs a fix. While it's not a > > > > security problem to put the socket in $CWD while upgrading (it is > > > > using -c unix_socket_permissions=0700), this behavior is pretty > > > > unexpecte

Re: [HACKERS] pg_receivexlog and replication slots

2014-07-11 Thread Andres Freund
On 2014-07-11 11:18:58 +0200, Magnus Hagander wrote: > On Fri, Jul 11, 2014 at 11:14 AM, Andres Freund > wrote: > > On 2014-07-11 11:08:48 +0200, Magnus Hagander wrote: > >> Is there a particular reason why pg_receivexlog only supports using > >> manually created slots but pg_recvlogical supports

Re: [HACKERS] tweaking NTUP_PER_BUCKET

2014-07-11 Thread Tomas Vondra
On 11 Červenec 2014, 9:27, Simon Riggs wrote: > On 9 July 2014 18:54, Tomas Vondra wrote: > >> (1) size the buckets for NTUP_PER_BUCKET=1 (and use whatever number >> of batches this requires) > > If we start off by assuming NTUP_PER_BUCKET = 1, how much memory does > it save to recalculate the

[HACKERS] Incorrect comment in postgres_fdw.c

2014-07-11 Thread Etsuro Fujita
I think the following comment for store_returning_result() in postgres_fdw.c is not right. /* PGresult must be released before leaving this function. */ I think PGresult should not be released before leaving this function *on success* in that function. (I guess the comment has been copied an

Re: [HACKERS] pg_receivexlog and replication slots

2014-07-11 Thread Magnus Hagander
On Fri, Jul 11, 2014 at 11:14 AM, Andres Freund wrote: > On 2014-07-11 11:08:48 +0200, Magnus Hagander wrote: >> Is there a particular reason why pg_receivexlog only supports using >> manually created slots but pg_recvlogical supports creating and >> dropping them? Wouldn't it be good for consiste

Re: [HACKERS] pg_receivexlog and replication slots

2014-07-11 Thread Andres Freund
On 2014-07-11 11:08:48 +0200, Magnus Hagander wrote: > Is there a particular reason why pg_receivexlog only supports using > manually created slots but pg_recvlogical supports creating and > dropping them? Wouldn't it be good for consistency there? I've added it to recvlogical because logical deco

[HACKERS] pg_receivexlog and replication slots

2014-07-11 Thread Magnus Hagander
Is there a particular reason why pg_receivexlog only supports using manually created slots but pg_recvlogical supports creating and dropping them? Wouldn't it be good for consistency there? I'm guessing it's related to not being exposed over the replication protocol? We had a discussion earlier th

Re: [HACKERS] RLS Design

2014-07-11 Thread Stephen Frost
On Thursday, July 10, 2014, Robert Haas wrote: > On Wed, Jul 9, 2014 at 2:13 AM, Stephen Frost > wrote: > > Yes, this would be possible (and is nearly identical to the original > > patch, except that this includes per-role considerations), however, my > > thinking is that it'd be simpler to work

[HACKERS] No exact/lossy block information in EXPLAIN ANALYZE for a bitmap heap scan

2014-07-11 Thread Etsuro Fujita
I've noticed that EXPLAIN ANALYZE shows no information on exact/lossy blocks for a bitmap heap scan when both the numbers of exact/lossy pages retrieved by the node are zero. Such an example is shown below. I think it would be better to suppress the 'Heap Blocks' line in that case, based on the s

Re: [HACKERS] [REVIEW] Re: Fix xpath() to return namespace definitions

2014-07-11 Thread Ali Akbar
Greetings, Attached modified patch to handle xmlCopyNode returning NULL. The patch is larger because xmlerrcxt must be passed to xml_xmlnodetoxmltype (xmlerrcxt is needed for calling xml_ereport). >From libxml2 source, it turns out that the other cases that xmlCopyNode will return NULL will not h

[HACKERS] Is there a way to temporarily disable a index

2014-07-11 Thread Benedikt Grundmann
That is it possible to tell the planner that index is off limits i.e. don't ever generate a plan using it? Rationale: Schema changes on big tables. I might have convinced myself / strong beliefs that for all queries that I need to be fast the planner does not need to use a given index (e.g. othe

Re: [HACKERS] tweaking NTUP_PER_BUCKET

2014-07-11 Thread Simon Riggs
On 9 July 2014 18:54, Tomas Vondra wrote: > (1) size the buckets for NTUP_PER_BUCKET=1 (and use whatever number > of batches this requires) If we start off by assuming NTUP_PER_BUCKET = 1, how much memory does it save to recalculate the hash bucket at 10 instead? Resizing sounds like it will

Re: [HACKERS] add line number as prompt option to psql

2014-07-11 Thread Jeevan Chalke
Hi, Found new issues with latest patch: > Thank you for reviewing the patch with variable cases. > I have revised the patch, and attached latest patch. > > > A: > > Will you please explain the idea behind these changes ? > I thought wrong about adding new to tail of query_buf. > The latest patch

Re: [HACKERS] inherit support for foreign tables

2014-07-11 Thread Etsuro Fujita
(2014/07/11 15:50), Etsuro Fujita wrote: (2014/07/10 18:12), Shigeru Hanada wrote: IIUC, you mean that tableoid can't be retrieved when a foreign table is accessed via parent table, No. What I want to say is that tableoid *can* be retrieved when a foreign table is accessed via the parent ta

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 10 July 2014 00:13, Alvaro Herrera wrote: > Josh Berkus wrote: >> On 07/09/2014 02:16 PM, Alvaro Herrera wrote: >> > The way it works now, each opclass needs to have three support >> > procedures; I've called them getOpers, maybeUpdateValues, and compare. >> > (I realize these names are pretty

Re: [HACKERS] Minmax indexes

2014-07-11 Thread Simon Riggs
On 9 July 2014 23:54, Peter Geoghegan wrote: > On Wed, Jul 9, 2014 at 2:16 PM, Alvaro Herrera > wrote: >> All this being said, I'm sticking to the name "Minmax indexes". There >> was a poll in pgsql-advocacy >> http://www.postgresql.org/message-id/53a0b4f8.8080...@agliodbs.com >> about a new na