Re: [HACKERS] Re: [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-14 Thread Tom Lane
Robert Haas writes: > On Sun, Feb 14, 2016 at 1:33 PM, Tom Lane wrote: >> pgprocno of the specific PGPROC, or of the group leader? If it's >> the former, I'm pretty suspicious that there are deadlock and/or >> linked-list-corruption hazards here. If

Re: [HACKERS] Bool btree_gin index not chosen on equality contraint, but on greater/lower?

2016-02-14 Thread Tom Lane
Patric Bechtel writes: > Tom Lane schrieb am 14.02.2016 um 17:51: >> I think your problem is that the planner won't apply >> match_boolean_index_clause() or >> expand_boolean_index_clause(), because it has a hard-wired idea of which >> index opclasses could >>

Re: [HACKERS] extend pgbench expressions with functions

2016-02-14 Thread Robert Haas
On Sun, Feb 14, 2016 at 11:28 AM, Fabien COELHO wrote: > Here is a 3 part v29: > > a) add support for integer functions in pgbench, including turning >operators as functions, as well as some minimal infrastructure for >additional types (this allows to minimize the

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-14 Thread Robert Haas
On Sun, Feb 14, 2016 at 12:19 AM, Masahiko Sawada wrote: > Thank you for reviewing this patch. > I've divided 000 patch into two patches, and attached latest 4 patches in > total. Thank you! I'll go through this again as soon as I have a free moment. -- Robert Haas

Re: [HACKERS] pl/pgSQL, get diagnostics and big data

2016-02-14 Thread Christian Ullrich
* Andreas 'ads' Scherbaum wrote: Attached is a new version of the patch, with %lu replaced by %zu. I re-ran all the tests, especially the long test with 2^32+x rows, and it produces the same result as before. To paraphrase Twain: "Sire, the Board finds this patch perfect in all the

[HACKERS] Refectoring of receivelog.c

2016-02-14 Thread Magnus Hagander
I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup, and rapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just passes an insane number of parameters around. Adding or changing

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-02-14 Thread David Rowley
On 14/02/2016 5:11 pm, "Tom Lane" wrote: > > David Rowley writes: > > On 12/02/2016 12:01 am, "Tom Lane" wrote: > > I can't quite understand what you're seeing here. > > The second loop is iterating through the original GROUP

Re: [HACKERS] Re: [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-14 Thread Tom Lane
Robert Haas writes: > On Sun, Feb 14, 2016 at 1:33 PM, Tom Lane wrote: >> Robert Haas writes: >>> ... the lock manager lock that protects the fields in a >>> given PGPROC is chosen by taking pgprocno modulo the number of lock >>>

Re: [HACKERS] Bool btree_gin index not chosen on equality contraint, but on greater/lower?

2016-02-14 Thread Patric Bechtel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Tom, Tom Lane schrieb am 14.02.2016 um 17:51: > Patric Bechtel writes: >> I tried to add bool support to the btree_gin contrib module, and as far as I >> can tell, it >> seems to work (wasn't that complicated,

Re: [HACKERS] [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-14 Thread Tom Lane
Robert Haas writes: > First, the overall concept here is that processes can either be a > member of a lock group or a member of no lock group. Check. > Second, I can review the data structure changes and the associated > invariants. The data structure additions seemed

[HACKERS] Re: [COMMITTERS] pgsql: Introduce group locking to prevent parallel processes from deadl

2016-02-14 Thread Robert Haas
On Sun, Feb 14, 2016 at 1:33 PM, Tom Lane wrote: > Robert Haas writes: >> First, the overall concept here is that processes can either be a >> member of a lock group or a member of no lock group. > > Check. > >> Second, I can review the data structure

Re: [HACKERS] Defaults for replication/backup

2016-02-14 Thread Magnus Hagander
On Sun, Feb 14, 2016 at 2:00 AM, Robert Haas wrote: > On Sat, Feb 13, 2016 at 11:31 AM, Andres Freund > wrote: > > On 2016-02-13 11:10:58 -0500, Tom Lane wrote: > >> Magnus Hagander writes: > >> > The big thing is, IIRC, that we

[HACKERS] Bool btree_gin index not chosen on equality contraint, but on greater/lower?

2016-02-14 Thread Patric Bechtel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I tried to add bool support to the btree_gin contrib module, and as far as I can tell, it seems to work (wasn't that complicated, actually). But now I'm stuck, as PostgreSQL doesn't seem to like to use my new index, if I use equality or

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Stephen Frost
Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > On Sat, Feb 13, 2016 at 3:05 AM, David Steele wrote: > > On 11/16/15 8:53 AM, Michael Paquier wrote: > >> On Sat, Sep 5, 2015 at 9:31 AM, Bruce Momjian wrote: > >>> On Fri, Sep 4, 2015 at 04:51:33PM -0400,

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 9:17 AM, Stephen Frost wrote: > Michael, > > * Michael Paquier (michael.paqu...@gmail.com) wrote: >> On Sat, Feb 13, 2016 at 3:05 AM, David Steele wrote: >> > On 11/16/15 8:53 AM, Michael Paquier wrote: >> >> On Sat, Sep 5, 2015 at

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Stephen Frost
Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > On Mon, Feb 15, 2016 at 9:17 AM, Stephen Frost wrote: > > That said, per various discussions, we'd really want a more-or-less > > fully formed patch to land prior to the last CF, for this to have any > > chance.

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 9:56 AM, Stephen Frost wrote: > * Michael Paquier (michael.paqu...@gmail.com) wrote: >> We'd need as well to switch pg_shadow to have an array of elements >> made of protocol:identifier instead of a single password field. There >> can be only one valid

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Stephen Frost
Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > It seems to me that applications are going to need a refresh anyway... Indeed. > Among the other possibilities I can foresee: > - Add a column "protocol" to pg_shadow and produce one row per > protocol, so one user will be listed

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Tom Lane
Stephen Frost writes: > Why do we need pg_shadow or pg_user or related views at all..? A lot of code looks at those just to get usernames. I am not in favor of breaking such stuff without need. How about we just say that the password in these old views always reads out as

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 10:23 AM, Stephen Frost wrote: > I would start by pointing out that pg_user currently uses pg_shadow.. > Why do we need pg_shadow or pg_user or related views at all..? pg_user/pg_shadow have the advantage to be world-readable and mask password values.

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Why do we need pg_shadow or pg_user or related views at all..? > > A lot of code looks at those just to get usernames. I am not in favor of > breaking such stuff without need. Alright. > How about we just

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Stephen Frost
* Michael Paquier (michael.paqu...@gmail.com) wrote: > On Mon, Feb 15, 2016 at 10:23 AM, Stephen Frost wrote: > > I would start by pointing out that pg_user currently uses pg_shadow.. > > Why do we need pg_shadow or pg_user or related views at all..? > > pg_user/pg_shadow

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> How about we just say that the password in these old views always reads >> out as '' even when there is a password, and we invent new views >> that carry real auth information? (Hopefully in an

Re: [HACKERS] WIP: SCRAM authentication

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 10:51 AM, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Stephen Frost writes: >> > Why do we need pg_shadow or pg_user or related views at all..? >> >> A lot of code looks at those just to get usernames. I am not

Re: [HACKERS] Using quicksort for every external sort run

2016-02-14 Thread Peter Geoghegan
On Sun, Feb 7, 2016 at 4:50 PM, Peter Geoghegan wrote: >> I'm not even sure this is necessary. The idea of missing out on >> producing a single sorted run sounds bad but in practice since we >> normally do the final merge on the fly there doesn't seem like there's >> really any

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-14 Thread Kyotaro HORIGUCHI
Hello, At Wed, 10 Feb 2016 18:36:43 +0900, Michael Paquier wrote in > On Wed, Feb 10, 2016 at 5:34 PM, Kyotaro HORIGUCHI > wrote: > > > > +sync_node_group: > > > > +

Re: [HACKERS] Incorrect formula for SysV IPC parameters

2016-02-14 Thread Kyotaro HORIGUCHI
Thanks for looking at this. At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao wrote in > >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS > >> under the Table 17-1. > > > >

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2016-02-14 Thread Etsuro Fujita
On 2016/02/12 21:46, Robert Haas wrote: On Fri, Feb 12, 2016 at 7:19 AM, Etsuro Fujita wrote: I think that displaying target lists would be confusing for users. Here is an example: EXPLAIN (verbose, costs off) DELETE FROM rem1; -- can be pushed

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 2:11 PM, Kyotaro HORIGUCHI wrote: > Surprizingly yes. The list is handled as an identifier list and > parsed by SplitIdentifierString thus it can accept double-quoted > names. Good point. I was not aware of this trick. -- Michael -- Sent via pgsql-hackers mailing list

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2016-02-14 Thread Kyotaro HORIGUCHI
Hello, thank you for reviewing this. > On Thu, Jan 7, 2016 at 3:36 AM, Kyotaro HORIGUCHI > wrote: > > - 0001-Prepare-for-sharing-psqlscan-with-pgbench.patch > > > > This diff looks a bit large but most of them is cut'n-paste > > work and the substantial

Re: [HACKERS] Refectoring of receivelog.c

2016-02-14 Thread Craig Ringer
On 15 February 2016 at 04:48, Magnus Hagander wrote: > I was working on adding the tar streaming functionality we talked about at > the developer meeting to pg_basebackup, and rapidly ran across the issue > that Andres has been complaining about for a while. The code in >

Re: [HACKERS] innocuous: pgbench does FD_ISSET on invalid socket

2016-02-14 Thread Michael Paquier
On Sat, Feb 13, 2016 at 6:25 AM, Alvaro Herrera wrote: > I noticed that pgbench calls FD_ISSET on a socket returned by > PQsocket() without first checking that it's not invalid. I don't think > there's a real problem here because the socket was already checked a few >

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2016-02-14 Thread Rushabh Lathia
On Fri, Feb 12, 2016 at 5:40 PM, Etsuro Fujita wrote: > Hi Rushabh and Thom, > > Thanks for the review! > > On 2016/02/10 22:37, Thom Brown wrote: > >> On 10 February 2016 at 08:00, Rushabh Lathia >> wrote: >> >>> Fujita-san, I am attaching

Re: [HACKERS] innocuous: pgbench does FD_ISSET on invalid socket

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 3:20 PM, Michael Paquier wrote: > On Sat, Feb 13, 2016 at 6:25 AM, Alvaro Herrera > wrote: >> I noticed that pgbench calls FD_ISSET on a socket returned by >> PQsocket() without first checking that it's not invalid. I

Re: [HACKERS] extend pgbench expressions with functions

2016-02-14 Thread Michael Paquier
On Mon, Feb 15, 2016 at 5:21 AM, Robert Haas wrote: > On Sun, Feb 14, 2016 at 11:28 AM, Fabien COELHO wrote: >> Here is a 3 part v29: >> >> a) add support for integer functions in pgbench, including turning >>operators as functions, as well as some

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2016-02-14 Thread Etsuro Fujita
On 2016/02/10 4:16, Robert Haas wrote: On Tue, Feb 9, 2016 at 8:39 AM, Ashutosh Bapat wrote: Thanks Jeevan for your review and comments. PFA the patch which fixes those. Committed with a couple more small adjustments. Thanks for working on this, Robert,

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-02-14 Thread Tom Lane
David Rowley writes: > On 12/02/2016 12:01 am, "Tom Lane" wrote: >> Um, AFAICS, you *do* need to check again in the second loop, else you'll >> be accessing a surplusvars[] entry that might not exist at all, and in >> any case might falsely tell

Re: [HACKERS] proposal: enhancing slow query log, and autoexplain log about waiting on lock before query exec time

2016-02-14 Thread Tom Lane
Pavel Stehule writes: > We have a patch, that inject logs about the time waiting on locks before > query execution. This feature helps us lot of, and I hope, it can be > generally useful. Doesn't log_lock_waits cover that territory already?

Re: [HACKERS] proposal: enhancing slow query log, and autoexplain log about waiting on lock before query exec time

2016-02-14 Thread Pavel Stehule
2016-02-14 17:46 GMT+01:00 Tom Lane : > Pavel Stehule writes: > > We have a patch, that inject logs about the time waiting on locks before > > query execution. This feature helps us lot of, and I hope, it can be > > generally useful. > > Doesn't

Re: [HACKERS] Bool btree_gin index not chosen on equality contraint, but on greater/lower?

2016-02-14 Thread Tom Lane
Patric Bechtel writes: > I tried to add bool support to the btree_gin contrib module, and as far as I > can tell, it seems to > work (wasn't that complicated, actually). > But now I'm stuck, as PostgreSQL doesn't seem to like to use my new index, if > I use equality or

Re: [HACKERS] extend pgbench expressions with functions

2016-02-14 Thread Fabien COELHO
Hello Michaƫl, I'll be happy if you do the review of the resulting split. OK, I am fine with this scenario as well. I have luckily done nothing yet. Here is a 3 part v29: a) add support for integer functions in pgbench, including turning operators as functions, as well as some minimal

[HACKERS] proposal: enhancing slow query log, and autoexplain log about waiting on lock before query exec time

2016-02-14 Thread Pavel Stehule
Hi, the interpretation of slow queries or entries from auto-explain log can be difficult some times, because the the main time of query evaluation is waiting on lock, and this number isn't in related entry. Our situation is little bit difficult, because we have not direct access to PostgreSQL

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-02-14 Thread David Rowley
On 12/02/2016 12:01 am, "Tom Lane" wrote: > > David Rowley writes: > > [ prune_group_by_clause_ab4f491_2016-01-23.patch ] > > [ check_functional_grouping_refactor.patch ] > > I've committed this with mostly-cosmetic revisions (principally,

Re: [HACKERS] extend pgbench expressions with functions

2016-02-14 Thread Michael Paquier
On Sun, Feb 14, 2016 at 4:42 PM, Fabien COELHO wrote: > >> So I would be fine to do a portion of the legwork and extract from this >> patch something smaller that adds only functions as a first step, with the >> minimum set of functions I mentioned upthread. Robert, Alvaro,