Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
On 08/03/2015 04:25 PM, Ildus Kurbangaliev wrote: On 07/28/2015 10:28 PM, Heikki Linnakangas wrote: On 07/27/2015 01:20 PM, Ildus Kurbangaliev wrote: Hello. In the attached patch I've made a refactoring for tranches. The prefix for them was extended, and I've did a split of LWLockAssign to

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread Ashutosh Bapat
On Tue, Aug 4, 2015 at 4:00 PM, David Rowley david.row...@2ndquadrant.com wrote: On 4 August 2015 at 21:56, Ashutosh Bapat ashutosh.ba...@enterprisedb.com wrote: This looks like one example of general problem of finding optimal order for SQL operations. Consider a query defined as

Re: [HACKERS] [PATCH] postgres_fdw extension support

2015-08-04 Thread Paul Ramsey
Thanks so much Michael! Let me know when you have further feedback I should incorporate. ATB, P.  --  http://postgis.net http://cleverelephant.ca On July 25, 2015 at 4:52:11 AM, Michael Paquier (michael.paqu...@gmail.com) wrote: On Sat, Jul 25, 2015 at 2:19 AM, Paul Ramsey

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
On 08/04/2015 03:15 PM, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangashlinn...@iki.fi wrote: * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync with the list of individual locks in lwlock.h. Sooner or later someone will add an LWLock and forget

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 06:03, Jeff Janes jeff.ja...@gmail.com wrote: The attached proof of concept patch greatly improves the bloat for both the insert and the update cases. You need to turn on both features: adding the pages to fsm, and vacuuming the fsm, to get the benefit (so JJ_GIN=3). The

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2015-08-04 Thread Shigeru Hanada
Hi Ashutosh, Sorry for leaving the thread. 2015-07-20 16:09 GMT+09:00 Ashutosh Bapat ashutosh.ba...@enterprisedb.com: In find_user_mapping(), if the first cache search returns a valid tuple, it is checked twice for validity, un-necessarily. Instead if the first search returns a valid tuple,

Re: [HACKERS] cost_agg() with AGG_HASHED does not account for startup costs

2015-08-04 Thread Tom Lane
David Rowley david.row...@2ndquadrant.com writes: During working on allowing the planner to perform GROUP BY before joining I've noticed that cost_agg() completely ignores input_startup_cost when aggstrategy == AGG_HASHED. Isn't your proposed patch double-counting the input startup cost?

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 03:15 PM, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas hlinn...@iki.fi wrote: * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync with the list of individual locks in lwlock.h. Sooner or later someone will add an LWLock and forget

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 04:35 PM, Simon Riggs wrote: This and the OP seem like 9.5 open items to me. Why? This is nothing new in 9.5. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 14:55, Heikki Linnakangas hlinn...@iki.fi wrote: On 08/04/2015 04:35 PM, Simon Riggs wrote: This and the OP seem like 9.5 open items to me. Why? This is nothing new in 9.5. gin_pending_list_limit is new in 9.5 We're in Alpha, so if something has been added and isn't

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Andres Freund
On 2015-08-04 14:59:11 +0100, Simon Riggs wrote: On 4 August 2015 at 14:55, Heikki Linnakangas hlinn...@iki.fi wrote: On 08/04/2015 04:35 PM, Simon Riggs wrote: This and the OP seem like 9.5 open items to me. Why? This is nothing new in 9.5. gin_pending_list_limit is new in 9.5

Re: [HACKERS] Parallel Seq Scan

2015-08-04 Thread Amit Kapila
On Sun, Aug 2, 2015 at 8:06 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Amit, Let me ask three more detailed questions. Why Funnel has a valid qual of the subplan? The 2nd argument of make_funnel() is qualifier of the subplan (PartialSeqScan) then it is initialized at ExecInitFunnel,

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Beena Emerson
Michael Paquier wrote: And this is the case of any format as well. String format validation for a GUC occurs when server is reloaded or restarted, one advantage of JSON is that the parser validator is already here, so we don't need to reinvent a new machinery for that. IIUC correctly, we

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 05:56, Michael Paquier michael.paqu...@gmail.com wrote: Hi all, As mentioned in the thread related to lowering locks of autovacuum reloptions in ALTER TABLE SET ( http://www.postgresql.org/message-id/cafcns+ox7jvenc_3i54fdq3ibmogmknc2tmevdsmvojbsxg...@mail.gmail.com ),

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 3:27 PM, Beena Emerson memissemer...@gmail.com wrote: Michael Paquier wrote: And this is the case of any format as well. String format validation for a GUC occurs when server is reloaded or restarted, one advantage of JSON is that the parser validator is already here, so

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 2:43 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Michael Paquier wrote: As mentioned in the thread related to lowering locks of autovacuum reloptions in ALTER TABLE SET

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 08:03 AM, Jeff Janes wrote: For a GIN index with fastupdate turned on, both the user backends and autoanalyze routine will clear out the pending list, pushing the entries into the normal index structure and deleting the pages used by the pending list. But those deleted pages will

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 2:57 PM, Masahiko Sawada wrote: On Thu, Jul 30, 2015 at 2:16 PM, Beena Emerson wrote: Since there will not be many nesting and grouping, I still prefer new language to JSON. I understand one can easily, modify/add groups in JSON using in built functions but I think

Re: [HACKERS] WIP: SCRAM authentication

2015-08-04 Thread Michael Paquier
On Mon, Mar 30, 2015 at 7:52 PM, Heikki Linnakangas hlinn...@iki.fi wrote: There have been numerous threads on replacing our MD5 authentication method, so I started hacking on that to see what it might look like. Just to be clear, this is 9.6 material. Attached is a WIP patch series that adds

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 3:35 PM, Simon Riggs si...@2ndquadrant.com wrote: Please provide the link to the discussion of this. I don't see a problem here right now that can't be solved by saying Thread:

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Andres Freund
On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: On Sat, Aug 1, 2015 at 9:20 PM, Andres Freund wrote: On August 1, 2015 2:17:24 PM GMT+02:00, Michael Paquier wrote: For instance, if you told me to choose between ShareLock and ShareUpdateExclusiveLock I wouldn't know which one is

[HACKERS] GROUP BY before JOIN

2015-08-04 Thread David Rowley
== Overview == As of today we always perform GROUP BY at the final stage, after each relation in the query has been joined. This of course works, but it's not always the most optimal way of executing the query. Consider the following two relations: create table product (product_id int primary

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Amit Langote
On 2015-08-04 PM 05:58, Geoff Winkless wrote: ​Although it seems Amit has defined the problem better than I could, so this is a bit late to the party (!), yes, the table had been ALTERed after it was created (looking back through the history, that modification included at least one DROP

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Beena Emerson
Robert Haas wrote: Maybe shoehorning this into the GUC mechanism is the wrong thing, and what we really need is a new config file for this. The information we're proposing to store seems complex enough to justify that. I think the consensus is that JSON is better. And using a new file with

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 1:24 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Yeah, something like that. To paraphrase, if I'm now understanding it correctly, Peter's idea is: When all the tuples have been fed to tuplesort, and it's time to perform the sort, quicksort all the tuples currently in

Re: [HACKERS] Autonomous Transaction is back

2015-08-04 Thread Rajeev rastogi
On 03 August 2015 18:40, Merlin Moncure [mailto:mmonc...@gmail.com] Wrote: On Sun, Aug 2, 2015 at 11:37 PM, Rajeev rastogi rajeev.rast...@huawei.com wrote: On 31 July 2015 23:10, Robert Haas Wrote: I think we're going entirely down the wrong path here. Why is it ever useful for a backend's lock

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread Ashutosh Bapat
This looks like one example of general problem of finding optimal order for SQL operations. Consider a query defined as sql_op1(sql_op2(sql_op3(A, B), sql_op4(C, D), sql_op5(E, F where sql_op can be SQL operation like join, grouping, aggregation, projection, union, intersection, limit,

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 09:39, Simon Riggs si...@2ndquadrant.com wrote: On 4 August 2015 at 06:03, Jeff Janes jeff.ja...@gmail.com wrote: The attached proof of concept patch greatly improves the bloat for both the insert and the update cases. You need to turn on both features: adding the pages

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Tom Lane
Takashi Horikawa t-horik...@aj.jp.nec.com writes: Why does this cause a core dump? We could consider fixing whatever the problem is rather than capping the value. As far as I experiment with my own evaluation environment using PostgreSQL-9.4.4 on a x86_64 Linux, this problem can be fixed

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Andres Freund
On 2015-08-04 09:49:58 -0400, Tom Lane wrote: Takashi Horikawa t-horik...@aj.jp.nec.com writes: Why does this cause a core dump? We could consider fixing whatever the problem is rather than capping the value. As far as I experiment with my own evaluation environment using

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Fabrízio de Royes Mello
On Tue, Aug 4, 2015 at 5:55 AM, Andres Freund and...@anarazel.de wrote: On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: On Sat, Aug 1, 2015 at 9:20 PM, Andres Freund wrote: On August 1, 2015 2:17:24 PM GMT+02:00, Michael Paquier wrote: For instance, if you told me to choose between

Re: [HACKERS] Sharing aggregate states between different aggregate functions

2015-08-04 Thread Heikki Linnakangas
On 08/03/2015 08:53 AM, David Rowley wrote: Attached is a delta patched which is based on sharing_aggstate-heikki-2.patch to fix up the out-dated comments and also a few more test scenarios which test the sharing works with matching INITCOND and that it does not when they don't match. What do

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run external sort

2015-08-04 Thread Heikki Linnakangas
On 08/03/2015 11:36 PM, Robert Haas wrote: On Mon, Aug 3, 2015 at 3:33 PM, Peter Geoghegan p...@heroku.com wrote: When it's time to drain the heap, in performsort, divide the array into two arrays, based on the run number of each tuple, and then quicksort the arrays separately. The first array

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Geoff Winkless
On 4 August 2015 at 09:30, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-04 AM 02:57, Peter Geoghegan wrote: On Mon, Aug 3, 2015 at 8:53 AM, Geoff Winkless pgsqlad...@geoff.dj wrote: If I create a copy of the table using CREATE mytab (LIKE brokentab INCLUDING ALL);

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Amit Langote
On 2015-08-04 AM 02:57, Peter Geoghegan wrote: On Mon, Aug 3, 2015 at 8:53 AM, Geoff Winkless pgsqlad...@geoff.dj wrote: If I create a copy of the table using CREATE mytab (LIKE brokentab INCLUDING ALL); INSERT INTO mytab SELECT * FROM brokentab; Also, did you drop any columns from the

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread David Rowley
On 4 August 2015 at 21:56, Ashutosh Bapat ashutosh.ba...@enterprisedb.com wrote: This looks like one example of general problem of finding optimal order for SQL operations. Consider a query defined as sql_op1(sql_op2(sql_op3(A, B), sql_op4(C, D), sql_op5(E, F where sql_op can be SQL

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas hlinn...@iki.fi wrote: * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync with the list of individual locks in lwlock.h. Sooner or later someone will add an LWLock and forget to update the names-array. That needs to be

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 11:29:39 -0400, Robert Haas wrote: On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila amit.kapil...@gmail.com wrote: 1. I got rid of all of the typecasts. You're supposed to treat pg_atomic_u32 as a magic data type that is only manipulated via the primitives provided, not just cast

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Robert Haas
On Fri, Jul 31, 2015 at 3:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the real solution to this problem is to avoid use of GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far as I can see, that should completely close the hole. This requires patching

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 11:43:45 -0400, Robert Haas wrote: On Tue, Aug 4, 2015 at 11:33 AM, Andres Freund and...@anarazel.de wrote: Actually by far not all system calls are full barriers? How do we know which ones are and which ones are not? Good question. Reading the source code of all

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Robert Haas
On Sun, Aug 2, 2015 at 8:20 PM, Stephen Frost sfr...@snowman.net wrote: +1. I was doing testing the other day and ran into the pg_dump doesn't support shell types issue and it was annoyingly confusing. Is anyone working on this? Should it be added to the open items list? -- Robert Haas

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 15:18, Andres Freund and...@anarazel.de wrote: On 2015-08-04 14:59:11 +0100, Simon Riggs wrote: On 4 August 2015 at 14:55, Heikki Linnakangas hlinn...@iki.fi wrote: On 08/04/2015 04:35 PM, Simon Riggs wrote: This and the OP seem like 9.5 open items to me.

Re: [HACKERS] Autonomous Transaction is back

2015-08-04 Thread Merlin Moncure
On Tue, Aug 4, 2015 at 4:12 AM, Rajeev rastogi rajeev.rast...@huawei.com wrote: On 03 August 2015 18:40, Merlin Moncure [mailto:mmonc...@gmail.com] Wrote: On Sun, Aug 2, 2015 at 11:37 PM, Rajeev rastogi rajeev.rast...@huawei.com wrote: On 31 July 2015 23:10, Robert Haas Wrote: I think we're

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila amit.kapil...@gmail.com wrote: I agree and modified the patch to use 32-bit atomics based on idea suggested by Robert and didn't modify lwlock.c. While looking at patch, I found that the way it was initialising the list to be empty was wrong, it was

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 4 August 2015 at 15:18, Andres Freund and...@anarazel.de wrote: The only thing that variable does is change what the pending size limit is determined by. Previously it was work_mem, now it's gin_pending_list_limit. Imo that has pretty much nothing

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:33 AM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 11:29:39 -0400, Robert Haas wrote: On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila amit.kapil...@gmail.com wrote: 1. I got rid of all of the typecasts. You're supposed to treat pg_atomic_u32 as a magic data

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Amit Kapila
On Tue, Aug 4, 2015 at 8:59 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila amit.kapil...@gmail.com wrote: I agree and modified the patch to use 32-bit atomics based on idea suggested by Robert and didn't modify lwlock.c. While looking at patch, I

Re: [HACKERS] Incorrect comment about abbreviated keys

2015-08-04 Thread Robert Haas
On Sun, Aug 2, 2015 at 5:11 AM, Peter Geoghegan p...@heroku.com wrote: Attached patch fixes this issue. This was missed by 78efd5c1edb59017f06ef96773e64e6539bfbc86 Committed and back-patched to 9.5. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 21:20:20 +0530, Amit Kapila wrote: I have kept barriers based on comments on top of atomic read, refer below code: * No barrier semantics. */ STATIC_IF_INLINE uint32 pg_atomic_read_u32(volatile pg_atomic_uint32 *ptr) Note - The function header comments on

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 3:05 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Aug 4, 2015 at 2:41 AM, Michael Paquier michael.paqu...@gmail.com wrote: Yep, true as things stand now. But this would get broken if we add a new lock level between ShareRowExclusiveLock and AccessExclusiveLock

Re: [HACKERS] WIP: Make timestamptz_out less slow.

2015-08-04 Thread David Rowley
On 29 July 2015 at 03:25, Andres Freund and...@anarazel.de wrote: On 2015-07-29 03:10:41 +1200, David Rowley wrote: timestamp_out() = 2015-07-29 02:24:33.34 in 3.506000 timestamp_out_old() = 2015-07-29 02:24:33.034 in 64.518000 timestamp_out_af() = 2015-07-29 02:24:33.034 in 2.981000

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-08-04 Thread Amit Langote
On 2015-08-05 AM 06:11, Robert Haas wrote: On Mon, Aug 3, 2015 at 8:19 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-03 PM 09:24, Ashutosh Bapat wrote: For postgres_fdw it's a boolean server-level option 'twophase_compliant' (suggestions for name welcome). How about just

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 2:15 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andres Freund wrote: On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: As long as this only applies on master, this may be fine... We could basically pass a LOCKMASK to the multiple layers of tablecmds.c

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 11:47 AM, Noah Misch n...@leadboat.com wrote: On Tue, Aug 04, 2015 at 07:35:43AM +0100, Simon Riggs wrote: On 4 August 2015 at 05:56, Michael Paquier michael.paqu...@gmail.com wrote: The thing is that, as mentioned by Alvaro and Andres on this thread, we have no

Re: [HACKERS] Sharing aggregate states between different aggregate functions

2015-08-04 Thread David Rowley
On 5 August 2015 at 03:03, Heikki Linnakangas hlinn...@iki.fi wrote: On 08/03/2015 08:53 AM, David Rowley wrote: Attached is a delta patched which is based on sharing_aggstate-heikki-2.patch to fix up the out-dated comments and also a few more test scenarios which test the sharing works with

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 8:19 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-03 PM 09:24, Ashutosh Bapat wrote: On Sat, Aug 1, 2015 at 12:18 AM, Robert Haas robertmh...@gmail.com wrote: OK, sure. But let's make sure postgres_fdw gets a server-level option to control this.

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Alvaro Herrera
Alvaro Herrera wrote: Thankfully I found another way to solve it, which is to forgo usage of MVCC here and instead use SnapshotAny. There's already a mode in IndexBuildHeapRangeScan that uses SnapshotAny, but it needs some tweaks to do what we need. I'm going to propose a patch along those

Re: [HACKERS] [DOCS] max_worker_processes on the standby

2015-08-04 Thread Alvaro Herrera
Adding CC to hackers, since this is clearly not just a docs issue. Also CCing Petr and Craig since they are the ones that know how this is used in BDR. Robert Haas wrote: On Tue, Aug 4, 2015 at 12:41 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: The alternative is to turn the feature

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-08-04 Thread Tom Lane
Piotr Stefaniak postg...@piotr-stefaniak.me writes: On 08/03/2015 09:18 PM, Tom Lane wrote: ... but I can't reproduce it on HEAD with either of these queries. Not clear why you're getting different results. I'm terribly sorry, but I didn't notice that postgresql.conf was modified... Set

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-08-04 Thread Ewan Higgs
Hi there,I've been following the sqlsmith work and wanted to jump in and try it out. I took Peter's idea and tried building postgres with the flags suggested but it was hard to get anything working. I'm on commit 85e5e222b1dd02f135a8c3bf387d0d6d88e669bd (Tue Aug 4 14:55:32 2015 -0400)

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 2:29 AM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: Perhaps, it may have to do with how EXCLUDED pseudo-rel's targetlist is manipulated through parse-plan stage? I think so, yes. I'll look into writing a fix for this later in the week. Thanks for the report,

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 2:23 AM, Alvaro Herrera wrote: Michael Paquier wrote: On Tue, Aug 4, 2015 at 2:43 PM, Alvaro Herrera wrote: Now, let's take for example this case with locks A, B, C, D: - Lock A conflicts with ACD - B with BCD - C with itself - D with itself What would you choose as

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:50 AM, Amit Kapila amit.kapil...@gmail.com wrote: I have kept barriers based on comments on top of atomic read, refer below code: * No barrier semantics. */ STATIC_IF_INLINE uint32 pg_atomic_read_u32(volatile pg_atomic_uint32 *ptr) Note - The function header

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Aug 2, 2015 at 8:20 PM, Stephen Frost sfr...@snowman.net wrote: +1. I was doing testing the other day and ran into the pg_dump doesn't support shell types issue and it was annoyingly confusing. Is anyone working on this? Should it be added

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 13:52:54 -0400, Tom Lane wrote: Not sure whether we should consider it a back-patchable bug fix or something to do only in HEAD, though --- comments? Tentatively

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Alvaro Herrera
Andres Freund wrote: On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: As long as this only applies on master, this may be fine... We could basically pass a LOCKMASK to the multiple layers of tablecmds.c instead of LOCKMODE to track all the locks that need to be taken, and all the

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Alvaro Herrera
Michael Paquier wrote: On Tue, Aug 4, 2015 at 2:43 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Maybe the solution to this is to add the concept of addition of two lock modes, where the result is another lock mode that conflicts with any lock that would conflict with either of the

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:29 AM, Robert Haas robertmh...@gmail.com wrote: 4. I factored out the actual XID-clearing logic into a new function ProcArrayEndTransactionInternal instead of repeating it twice. On the flip side, I merged PushProcAndWaitForXidClear with PopProcsAndClearXids and

Re: [HACKERS] Tab completion for CREATE SEQUENCE

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 2:17 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Tue, Jul 7, 2015 at 9:14 PM, Andres Freund and...@anarazel.de wrote: On 2015-06-19 06:41:19 +, Brendan Jurd wrote: I'm marking this Waiting on Author. Once the problems have been corrected, it should be

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Andres Freund
On 2015-08-04 13:16:52 -0400, Robert Haas wrote: On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao masao.fu...@gmail.com wrote: track_commit_timestamp tracks COMMIT PREPARED as expected in standby server, but not in master server. Is this intentional? It should track COMMIT PREPARED even in

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Andres Freund
On 2015-08-04 13:52:54 -0400, Tom Lane wrote: Not sure whether we should consider it a back-patchable bug fix or something to do only in HEAD, though --- comments? Tentatively I'd say it's a bug and should be back-patched. Andres -- Sent via pgsql-hackers mailing list

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Alvaro Herrera
Jeff Janes wrote: The attached proof of concept patch greatly improves the bloat for both the insert and the update cases. You need to turn on both features: adding the pages to fsm, and vacuuming the fsm, to get the benefit (so JJ_GIN=3). The first of those two things could probably be

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 1:18 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 13:16:52 -0400, Robert Haas wrote: On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao masao.fu...@gmail.com wrote: track_commit_timestamp tracks COMMIT PREPARED as expected in standby server, but not in master

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 9:52 AM, Andres Freund and...@anarazel.de wrote: So my vote is for the original approach. (I've not read Josh's patch, so there might be something wrong with it in detail, but I like the basic approach.) +1 OK, committed and back-patched that all the way back to 9.0.

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 2:41 AM, Michael Paquier michael.paqu...@gmail.com wrote: Yep, true as things stand now. But this would get broken if we add a new lock level between ShareRowExclusiveLock and AccessExclusiveLock that does not respect the current monotone hierarchy between those. But

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao masao.fu...@gmail.com wrote: track_commit_timestamp tracks COMMIT PREPARED as expected in standby server, but not in master server. Is this intentional? It should track COMMIT PREPARED even in master? Otherwise, we cannot use commit_timestamp feature

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 13:52:54 -0400, Tom Lane wrote: Not sure whether we should consider it a back-patchable bug fix or something to do only in HEAD, though --- comments? Tentatively I'd say it's a bug and should be

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Alvaro Herrera
Robert Haas wrote: On Fri, Jul 31, 2015 at 3:45 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think the only way to close this hole is to have summarize_range() sleep until all open snapshots are gone after inserting the placeholder tuple and before acquiring the snapshot,

Re: [HACKERS] [sqlsmith] subplan variable reference / unassigned NestLoopParams (was: [sqlsmith] Failed assertion in joinrels.c)

2015-08-04 Thread Tom Lane
Andreas Seltenreich seltenre...@gmx.de writes: Tom Lane writes: Well, I certainly think all of these represent bugs: 3 | ERROR: plan should not reference subplan's variable 2 | ERROR: failed to assign all NestLoopParams to plan nodes These appear to be related. The following query

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Jeff Janes
On Tue, Aug 4, 2015 at 1:39 AM, Simon Riggs si...@2ndquadrant.com wrote: On 4 August 2015 at 06:03, Jeff Janes jeff.ja...@gmail.com wrote: The attached proof of concept patch greatly improves the bloat for both the insert and the update cases. You need to turn on both features: adding the

Re: [HACKERS] more-helpful-izing a debug message

2015-08-04 Thread Robert Haas
On Wed, Jul 8, 2015 at 5:38 AM, Marko Tiikkaja ma...@joh.to wrote: One of the debug messages related to logical replication could be more helpful than it currently is. The attached patch reorders the two operations to make it so. Please consider patching and back-patching. Andres, this

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 4:37 PM, Ildus Kurbangaliev i.kurbangal...@postgrespro.ru wrote: A new version of the patch. I used your idea with macros, and with tranches that allowed us to remove array with names (they can be written directly to the corresponding tranche). You seem not to have

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 3:55 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 15:45:44 -0400, Tom Lane wrote: I'm not sure that there's any great urgency about changing the instances that exist now; the real point of this discussion is that we will allow new code to use static inlines

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Robert Haas
On Fri, Jul 3, 2015 at 8:33 PM, Peter Geoghegan p...@heroku.com wrote: Since apparently we're back to development work, I thought it was time to share a patch implementing a few additional simple tricks to make sorting text under a non-C locale even faster than in 9.5. These techniques are

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Jeff Janes
On Tue, Aug 4, 2015 at 6:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On 4 August 2015 at 09:39, Simon Riggs si...@2ndquadrant.com wrote: On 4 August 2015 at 06:03, Jeff Janes jeff.ja...@gmail.com wrote: The attached proof of concept patch greatly improves the bloat for both the insert

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 4:47 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Aug 4, 2015 at 4:37 PM, Ildus Kurbangaliev i.kurbangal...@postgrespro.ru wrote: A new version of the patch. I used your idea with macros, and with tranches that allowed us to remove array with names (they can be

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Tom Lane
Andres Freund and...@anarazel.de writes: On 2015-08-04 15:20:14 -0400, Robert Haas wrote: OK, so do we want to rip out all instances of the static inline dance in favor of more straightforward coding? Do we then shut pandemelon and any other affected buildfarm members down as unsupported, or

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Alvaro Herrera
Ildus Kurbangaliev wrote: A new version of the patch. I used your idea with macros, and with tranches that allowed us to remove array with names (they can be written directly to the corresponding tranche). Just a bystander here, I haven't reviewed this patch at all, but I have two

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Andres Freund
On 2015-08-04 15:20:14 -0400, Robert Haas wrote: OK, so do we want to rip out all instances of the static inline dance in favor of more straightforward coding? Do we then shut pandemelon and any other affected buildfarm members down as unsupported, or what? I think all that happens is that

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 12:41 PM, Robert Haas robertmh...@gmail.com wrote: Interesting work. Thanks. 1. My biggest gripe with this patch is that the comments are not easy to understand. Of course everybody may prefer something different here; I'm just telling you what I think. I have

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 1:30 PM, Peter Geoghegan p...@heroku.com wrote: 2. I believe the change to bttextcmp_abbrev() should be pulled out into a separate patch and committed separately. That part seems like a slam dunk. Makes sense. BTW, I want to put the string_uint() macro in a common

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Andres Freund
On 2015-08-04 15:45:44 -0400, Tom Lane wrote: I'm not sure that there's any great urgency about changing the instances that exist now; the real point of this discussion is that we will allow new code to use static inlines in headers. I agree that we don't have to (and probably shouldn't) make

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Robert Haas
On Thu, Jul 2, 2015 at 12:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: So far this thread is all about the costs of desupporting compilers that don't have these features, and you're making a good argument (that I think we all agree with) that the cost is

Re: [HACKERS] Dependency between bgw_notify_pid and bgw_flags

2015-08-04 Thread Robert Haas
On Tue, Jul 7, 2015 at 4:34 AM, Ashutosh Bapat ashutosh.ba...@enterprisedb.com wrote: With that notion of backend, to fix the original problem I reported, PostmasterMarkPIDForWorkerNotify() should also look at the BackgroundWorkerList. As per the comments in the prologue of this function, it

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
On Aug 4, 2015, at 4:54 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 08/04/2015 03:15 PM, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas hlinn...@iki.fi wrote: * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync with the list of

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Andrew Dunstan
On 08/04/2015 02:09 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-04 13:52:54 -0400, Tom Lane wrote: Not sure whether we should consider it a back-patchable bug fix or something to do only in

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 21:04, Jeff Janes jeff.ja...@gmail.com wrote: Couple of questions here... * the docs say it's desirable to have pending-list cleanup occur in the background, but there is no way to invoke that, except via VACUUM. I think we need a separate function to be able to call

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Noah Misch
On Tue, Aug 04, 2015 at 07:35:43AM +0100, Simon Riggs wrote: On 4 August 2015 at 05:56, Michael Paquier michael.paqu...@gmail.com wrote: The thing is that, as mentioned by Alvaro and Andres on this thread, we have no guarantee that the different relation locks compared have a monotone

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Takashi Horikawa
... Josh's approach of restricting the buffer size seems a lot more robust. I understand that the capping of approach of restricting the buffer size is much more robust and is suitable in this case. I, howerver, think that the chane from 'page = XLogCtl-pages[firstIdx * XLOG_BLCKSZ];' to 'page

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 8:37 PM, Beena Emerson memissemer...@gmail.com wrote: Robert Haas wrote: Maybe shoehorning this into the GUC mechanism is the wrong thing, and what we really need is a new config file for this. The information we're proposing to store seems complex enough to justify that.

  1   2   >