Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Amit Kapila
On Tue, Aug 4, 2015 at 5:45 PM, Robert Haas wrote: > > On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas wrote: > > * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync > > with the list of individual locks in lwlock.h. Sooner or later someone will > > add an LWLock and f

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 11:47 AM, Noah Misch wrote: > On Tue, Aug 04, 2015 at 07:35:43AM +0100, Simon Riggs wrote: >> On 4 August 2015 at 05:56, Michael Paquier wrote: >> > The thing is that, as mentioned by Alvaro and Andres on this thread, >> > we have no guarantee that the different relation lo

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Noah Misch
On Tue, Aug 04, 2015 at 07:35:43AM +0100, Simon Riggs wrote: > On 4 August 2015 at 05:56, Michael Paquier wrote: > > The thing is that, as mentioned by Alvaro and Andres on this thread, > > we have no guarantee that the different relation locks compared have a > > monotone hierarchy and we may fin

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Michael Paquier
On Tue, Aug 4, 2015 at 8:37 PM, Beena Emerson wrote: > Robert Haas wrote: >>Maybe shoehorning this into the GUC mechanism is the wrong thing, and >>what we really need is a new config file for this. The information >>we're proposing to store seems complex enough to justify that. >> > > I think th

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 2:15 AM, Alvaro Herrera wrote: > Andres Freund wrote: >> On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: > >> > As long as this only applies on master, this may be fine... We could >> > basically pass a LOCKMASK to the multiple layers of tablecmds.c >> > instead of LOCK

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-08-04 Thread Amit Langote
On 2015-08-05 AM 06:11, Robert Haas wrote: > On Mon, Aug 3, 2015 at 8:19 PM, Amit Langote > wrote: >> On 2015-08-03 PM 09:24, Ashutosh Bapat wrote: >>> For postgres_fdw it's a boolean server-level option 'twophase_compliant' >>> (suggestions for name welcome). >>> >> >> How about just 'twophase'?

Re: [HACKERS] WIP: Make timestamptz_out less slow.

2015-08-04 Thread David Rowley
On 29 July 2015 at 03:25, Andres Freund wrote: > On 2015-07-29 03:10:41 +1200, David Rowley wrote: > > timestamp_out() = 2015-07-29 02:24:33.34 in 3.506000 > > timestamp_out_old() = 2015-07-29 02:24:33.034 in 64.518000 > > timestamp_out_af() = 2015-07-29 02:24:33.034 in 2.981000 > > > > timestamp

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 3:05 AM, Robert Haas wrote: > On Tue, Aug 4, 2015 at 2:41 AM, Michael Paquier > wrote: >> Yep, true as things stand now. But this would get broken if we add a >> new lock level between ShareRowExclusiveLock and AccessExclusiveLock >> that does not respect the current monoto

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Josh Berkus
On 06/25/2015 07:50 AM, Tom Lane wrote: > To do that, we'd have to change the semantics of the 'waiting' column so > that it becomes true for non-heavyweight-lock waits. I'm not sure whether > that's a good idea or not; I'm afraid there may be client-side code that > expects 'waiting' to indicate

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Takashi Horikawa
> ... Josh's approach of restricting the buffer size seems > a lot more robust. I understand that the capping of approach of restricting the buffer size is much more robust and is suitable in this case. I, howerver, think that the chane from 'page = &XLogCtl->pages[firstIdx * XLOG_BLCKSZ];' to 'pa

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-08-04 Thread Tom Lane
Piotr Stefaniak writes: > On 08/03/2015 09:18 PM, Tom Lane wrote: >> ... but I can't reproduce it on HEAD with either of these queries. >> Not clear why you're getting different results. > I'm terribly sorry, but I didn't notice that postgresql.conf was modified... > Set join_collapse_limit = 32

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Michael Paquier
On Wed, Aug 5, 2015 at 2:23 AM, Alvaro Herrera wrote: > Michael Paquier wrote: >> On Tue, Aug 4, 2015 at 2:43 PM, Alvaro Herrera wrote: >> Now, let's take for example this case with locks A, B, C, D: >> - Lock A conflicts with ACD >> - B with BCD >> - C with itself >> - D with itself >> What would

Re: [HACKERS] cost_agg() with AGG_HASHED does not account for startup costs

2015-08-04 Thread David Rowley
On 5 August 2015 at 01:54, Tom Lane wrote: > David Rowley writes: > > During working on allowing the planner to perform GROUP BY before joining > > I've noticed that cost_agg() completely ignores input_startup_cost > > when aggstrategy == AGG_HASHED. > > Isn't your proposed patch double-counting

Re: [HACKERS] [DOCS] max_worker_processes on the standby

2015-08-04 Thread Alvaro Herrera
Adding CC to hackers, since this is clearly not just a docs issue. Also CCing Petr and Craig since they are the ones that know how this is used in BDR. Robert Haas wrote: > On Tue, Aug 4, 2015 at 12:41 AM, Alvaro Herrera > wrote: > > The alternative is to turn the feature on automatically if it

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Alvaro Herrera
Alvaro Herrera wrote: > Thankfully I found > another way to solve it, which is to forgo usage of MVCC here and > instead use SnapshotAny. There's already a mode in > IndexBuildHeapRangeScan that uses SnapshotAny, but it needs some tweaks > to do what we need. I'm going to propose a patch along t

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-08-04 Thread Ewan Higgs
Hi there,I've been following the sqlsmith work and wanted to jump in and try it out. I took Peter's idea and tried building postgres with the flags suggested but it was hard to get anything working. I'm on commit 85e5e222b1dd02f135a8c3bf387d0d6d88e669bd (Tue Aug 4 14:55:32 2015 -0400) Configur

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 2:29 AM, Amit Langote wrote: > Perhaps, it may have to do with how EXCLUDED pseudo-rel's targetlist is > manipulated through parse-plan stage? I think so, yes. I'll look into writing a fix for this later in the week. Thanks for the report, Geoff, and thanks for the analys

Re: [HACKERS] Sharing aggregate states between different aggregate functions

2015-08-04 Thread David Rowley
On 5 August 2015 at 03:03, Heikki Linnakangas wrote: > On 08/03/2015 08:53 AM, David Rowley wrote: > >> Attached is a delta patched which is based >> on sharing_aggstate-heikki-2.patch to fix up the out-dated comments and >> also a few more test scenarios which test the sharing works with matchin

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 8:19 PM, Amit Langote wrote: > On 2015-08-03 PM 09:24, Ashutosh Bapat wrote: >> On Sat, Aug 1, 2015 at 12:18 AM, Robert Haas wrote: >>> >>> OK, sure. But let's make sure postgres_fdw gets a server-level option >>> to control this. >>> >>> >> For postgres_fdw it's a boolean

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 3:55 PM, Andres Freund wrote: > On 2015-08-04 15:45:44 -0400, Tom Lane wrote: >> I'm not sure that there's any great urgency about changing the instances >> that exist now; the real point of this discussion is that we will allow >> new code to use static inlines in headers.

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Alvaro Herrera
Ildus Kurbangaliev wrote: > A new version of the patch. I used your idea with macros, and with tranches > that > allowed us to remove array with names (they can be written directly to the > corresponding > tranche). Just a bystander here, I haven't reviewed this patch at all, but I have two que

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 4:47 PM, Robert Haas wrote: > On Tue, Aug 4, 2015 at 4:37 PM, Ildus Kurbangaliev > wrote: >> A new version of the patch. I used your idea with macros, and with tranches >> that >> allowed us to remove array with names (they can be written directly to the >> corresponding

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 21:04, Jeff Janes wrote: > Couple of questions here... >> >> * the docs say "it's desirable to have pending-list cleanup occur in the >> background", but there is no way to invoke that, except via VACUUM. I >> think we need a separate function to be able to call this as a bac

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 4:37 PM, Ildus Kurbangaliev wrote: > A new version of the patch. I used your idea with macros, and with tranches > that > allowed us to remove array with names (they can be written directly to the > corresponding > tranche). You seem not to have addressed a few of the poi

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 1:30 PM, Peter Geoghegan wrote: >> 2. I believe the change to bttextcmp_abbrev() should be pulled out >> into a separate patch and committed separately. That part seems like >> a slam dunk. > > Makes sense. BTW, I want to put the string_uint() macro in a common header now

Re: [HACKERS] more-helpful-izing a debug message

2015-08-04 Thread Robert Haas
On Wed, Jul 8, 2015 at 5:38 AM, Marko Tiikkaja wrote: > One of the debug messages related to logical replication could be more > helpful than it currently is. The attached patch reorders the two > operations to make it so. > > Please consider patching and back-patching. Andres, this looks like a

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
> On Aug 4, 2015, at 4:54 PM, Heikki Linnakangas wrote: > > On 08/04/2015 03:15 PM, Robert Haas wrote: >> On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas wrote: >>> * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync >>> with the list of individual locks in lwlock.h.

Re: [HACKERS] Dependency between bgw_notify_pid and bgw_flags

2015-08-04 Thread Robert Haas
On Tue, Jul 7, 2015 at 4:34 AM, Ashutosh Bapat wrote: > With that notion of backend, to fix the original problem I reported, > PostmasterMarkPIDForWorkerNotify() should also look at the > BackgroundWorkerList. As per the comments in the prologue of this function, > it assumes that only backends ne

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 12:41 PM, Robert Haas wrote: > Interesting work. Thanks. > 1. My biggest gripe with this patch is that the comments are not easy > to understand. > Of course everybody may prefer something different here; I'm just > telling you what I think. I have struggled with trying

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Jeff Janes
On Tue, Aug 4, 2015 at 6:35 AM, Simon Riggs wrote: > On 4 August 2015 at 09:39, Simon Riggs wrote: > >> On 4 August 2015 at 06:03, Jeff Janes wrote: >> >> >>> The attached proof of concept patch greatly improves the bloat for both >>> the insert and the update cases. You need to turn on both f

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Andres Freund
On 2015-08-04 15:45:44 -0400, Tom Lane wrote: > I'm not sure that there's any great urgency about changing the instances > that exist now; the real point of this discussion is that we will allow > new code to use static inlines in headers. I agree that we don't have to (and probably shouldn't) mak

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Tom Lane
Andres Freund writes: > On 2015-08-04 15:20:14 -0400, Robert Haas wrote: >> OK, so do we want to rip out all instances of the static inline dance >> in favor of more straightforward coding? Do we then shut pandemelon >> and any other affected buildfarm members down as unsupported, or what? > I t

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Jeff Janes
On Tue, Aug 4, 2015 at 1:39 AM, Simon Riggs wrote: > On 4 August 2015 at 06:03, Jeff Janes wrote: > > >> The attached proof of concept patch greatly improves the bloat for both >> the insert and the update cases. You need to turn on both features: adding >> the pages to fsm, and vacuuming the f

Re: [HACKERS] More work on SortSupport for text - strcoll() and strxfrm() caching

2015-08-04 Thread Robert Haas
On Fri, Jul 3, 2015 at 8:33 PM, Peter Geoghegan wrote: > Since apparently we're back to development work, I thought it was time > to share a patch implementing a few additional simple tricks to make > sorting text under a non-C locale even faster than in 9.5. These > techniques are mostly effectiv

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Andres Freund
On 2015-08-04 15:20:14 -0400, Robert Haas wrote: > OK, so do we want to rip out all instances of the static inline dance > in favor of more straightforward coding? Do we then shut pandemelon > and any other affected buildfarm members down as unsupported, or what? I think all that happens is that

Re: [HACKERS] Raising our compiler requirements for 9.6

2015-08-04 Thread Robert Haas
On Thu, Jul 2, 2015 at 12:10 PM, Tom Lane wrote: > Robert Haas writes: >> So far this thread is all about the costs of desupporting compilers >> that don't have these features, and you're making a good argument >> (that I think we all agree with) that the cost is small. But you >> haven't really

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Andrew Dunstan
On 08/04/2015 02:09 PM, Tom Lane wrote: Robert Haas writes: On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund wrote: On 2015-08-04 13:52:54 -0400, Tom Lane wrote: Not sure whether we should consider it a back-patchable bug fix or something to do only in HEAD, though --- comments? Tentatively I

Re: [HACKERS] [sqlsmith] subplan variable reference / unassigned NestLoopParams (was: [sqlsmith] Failed assertion in joinrels.c)

2015-08-04 Thread Tom Lane
Andreas Seltenreich writes: > Tom Lane writes: >> Well, I certainly think all of these represent bugs: >>> 3 | ERROR: plan should not reference subplan's variable >>> 2 | ERROR: failed to assign all NestLoopParams to plan nodes > These appear to be related. The following query produces the for

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Jul 31, 2015 at 3:45 PM, Alvaro Herrera > wrote: > > I think the only way to close this hole is to have summarize_range() > > sleep until all open snapshots are gone after inserting the placeholder > > tuple and before acquiring the snapshot, similarly to how CREATE I

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Tom Lane
Robert Haas writes: > On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund wrote: >> On 2015-08-04 13:52:54 -0400, Tom Lane wrote: >>> Not sure whether we should consider it a back-patchable bug fix or >>> something to do only in HEAD, though --- comments? >> Tentatively I'd say it's a bug and should b

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 2:41 AM, Michael Paquier wrote: > Yep, true as things stand now. But this would get broken if we add a > new lock level between ShareRowExclusiveLock and AccessExclusiveLock > that does not respect the current monotone hierarchy between those. But we're probably not going t

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 1:54 PM, Andres Freund wrote: > On 2015-08-04 13:52:54 -0400, Tom Lane wrote: >> Not sure whether we should consider it a back-patchable bug fix or >> something to do only in HEAD, though --- comments? > > Tentatively I'd say it's a bug and should be back-patched. Agreed.

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Andres Freund
On 2015-08-04 13:52:54 -0400, Tom Lane wrote: > Not sure whether we should consider it a back-patchable bug fix or > something to do only in HEAD, though --- comments? Tentatively I'd say it's a bug and should be back-patched. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Tom Lane
Robert Haas writes: > On Sun, Aug 2, 2015 at 8:20 PM, Stephen Frost wrote: >> +1. >> >> I was doing testing the other day and ran into the "pg_dump doesn't >> support shell types" issue and it was annoyingly confusing. > Is anyone working on this? Should it be added to the open items list? I

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:29 AM, Robert Haas wrote: > 4. I factored out the actual XID-clearing logic into a new function > ProcArrayEndTransactionInternal instead of repeating it twice. On the > flip side, I merged PushProcAndWaitForXidClear with > PopProcsAndClearXids and renamed the result to P

Re: [HACKERS] tablecmds.c and lock hierarchy

2015-08-04 Thread Alvaro Herrera
Michael Paquier wrote: > On Tue, Aug 4, 2015 at 2:43 PM, Alvaro Herrera > wrote: > > Maybe the solution to this is to add the concept of "addition" of two > > lock modes, where the result is another lock mode that conflicts with > > any lock that would conflict with either of the two operand loc

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 1:18 PM, Andres Freund wrote: > On 2015-08-04 13:16:52 -0400, Robert Haas wrote: >> On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao wrote: >> > track_commit_timestamp tracks COMMIT PREPARED as expected in standby >> > server, >> > but not in master server. Is this intentional

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Alvaro Herrera
Jeff Janes wrote: > The attached proof of concept patch greatly improves the bloat for both the > insert and the update cases. You need to turn on both features: adding the > pages to fsm, and vacuuming the fsm, to get the benefit (so JJ_GIN=3). The > first of those two things could probably be

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Andres Freund
On 2015-08-04 13:16:52 -0400, Robert Haas wrote: > On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao wrote: > > track_commit_timestamp tracks COMMIT PREPARED as expected in standby server, > > but not in master server. Is this intentional? It should track COMMIT > > PREPARED > > even in master? Otherw

Re: [HACKERS] track_commit_timestamp and COMMIT PREPARED

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 10:31 AM, Fujii Masao wrote: > track_commit_timestamp tracks COMMIT PREPARED as expected in standby server, > but not in master server. Is this intentional? It should track COMMIT PREPARED > even in master? Otherwise, we cannot use commit_timestamp feature to check > the rep

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Alvaro Herrera
Andres Freund wrote: > On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: > > As long as this only applies on master, this may be fine... We could > > basically pass a LOCKMASK to the multiple layers of tablecmds.c > > instead of LOCKMODE to track all the locks that need to be taken, and > > all

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 9:52 AM, Andres Freund wrote: >> So my vote is for the original approach. (I've not read Josh's patch, >> so there might be something wrong with it in detail, but I like the >> basic approach.) > > +1 OK, committed and back-patched that all the way back to 9.0. -- Robert

Re: [HACKERS] Tab completion for CREATE SEQUENCE

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 2:17 AM, Michael Paquier wrote: > On Tue, Jul 7, 2015 at 9:14 PM, Andres Freund wrote: >> On 2015-06-19 06:41:19 +, Brendan Jurd wrote: >>> I'm marking this "Waiting on Author". Once the problems have been >>> corrected, it should be ready for a committer. >> >> Vik, a

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:50 AM, Amit Kapila wrote: > I have kept barriers based on comments on top of atomic read, refer > below code: > > * No barrier semantics. > */ > STATIC_IF_INLINE uint32 > pg_atomic_read_u32(volatile pg_atomic_uint32 *ptr) > > Note - The function header comments on pg_at

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 21:20:20 +0530, Amit Kapila wrote: > I have kept barriers based on comments on top of atomic read, refer > below code: > * No barrier semantics. > */ > STATIC_IF_INLINE uint32 > pg_atomic_read_u32(volatile pg_atomic_uint32 *ptr) > > Note - The function header comments on pg_atomi

Re: [HACKERS] Incorrect comment about abbreviated keys

2015-08-04 Thread Robert Haas
On Sun, Aug 2, 2015 at 5:11 AM, Peter Geoghegan wrote: > Attached patch fixes this issue. This was missed by > 78efd5c1edb59017f06ef96773e64e6539bfbc86 Committed and back-patched to 9.5. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pg

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 11:43:45 -0400, Robert Haas wrote: > On Tue, Aug 4, 2015 at 11:33 AM, Andres Freund wrote: > > Actually by far not all system calls are full barriers? > > How do we know which ones are and which ones are not? Good question. Reading the source code of all implementations I suppose :

Re: [HACKERS] upgrade failure from 9.5 to head

2015-08-04 Thread Robert Haas
On Sun, Aug 2, 2015 at 8:20 PM, Stephen Frost wrote: > +1. > > I was doing testing the other day and ran into the "pg_dump doesn't > support shell types" issue and it was annoyingly confusing. Is anyone working on this? Should it be added to the open items list? -- Robert Haas EnterpriseDB: ht

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Amit Kapila
On Tue, Aug 4, 2015 at 8:59 PM, Robert Haas wrote: > > On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila wrote: > >> I agree and modified the patch to use 32-bit atomics based on idea > >> suggested by Robert and didn't modify lwlock.c. > > > > While looking at patch, I found that the way it was initia

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Tue, Aug 4, 2015 at 11:33 AM, Andres Freund wrote: > On 2015-08-04 11:29:39 -0400, Robert Haas wrote: >> On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila wrote: >> 1. I got rid of all of the typecasts. You're supposed to treat >> pg_atomic_u32 as a magic data type that is only manipulated via the

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-08-04 Thread Robert Haas
On Fri, Jul 31, 2015 at 3:45 PM, Alvaro Herrera wrote: >> I think the real solution to this problem is to avoid use of >> GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far >> as I can see, that should completely close the hole. This requires >> patching IndexBuildHeapRangeSca

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Andres Freund
On 2015-08-04 11:29:39 -0400, Robert Haas wrote: > On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila wrote: > 1. I got rid of all of the typecasts. You're supposed to treat > pg_atomic_u32 as a magic data type that is only manipulated via the > primitives provided, not just cast back and forth between

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Tom Lane
Simon Riggs writes: > On 4 August 2015 at 15:18, Andres Freund wrote: >> The only thing that variable does is change what the pending size limit >> is determined by. Previously it was work_mem, now it's >> gin_pending_list_limit. Imo that has pretty much nothing to do with not >> registering page

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-04 Thread Robert Haas
On Mon, Aug 3, 2015 at 8:39 AM, Amit Kapila wrote: >> I agree and modified the patch to use 32-bit atomics based on idea >> suggested by Robert and didn't modify lwlock.c. > > While looking at patch, I found that the way it was initialising the list > to be empty was wrong, it was using pgprocno a

Re: [HACKERS] Autonomous Transaction is back

2015-08-04 Thread Merlin Moncure
On Tue, Aug 4, 2015 at 4:12 AM, Rajeev rastogi wrote: > On 03 August 2015 18:40, Merlin Moncure [mailto:mmonc...@gmail.com] Wrote: >>On Sun, Aug 2, 2015 at 11:37 PM, Rajeev rastogi >> wrote: >>> On 31 July 2015 23:10, Robert Haas Wrote: I think we're going entirely down the wrong path here. W

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 15:18, Andres Freund wrote: > On 2015-08-04 14:59:11 +0100, Simon Riggs wrote: > > On 4 August 2015 at 14:55, Heikki Linnakangas wrote: > > > > > On 08/04/2015 04:35 PM, Simon Riggs wrote: > > > > > >> This and the OP seem like 9.5 open items to me. > > >> > > > > > > Why? Th

Re: [HACKERS] Sharing aggregate states between different aggregate functions

2015-08-04 Thread Heikki Linnakangas
On 08/03/2015 08:53 AM, David Rowley wrote: Attached is a delta patched which is based on sharing_aggstate-heikki-2.patch to fix up the out-dated comments and also a few more test scenarios which test the sharing works with matching INITCOND and that it does not when they don't match. What do yo

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Fabrízio de Royes Mello
On Tue, Aug 4, 2015 at 5:55 AM, Andres Freund wrote: > > On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: > > On Sat, Aug 1, 2015 at 9:20 PM, Andres Freund wrote: > > > On August 1, 2015 2:17:24 PM GMT+02:00, Michael Paquier wrote: > > >>> For instance, if you told me to choose between ShareLo

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Andres Freund
On 2015-08-04 14:59:11 +0100, Simon Riggs wrote: > On 4 August 2015 at 14:55, Heikki Linnakangas wrote: > > > On 08/04/2015 04:35 PM, Simon Riggs wrote: > > > >> This and the OP seem like 9.5 open items to me. > >> > > > > Why? This is nothing new in 9.5. > > > gin_pending_list_limit is new in

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 14:55, Heikki Linnakangas wrote: > On 08/04/2015 04:35 PM, Simon Riggs wrote: > >> This and the OP seem like 9.5 open items to me. >> > > Why? This is nothing new in 9.5. gin_pending_list_limit is new in 9.5 We're in Alpha, so if something has been added and isn't very usab

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 04:35 PM, Simon Riggs wrote: This and the OP seem like 9.5 open items to me. Why? This is nothing new in 9.5. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] cost_agg() with AGG_HASHED does not account for startup costs

2015-08-04 Thread Tom Lane
David Rowley writes: > During working on allowing the planner to perform GROUP BY before joining > I've noticed that cost_agg() completely ignores input_startup_cost > when aggstrategy == AGG_HASHED. Isn't your proposed patch double-counting the input startup cost? input_total_cost already includ

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 03:15 PM, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas wrote: * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync with the list of individual locks in lwlock.h. Sooner or later someone will add an LWLock and forget to update the na

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Andres Freund
On 2015-08-04 09:49:58 -0400, Tom Lane wrote: > Takashi Horikawa writes: > Why does this cause a core dump? We could consider fixing whatever > the problem is rather than capping the value. > > > As far as I experiment with my own evaluation environment using > > PostgreSQL-9.4.4 on a

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-08-04 Thread Tom Lane
Takashi Horikawa writes: Why does this cause a core dump? We could consider fixing whatever the problem is rather than capping the value. > As far as I experiment with my own evaluation environment using > PostgreSQL-9.4.4 on a x86_64 Linux, this problem can be fixed with the patch >

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 09:39, Simon Riggs wrote: > On 4 August 2015 at 06:03, Jeff Janes wrote: > > >> The attached proof of concept patch greatly improves the bloat for both >> the insert and the update cases. You need to turn on both features: adding >> the pages to fsm, and vacuuming the fsm, t

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
On 08/04/2015 03:15 PM, Robert Haas wrote: On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas wrote: >* The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync >with the list of individual locks in lwlock.h. Sooner or later someone will >add an LWLock and forget to update the

Re: [HACKERS] [PATCH] postgres_fdw extension support

2015-08-04 Thread Paul Ramsey
Thanks so much Michael! Let me know when you have further feedback I should incorporate. ATB, P.  --  http://postgis.net http://cleverelephant.ca On July 25, 2015 at 4:52:11 AM, Michael Paquier (michael.paqu...@gmail.com) wrote: On Sat, Jul 25, 2015 at 2:19 AM, Paul Ramsey wrote: > On Th

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Robert Haas
On Tue, Jul 28, 2015 at 3:28 PM, Heikki Linnakangas wrote: > * The patch requires that the LWLOCK_INDIVIDUAL_NAMES array is kept in sync > with the list of individual locks in lwlock.h. Sooner or later someone will > add an LWLock and forget to update the names-array. That needs to be made > less

Re: [HACKERS] Support for N synchronous standby servers - take 2

2015-08-04 Thread Beena Emerson
Robert Haas wrote: >Maybe shoehorning this into the GUC mechanism is the wrong thing, and >what we really need is a new config file for this. The information >we're proposing to store seems complex enough to justify that. > I think the consensus is that JSON is better. And using a new file with m

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread Ashutosh Bapat
On Tue, Aug 4, 2015 at 4:00 PM, David Rowley wrote: > On 4 August 2015 at 21:56, Ashutosh Bapat > wrote: > >> This looks like one example of general problem of finding optimal order >> for SQL operations. Consider a query defined as sql_op1(sql_op2(sql_op3(A, >> B), sql_op4(C, D), sql_op5(E, F))

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-08-04 Thread Ildus Kurbangaliev
On 08/03/2015 04:25 PM, Ildus Kurbangaliev wrote: On 07/28/2015 10:28 PM, Heikki Linnakangas wrote: On 07/27/2015 01:20 PM, Ildus Kurbangaliev wrote: Hello. In the attached patch I've made a refactoring for tranches. The prefix for them was extended, and I've did a split of LWLockAssign to two

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread David Rowley
On 4 August 2015 at 21:56, Ashutosh Bapat wrote: > This looks like one example of general problem of finding optimal order > for SQL operations. Consider a query defined as sql_op1(sql_op2(sql_op3(A, > B), sql_op4(C, D), sql_op5(E, F where sql_op can be SQL operation like > join, grouping, ag

Re: [HACKERS] GROUP BY before JOIN

2015-08-04 Thread Ashutosh Bapat
This looks like one example of general problem of finding optimal order for SQL operations. Consider a query defined as sql_op1(sql_op2(sql_op3(A, B), sql_op4(C, D), sql_op5(E, F where sql_op can be SQL operation like join, grouping, aggregation, projection, union, intersection, limit, ordering

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Amit Langote
On 2015-08-04 PM 05:58, Geoff Winkless wrote: > > ​Although it seems Amit has defined the problem better than I could, so > this is a bit late to the party (!), yes, the table had been ALTERed after > it was created (looking back through the history, that modification > included at least one DROP

Re: [HACKERS] Autonomous Transaction is back

2015-08-04 Thread Rajeev rastogi
On 03 August 2015 18:40, Merlin Moncure [mailto:mmonc...@gmail.com] Wrote: >On Sun, Aug 2, 2015 at 11:37 PM, Rajeev rastogi > wrote: >> On 31 July 2015 23:10, Robert Haas Wrote: >>>I think we're going entirely down the wrong path here. Why is it ever >useful for a backend's lock requests to confli

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Geoff Winkless
On 4 August 2015 at 09:30, Amit Langote wrote: > On 2015-08-04 AM 02:57, Peter Geoghegan wrote: > > On Mon, Aug 3, 2015 at 8:53 AM, Geoff Winkless > wrote: > >> If I create a copy of the table using > >> > >> CREATE mytab (LIKE brokentab INCLUDING ALL); > >> INSERT INTO mytab SELECT * FROM broke

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-08-04 Thread Andres Freund
On 2015-08-03 14:15:27 +0900, Michael Paquier wrote: > On Sat, Aug 1, 2015 at 9:20 PM, Andres Freund wrote: > > On August 1, 2015 2:17:24 PM GMT+02:00, Michael Paquier wrote: > >>> For instance, if you told me to choose between ShareLock and > >>> ShareUpdateExclusiveLock I wouldn't know which one

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2015-08-04 Thread Shigeru Hanada
Hi Ashutosh, Sorry for leaving the thread. 2015-07-20 16:09 GMT+09:00 Ashutosh Bapat : > In find_user_mapping(), if the first cache search returns a valid tuple, it > is checked twice for validity, un-necessarily. Instead if the first search > returns a valid tuple, it should be returned immediat

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run "external sort"

2015-08-04 Thread Peter Geoghegan
On Tue, Aug 4, 2015 at 1:24 AM, Heikki Linnakangas wrote: > Yeah, something like that. To paraphrase, if I'm now understanding it > correctly, Peter's idea is: > > When all the tuples have been fed to tuplesort, and it's time to perform the > sort, quicksort all the tuples currently in the heap, i

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Simon Riggs
On 4 August 2015 at 06:03, Jeff Janes wrote: > The attached proof of concept patch greatly improves the bloat for both > the insert and the update cases. You need to turn on both features: adding > the pages to fsm, and vacuuming the fsm, to get the benefit (so JJ_GIN=3). > The first of those t

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-08-04 Thread Amit Langote
On 2015-08-04 AM 02:57, Peter Geoghegan wrote: > On Mon, Aug 3, 2015 at 8:53 AM, Geoff Winkless wrote: >> If I create a copy of the table using >> >> CREATE mytab (LIKE brokentab INCLUDING ALL); >> INSERT INTO mytab SELECT * FROM brokentab; > > Also, did you drop any columns from the original "br

[HACKERS] GROUP BY before JOIN

2015-08-04 Thread David Rowley
== Overview == As of today we always perform GROUP BY at the final stage, after each relation in the query has been joined. This of course works, but it's not always the most optimal way of executing the query. Consider the following two relations: create table product (product_id int primary ke

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run "external sort"

2015-08-04 Thread Heikki Linnakangas
On 08/03/2015 11:36 PM, Robert Haas wrote: On Mon, Aug 3, 2015 at 3:33 PM, Peter Geoghegan wrote: When it's time to drain the heap, in performsort, divide the array into two arrays, based on the run number of each tuple, and then quicksort the arrays separately. The first array becomes the in-m

Re: [HACKERS] WIP: SCRAM authentication

2015-08-04 Thread Michael Paquier
On Mon, Mar 30, 2015 at 7:52 PM, Heikki Linnakangas wrote: > There have been numerous threads on replacing our MD5 authentication > method, so I started hacking on that to see what it might look like. Just > to be clear, this is 9.6 material. Attached is a WIP patch series that adds > support for

Re: [HACKERS] FSM versus GIN pending list bloat

2015-08-04 Thread Heikki Linnakangas
On 08/04/2015 08:03 AM, Jeff Janes wrote: For a GIN index with fastupdate turned on, both the user backends and autoanalyze routine will clear out the pending list, pushing the entries into the normal index structure and deleting the pages used by the pending list. But those deleted pages will n