Re: [HACKERS] [Reveiw] Idle In Transaction Session Timeout, revived

2016-02-06 Thread Stéphane Schildknecht
On 31/01/2016 14:33, Vik Fearing wrote: > Attached is a rebased and revised version of my > idle_in_transaction_session_timeout patch from last year. > > This version does not suffer the problems the old one did where it would > jump out of SSL code thanks to Andres' patch in commit > 4f85fde8eb86

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 02:34 AM, Tom Lane wrote: I have sussed what's happening in bug #13908. Basically, commit 45f6240a8fa9d355 ("Pack tuples in a hash join batch densely, to save memory") broke things for the case where a hash join is using a skew table. The reason is that that commit only changed the

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Michael Paquier
On Sat, Feb 6, 2016 at 2:11 AM, Tomas Vondra wrote: > On 02/04/2016 09:59 AM, Michael Paquier wrote: >> >> On Tue, Feb 2, 2016 at 9:59 AM, Andres Freund wrote: >>> >>> On 2016-02-02 09:56:40 +0900, Michael Paquier wrote: And there is no actual risk of data loss >>> >>> >>> Huh? >> >> >>

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Michael Paquier
On Thu, Feb 4, 2016 at 6:38 PM, Michael Paquier wrote: > On Thu, Feb 4, 2016 at 4:10 PM, Andres Freund wrote: >> On 2016-02-04 18:21:41 +0530, Amit Kapila wrote: >>> I think generally it is good idea, but one thing worth a thought is that >>> by doing so, we need to acquire all WAL Insertion lock

Re: [HACKERS] proposal: make NOTIFY list de-duplication optional

2016-02-06 Thread Brendan Jurd
On Sat, 6 Feb 2016 at 12:50 Tom Lane wrote: > Robert Haas writes: > > I agree with what Merlin said about this: > > > http://www.postgresql.org/message-id/CAHyXU0yoHe8Qc=yc10ahu1nfia1tbhsg+35ds-oeueuapo7...@mail.gmail.com > > Yeah, I agree that a GUC for this is quite unappetizing. > How would

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Tomas Vondra
Hi, On 02/06/2016 01:16 PM, Michael Paquier wrote: On Sat, Feb 6, 2016 at 2:11 AM, Tomas Vondra wrote: On 02/04/2016 09:59 AM, Michael Paquier wrote: On Tue, Feb 2, 2016 at 9:59 AM, Andres Freund wrote: On 2016-02-02 09:56:40 +0900, Michael Paquier wrote: And there is no actual risk of

Re: [HACKERS] proposal: make NOTIFY list de-duplication optional

2016-02-06 Thread Tom Lane
Brendan Jurd writes: > On Sat, 6 Feb 2016 at 12:50 Tom Lane wrote: >> Yeah, I agree that a GUC for this is quite unappetizing. > How would you feel about a variant for calling NOTIFY? If we decide that this ought to be user-visible, then an extra NOTIFY parameter would be the way to do it. I'd

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Andres Freund
On 2016-02-06 17:43:48 +0100, Tomas Vondra wrote: > >Still the data is here... But well. I won't insist. > > Huh? This thread started by an example how to cause loss of committed > transactions. That fits my definition of "data loss" quite well. Agreed, that view doesn't seem to make much sense.

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tom Lane
Tomas Vondra writes: > On 02/06/2016 02:34 AM, Tom Lane wrote: >> I have sussed what's happening in bug #13908. Basically, commit >> 45f6240a8fa9d355 ("Pack tuples in a hash join batch densely, to save >> memory") broke things for the case where a hash join is using a skew >> table. > Damn, that'

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Andres Freund
On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: > The patch attached will apply on master, on 9.5 there is one minor > conflict. For older versions we will need another reworked patch. FWIW, I don't think we should backpatch this. It'd look noticeably different in the back branches, and this

Re: [HACKERS] proposal: make NOTIFY list de-duplication optional

2016-02-06 Thread Andrew Dunstan
On 02/05/2016 08:49 PM, Tom Lane wrote: Yeah, I agree that a GUC for this is quite unappetizing. Agreed. One idea would be to build a hashtable to aid with duplicate detection (perhaps only once the pending-notify list gets long). Another thought is that it's already the case that duplic

[HACKERS] Optimization- Check the set of conditionals on a WHERE clause against CHECK constraints.

2016-02-06 Thread Shubham Barai
Hello hackers, I was searching for project ideas and found this 1.Optimization- Check the set of conditionals on a WHERE clause against CHECK constraints on the table being queried and remove any conditionals which *must* be true due to the CHECK constraints. Is it expensive for simple queries

Re: [HACKERS] proposal: make NOTIFY list de-duplication optional

2016-02-06 Thread Filip Rembiałkowski
On Sat, Feb 6, 2016 at 5:52 PM, Tom Lane wrote: > Brendan Jurd writes: >> On Sat, 6 Feb 2016 at 12:50 Tom Lane wrote: >>> Yeah, I agree that a GUC for this is quite unappetizing. > >> How would you feel about a variant for calling NOTIFY? > > If we decide that this ought to be user-visible, then

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 06:47 PM, Tom Lane wrote: Tomas Vondra writes: On 02/06/2016 02:34 AM, Tom Lane wrote: I have sussed what's happening in bug #13908. Basically, commit 45f6240a8fa9d355 ("Pack tuples in a hash join batch densely, to save memory") broke things for the case where a hash join is usin

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Andres Freund
On 2016-02-06 20:34:07 +0100, Tomas Vondra wrote: > On 02/06/2016 06:47 PM, Tom Lane wrote: > >* It incorporates a bespoke reimplementation of palloc into hash > >joins. This is not a maintainable/scalable way to go about reducing > >memory consumption. It should have been done with an arm's-length

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 08:39 PM, Andres Freund wrote: On 2016-02-06 20:34:07 +0100, Tomas Vondra wrote: On 02/06/2016 06:47 PM, Tom Lane wrote: * It incorporates a bespoke reimplementation of palloc into hash joins. This is not a maintainable/scalable way to go about reducing memory consumption. It sh

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tom Lane
Tomas Vondra writes: > On 02/06/2016 08:39 PM, Andres Freund wrote: >> FWIW, I've done that at some point. Noticeable speedups (that's what >> I cared about), but a bit annoying to use. There's many random >> pfree()s around, and then there's MemoryContextContains(), >> GetMemoryChunkContext(), Ge

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tom Lane
Tomas Vondra writes: > On 02/06/2016 06:47 PM, Tom Lane wrote: >> I note also that while the idea of ExecHashRemoveNextSkewBucket is >> to reduce memory consumed by the skew table to make it available to >> the main hash table, in point of fact it's unlikely that the freed >> space will be of any

Re: [HACKERS] Optimization- Check the set of conditionals on a WHERE clause against CHECK constraints.

2016-02-06 Thread Tom Lane
Shubham Barai writes: > I was searching for project ideas and found this > 1.Optimization- Check the set of conditionals on a WHERE clause against > CHECK constraints on the table being queried and remove any conditionals > which *must* be true due to the CHECK constraints. > Is it expensive fo

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 09:55 PM, Tom Lane wrote: Tomas Vondra writes: On 02/06/2016 06:47 PM, Tom Lane wrote: I note also that while the idea of ExecHashRemoveNextSkewBucket is to reduce memory consumed by the skew table to make it available to the main hash table, in point of fact it's unlikely that

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tom Lane
Tomas Vondra writes: > I believe the attached patch should fix this by actually copying the > tuples into the densely allocated chunks. Haven't tested it though, will > do in a few hours. BTW, I confirmed that this patch fixes the wrong-number-of-output-tuples issue in the test case from bug #1

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tom Lane
Tomas Vondra writes: > What about using the dense allocation even for the skew buckets, but not > one context for all skew buckets but one per bucket? Then when we delete > a bucket, we simply destroy the context (and free the chunks, just like > we do with the current dense allocator). Yeah,

[HACKERS] Recently added typedef "string" is a horrid idea

2016-02-06 Thread Tom Lane
I see that commit b47b4dbf6 added this to varlena.c: typedef struct varlena string; This is a remarkably bad idea. It will cause pgindent to do strange things anywhere it sees a variable or field named "string", of which we have quite a few. Remember that the effects of typedef names ar

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-02-06 Thread Michael Paquier
On Sun, Feb 7, 2016 at 2:49 AM, Andres Freund wrote: > On 2016-02-06 22:03:15 +0900, Michael Paquier wrote: >> + /* >> + * Update the progress LSN positions. At least one WAL insertion lock >> + * is already taken appropriately before doing that, and it is just >> more >> + * s

Re: [HACKERS] Recently added typedef "string" is a horrid idea

2016-02-06 Thread Robert Haas
On Sat, Feb 6, 2016 at 5:11 PM, Tom Lane wrote: > I see that commit b47b4dbf6 added this to varlena.c: > > typedef struct varlena string; > > This is a remarkably bad idea. It will cause pgindent to do strange > things anywhere it sees a variable or field named "string", of which > we hav

Re: [HACKERS] proposal: make NOTIFY list de-duplication optional

2016-02-06 Thread Filip Rembiałkowski
+1 ... and a patch (only adding ALL keyword, no hash table implemented yet). On Sat, Feb 6, 2016 at 2:35 PM, Brendan Jurd wrote: > On Sat, 6 Feb 2016 at 12:50 Tom Lane wrote: >> >> Robert Haas writes: >> > I agree with what Merlin said about this: >> > >> > http://www.postgresql.org/message-

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2016-02-06 Thread Robert Haas
On Sat, Feb 6, 2016 at 12:46 AM, Ashutosh Bapat wrote: > Here it is rebased. Thanks for the pgindent run and committing core changes. > I have to manage only one patch now :) > > pgindent is giving trouble with following two comments > > 2213 /* Run time cost includes: > 2214

Re: [HACKERS] pgbench small bug fix

2016-02-06 Thread Robert Haas
On Wed, Jan 27, 2016 at 2:31 PM, Fabien COELHO wrote: > > While testing for something else I encountered two small bugs under very low > rate (--rate=0.1). The attached patches fixes these. > > - when a duration (-T) is specified, ensure that pgbench ends at that >time (i.e. do not wait for a

Re: [HACKERS] WIP: Make timestamptz_out less slow.

2016-02-06 Thread Tom Lane
David Rowley writes: [ timestamp_out_speedup_2015-11-05.patch ] Pushed with a bunch of cosmetic tweaks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsq

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

2016-02-06 Thread Robert Haas
On Wed, Feb 3, 2016 at 11:57 PM, Kouhei Kaigai wrote: > At this moment, I tried to write up description at nodes/nodes.h. > The amount of description is about 100lines. It is on a borderline > whether we split off this chunk into another header file, in my sense. > > > On the other hands, I notice

Re: [HACKERS] Patch: fix lock contention for HASHHDR.mutex

2016-02-06 Thread Robert Haas
On Thu, Jan 21, 2016 at 9:03 AM, Anastasia Lubennikova wrote: > First of all, why not merge both patches into one? They aren't too big > anyway. So looking over this patch, it institutes a new coding rule that all shared hash tables must have the same number of partitions, and that number is hard

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-06 Thread Amit Kapila
On Tue, Feb 2, 2016 at 5:28 AM, Andres Freund wrote: > > Hi, > > currently if, when not in standby mode, we can't read a checkpoint > record, we automatically fall back to the previous checkpoint, and start > replay from there. > > Doing so without user intervention doesn't actually seem like a go

Re: [HACKERS] "using previous checkpoint record at" maybe not the greatest idea?

2016-02-06 Thread Amit Kapila
On Sun, Feb 7, 2016 at 10:54 AM, Amit Kapila wrote: > > On Tue, Feb 2, 2016 at 5:28 AM, Andres Freund wrote: > > > > Hi, > > > > currently if, when not in standby mode, we can't read a checkpoint > > record, we automatically fall back to the previous checkpoint, and start > > replay from there. >