Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-19 Thread Robert Haas
On Mon, Jul 15, 2013 at 1:41 PM, Jeff Davis pg...@j-davis.com wrote: I'm of the opinion that we ought to extract the parts of the patch that hold the VM pin for longer, review those separately, and if they're good and desirable, apply them. I'm confused. My patch holds a VM page pinned for

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-15 Thread Jeff Davis
On Sun, 2013-07-14 at 23:06 -0400, Robert Haas wrote: Of course, there's a reason that PD_ALL_VISIBLE is not like a normal hint: we need to make sure that inserts/updates/deletes clear the VM bit. But my patch already addresses that by keeping the VM page pinned. I'm of the opinion that

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-14 Thread Robert Haas
On Fri, Jul 5, 2013 at 4:18 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-07-02 at 10:12 -0700, Jeff Davis wrote: Regardless, this is at least a concrete issue that I can focus on, and I appreciate that. Are scans of small tables the primary objection to this patch, or are there others?

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-05 Thread Jeff Davis
On Tue, 2013-07-02 at 10:12 -0700, Jeff Davis wrote: Regardless, this is at least a concrete issue that I can focus on, and I appreciate that. Are scans of small tables the primary objection to this patch, or are there others? If I solve it, will this patch make real progress? I had an idea

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-02 Thread Andres Freund
On 2013-07-01 19:52:57 -0700, Jeff Davis wrote: 2. Can you be more specific about the scenarios that you *are* concerned about? Preferably in a form that could be tested on a 64-core box; but at least some kind of analysis involving numbers. More page accesses is scary, but completely

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-02 Thread Andres Freund
On 2013-07-02 14:02:22 +0200, Andres Freund wrote: Data loaded: load.sql. Attached... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services DROP TABLE IF EXISTS kv; CREATE TABLE kv( id serial

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-02 Thread Jeff Davis
On Tue, 2013-07-02 at 14:02 +0200, Andres Freund wrote: Ok, so you want some benchmark results. I spent 20 minutes concocting some quick tests. Here you go: master (384f933046dc9e9a2b416f5f7b3be30b93587c63): tps = 155075.448341 (including connections establishing) tps = 155259.752267

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-02 Thread Andres Freund
On 2013-07-02 10:12:31 -0700, Jeff Davis wrote: On Tue, 2013-07-02 at 14:02 +0200, Andres Freund wrote: Ok, so you want some benchmark results. I spent 20 minutes concocting some quick tests. Here you go: master (384f933046dc9e9a2b416f5f7b3be30b93587c63): tps = 155075.448341 (including

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-02 Thread Jeff Davis
On Tue, 2013-07-02 at 19:34 +0200, Andres Freund wrote: Well, I still have my doubts that it's a good idea to remove this knowledge from the page level. And that's not primarily related to performance. Unfortunately a good part of judging architectural questions is gut feeling... The only

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Jeff Davis
On Sun, 2013-06-30 at 22:58 -0400, Robert Haas wrote: I thought that Jeff withdrew this patch. No -- was there a reason you thought that? I know it could use another round of testing before commit, and there may be a couple other things to clear up. But I don't want to invest a lot of time there

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Robert Haas
On Mon, Jul 1, 2013 at 1:21 PM, Jeff Davis pg...@j-davis.com wrote: On Sun, 2013-06-30 at 22:58 -0400, Robert Haas wrote: I thought that Jeff withdrew this patch. No -- was there a reason you thought that? I thought I remembered you saying you were going to abandon it in the face of

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Jeff Davis
On Mon, 2013-07-01 at 16:05 -0400, Robert Haas wrote: The other concern I remember being expressed (and not just by me, but by a number of people) is that your patch turns loss of a visibility map bit into a data corruption scenario, which it currently isn't. Right now, if your visibility map

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Robert Haas
On Mon, Jul 1, 2013 at 8:23 PM, Jeff Davis pg...@j-davis.com wrote: Can you point me to that criticism? Why can't you just drop the VM completely if it becomes corrupted? (You might be referring to another idea of mine that was related to Andres's proposal for getting rid of freezing.) One

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Jeff Davis
On Mon, 2013-07-01 at 20:59 -0400, Robert Haas wrote: One of several relevant emails is at: http://www.postgresql.org/message-id/51a7473c.6070...@vmware.com It is definitely possible that I am mixing up two different things. But if I am, I don't know what the other one is. I believe you

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-07-01 Thread Jeff Davis
On Mon, 2013-07-01 at 16:05 -0400, Robert Haas wrote: Well, I don't believe there's any way to really eliminate the contention concern completely. There's no way around the fact that it means more access to the visibility map, and I've seen recent (albeit circumstantial thus far) evidence

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-06-30 Thread Robert Haas
On Sat, Jun 29, 2013 at 11:24 AM, Robins rob...@pobox.com wrote: On 10 June 2013 00:17, Jeff Davis pg...@j-davis.com wrote: On Thu, 2013-05-30 at 10:07 -0700, Jeff Davis wrote: Come to think of it, even without the torn page checksum issue, do we really want to actively clear the

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-06-30 Thread Josh Berkus
I thought that Jeff withdrew this patch. He did, but nobody removed it from the commitfest --- partly because of a change of subject line breaking the thread. Bounced to returned with feedback now. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-06-29 Thread Robins
On 10 June 2013 00:17, Jeff Davis pg...@j-davis.com wrote: On Thu, 2013-05-30 at 10:07 -0700, Jeff Davis wrote: Come to think of it, even without the torn page checksum issue, do we really want to actively clear the all-visible flags after upgrade? Removed that from the patch and

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-05-30 Thread Heikki Linnakangas
On 30.05.2013 06:54, Jeff Davis wrote: Continuation of: http://www.postgresql.org/message-id/1353551097.11440.128.camel@sussancws0025 Rebased patch attached; no other changes. @@ -675,6 +675,16 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, } /*

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-05-30 Thread Heikki Linnakangas
On 30.05.2013 11:26, Heikki Linnakangas wrote: On 30.05.2013 06:54, Jeff Davis wrote: Continuation of: http://www.postgresql.org/message-id/1353551097.11440.128.camel@sussancws0025 Rebased patch attached; no other changes. @@ -675,6 +675,16 @@ lazy_scan_heap(Relation onerel, LVRelStats

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-05-30 Thread Jeff Davis
On Thu, 2013-05-30 at 11:32 +0300, Heikki Linnakangas wrote: That could cause a torn page and checksum failure if checksums are enabled. Thank you, I missed that in the rebase; it should be MarkBufferDirtyHint(). Come to think of it, even without the torn page checksum issue, do we