Re: [HACKERS] visibility maps and heap_prune

2010-02-27 Thread Heikki Linnakangas
Bruce Momjian wrote: Pavan Deolasee wrote: On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: Though Alex did some useful tests and review, and in fact

Re: [HACKERS] visibility maps and heap_prune

2010-02-27 Thread Bruce Momjian
Heikki Linnakangas wrote: Bruce Momjian wrote: Pavan Deolasee wrote: On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: Though Alex did some useful

Re: [HACKERS] visibility maps and heap_prune

2010-02-26 Thread Bruce Momjian
Pavan Deolasee wrote: On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: Though Alex did some useful tests and review, and in fact confirmed that the

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Bruce Momjian
Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: https://commitfest.postgresql.org/action/patch_view?id=75 --- Pavan Deolasee wrote: ISTM that the

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this?  It was in the first 9.0 commitfest but was returned with feedback but never updated:        https://commitfest.postgresql.org/action/patch_view?id=75 Well, the patch author chose not to pursue

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Bruce Momjian
Robert Haas wrote: On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? ?It was in the first 9.0 commitfest but was returned with feedback but never updated: ? ? ? ?https://commitfest.postgresql.org/action/patch_view?id=75 Well, the patch

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 10:32 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Thu, Feb 25, 2010 at 9:49 PM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? ?It was in the first 9.0 commitfest but was returned with feedback but never updated: ? ? ?

Re: [HACKERS] visibility maps and heap_prune

2010-02-25 Thread Pavan Deolasee
On Fri, Feb 26, 2010 at 8:19 AM, Bruce Momjian br...@momjian.us wrote: Whatever happened to this? It was in the first 9.0 commitfest but was returned with feedback but never updated: Though Alex did some useful tests and review, and in fact confirmed that the VACUUM time dropped from 16494

Re: [HACKERS] visibility maps and heap_prune

2009-07-25 Thread Robert Haas
On Tue, Jul 21, 2009 at 2:37 AM, Pavan Deolaseepavan.deola...@gmail.com wrote: On Tue, Jul 21, 2009 at 10:38 AM, Robert Haasrobertmh...@gmail.com wrote: Pavan, are you planning to respond to Alex's comments and/or update this patch? Yes, I will. Hopefully  by end of this week. Since it has

Re: [HACKERS] visibility maps and heap_prune

2009-07-21 Thread Pavan Deolasee
On Tue, Jul 21, 2009 at 10:38 AM, Robert Haasrobertmh...@gmail.com wrote: Pavan, are you planning to respond to Alex's comments and/or update this patch? Yes, I will. Hopefully by end of this week. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com -- Sent

Re: [HACKERS] visibility maps and heap_prune

2009-07-20 Thread Robert Haas
On Wed, Jul 15, 2009 at 11:44 PM, Alex Hunsakerbada...@gmail.com wrote: On Mon, Dec 8, 2008 at 06:56, Pavan Deolaseepavan.deola...@gmail.com wrote: Here is a patch which implements this. The PD_ALL_VISIBLE flag is set if all tuples in the page are visible to all transactions and there are no

Re: [HACKERS] visibility maps and heap_prune

2009-07-15 Thread Alex Hunsaker
On Mon, Dec 8, 2008 at 06:56, Pavan Deolaseepavan.deola...@gmail.com wrote: Here is a patch which implements this. The PD_ALL_VISIBLE flag is set if all tuples in the page are visible to all transactions and there are no DEAD line pointers in the page. The second check is required so that

Re: [HACKERS] visibility maps and heap_prune

2009-01-20 Thread Bruce Momjian
Pavan Deolasee wrote: On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee pavan.deola...@gmail.comwrote: On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: If you see a straightforward way, please submit a patch! Will do that.

Re: [HACKERS] visibility maps

2009-01-20 Thread Bruce Momjian
Tom Lane wrote: Pavan Deolasee pavan.deola...@gmail.com writes: OTOH I think we can still set PD_ALL_VISIBLE page header flag even when there are DEAD line pointers. That would mean the header flag means something different than the map bit does, which would mean extra tests need to be

Re: [HACKERS] visibility maps and heap_prune

2009-01-20 Thread Heikki Linnakangas
Bruce Momjian wrote: Pavan Deolasee wrote: On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee pavan.deola...@gmail.comwrote: On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: If you see a straightforward way, please submit a patch! Will do that.

Re: [HACKERS] visibility maps

2009-01-20 Thread Heikki Linnakangas
Bruce Momjian wrote: Tom Lane wrote: Pavan Deolasee pavan.deola...@gmail.com writes: OTOH I think we can still set PD_ALL_VISIBLE page header flag even when there are DEAD line pointers. That would mean the header flag means something different than the map bit does, which would mean extra

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Pavan Deolasee
On Thu, Jan 15, 2009 at 7:10 AM, Bruce Momjian br...@momjian.us wrote: Is this something for 8.4 CVS? I worked out the patch as per Heikki's suggestion. So I think he needs to review and decide it's fate. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com -- Sent

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Jan 15, 2009 at 7:10 AM, Bruce Momjian br...@momjian.us wrote: Is this something for 8.4 CVS? I worked out the patch as per Heikki's suggestion. So I think he needs to review and decide it's fate. Yeah, I dropped the ball on that one. It's been knocking in the

Re: [HACKERS] visibility maps and heap_prune

2009-01-15 Thread Robert Haas
Yeah, I dropped the ball on that one. It's been knocking in the back of my head since, but I've never gotten around. I'm feeling reluctant to review it since it's not really a high priority thing, and I'm not sure whether we want it or not. In that case perhaps we should add it to

Re: [HACKERS] visibility maps and heap_prune

2009-01-14 Thread Bruce Momjian
Is this something for 8.4 CVS? --- Pavan Deolasee wrote: On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee pavan.deola...@gmail.comwrote: On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas

Re: [HACKERS] visibility maps

2009-01-14 Thread Bruce Momjian
Is there anything to do with the below issue? --- Pavan Deolasee wrote: On Wed, Dec 17, 2008 at 7:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think what you are suggesting is that we should set the visibility map

Re: [HACKERS] visibility maps

2008-12-17 Thread Tom Lane
Pavan Deolasee pavan.deola...@gmail.com writes: OTOH I think we can still set PD_ALL_VISIBLE page header flag even when there are DEAD line pointers. That would mean the header flag means something different than the map bit does, which would mean extra tests need to be made before propagating

Re: [HACKERS] visibility maps

2008-12-17 Thread Pavan Deolasee
On Wed, Dec 17, 2008 at 3:29 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't quite understand this paragraph. If there's any DEAD tuples or line-pointers, the all-visible flag can't be set. After an UPDATE or DELETE, it indeed takes two vacuums until the bits in the

Re: [HACKERS] visibility maps

2008-12-17 Thread Heikki Linnakangas
Pavan Deolasee wrote: Another thing I noticed is the since VACUUM tries to set the bit in the first phase, it's working only because HOT prunes DEAD tuples just before we do another scan on line pointers (which I had earlier talked about getting rid of. May be its time I do that). Otherwise, the

Re: [HACKERS] visibility maps

2008-12-17 Thread Pavan Deolasee
On Wed, Dec 17, 2008 at 7:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think what you are suggesting is that we should set the visibility map bit while dead line pointers (tombstones) still remain. If that's what you meant it's a bad idea. No, I'm not suggesting that. I understand the

Re: [HACKERS] visibility maps

2008-12-17 Thread Tom Lane
Pavan Deolasee pavan.deola...@gmail.com writes: On Wed, Dec 17, 2008 at 3:29 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't quite understand this paragraph. If there's any DEAD tuples or line-pointers, the all-visible flag can't be set. No, I am saying, HOT-prune

Re: [HACKERS] visibility maps

2008-12-12 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 8:09 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Pavan Deolasee wrote: I can do some if we don't have already. Oh, yes please! I did some tests today with pgbench on a decent SMP machine. The goal was to see if multiple clients (20 in the test)

Re: [HACKERS] visibility maps

2008-12-12 Thread Dimitri Fontaine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Le 12 déc. 08 à 13:11, Pavan Deolasee a écrit : I did some tests today with pgbench on a decent SMP machine. The goal was to see if multiple clients (20 in the test) tries to update tuples in different data blocks, if the EX lock on the VM page

Re: [HACKERS] visibility maps

2008-12-11 Thread Zdenek Kotala
Heikki Linnakangas napsal(a): Pavan Deolasee wrote: /* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] (1 mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala [EMAIL PROTECTED] wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to assembler. But this code seems to me

Re: [HACKERS] visibility maps

2008-12-11 Thread Zdenek Kotala
Pavan Deolasee napsal(a): On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala [EMAIL PROTECTED] wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to assembler. But

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 7:03 PM, Zdenek Kotala [EMAIL PROTECTED] wrote: Yes, because it is not simple write operation. You need to read byte from memory to register, set bit and write it back. Write memory itself is atomic but somebody could change other bits between read and write. Yeah,

Re: [HACKERS] visibility maps

2008-12-11 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala [EMAIL PROTECTED] wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to assembler. But this

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 7:15 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Yeah, if we accept that bits can be bogusly set. There is scenarios where that can happen already, but they involve crashing, not during normal operation and clean shut down. In the future, I'd like to move in the

Re: [HACKERS] visibility maps

2008-12-11 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: On Thu, Dec 11, 2008 at 5:01 PM, Zdenek Kotala [EMAIL PROTECTED] wrote: IIRC, Memory reading/writing is atomic operation. Only one CPU(hw thread) can access to the same memory address(es)* in same time*. The question is how compiler compile C code to

Re: [HACKERS] visibility maps

2008-12-11 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Thu, Dec 11, 2008 at 7:15 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Do we have any tests to prove that the VM page lock does not indeed become a bottleneck ? No. I can do some if we don't have already. Oh, yes please! Only the first update to a page needs

Re: [HACKERS] visibility maps

2008-12-11 Thread Pavan Deolasee
On Thu, Dec 11, 2008 at 8:09 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: I'm not sure if we should set the bits in very aggressively. If we're more aggressive about setting the bits, it also means that we have to clear the bits more often, increasing the likelihood of contention that you

Re: [HACKERS] visibility maps and heap_prune

2008-12-08 Thread Pavan Deolasee
On Mon, Dec 8, 2008 at 11:33 AM, Pavan Deolasee [EMAIL PROTECTED]wrote: On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: If you see a straightforward way, please submit a patch! Will do that. Here is a patch which implements this. The PD_ALL_VISIBLE flag

Re: [HACKERS] visibility maps and heap_prune

2008-12-07 Thread Pavan Deolasee
On Sat, Dec 6, 2008 at 8:08 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: If you see a straightforward way, please submit a patch! Will do that. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: /* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] (1 mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read atomically on all platforms.

Re: [HACKERS] visibility maps and heap_prune

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: ISTM that the PD_ALL_VISIBLE flag and the visibility map bit can be set at the end of pruning operation if we know that there are only tuples visible to all transactions left in the page. Right. The way pruning is done, I think it would be straight forward to get this

Re: [HACKERS] visibility maps

2008-12-06 Thread Pavan Deolasee
On Sat, Dec 6, 2008 at 7:57 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Umm, what non-atomic state could the bit be in? Half-set, half-cleared? Or do you think that if some other bit in proximity is changed, the other bit would temporarily flip 0-1-0, or something like that? I don't

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: On Sat, Dec 6, 2008 at 7:57 PM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Umm, what non-atomic state could the bit be in? Half-set, half-cleared? Or do you think that if some other bit in proximity is changed, the other bit would temporarily flip 0-1-0, or something

Re: [HACKERS] visibility maps

2008-12-06 Thread Heikki Linnakangas
Pavan Deolasee wrote: /* * Size of the bitmap on each visibility map page, in bytes. There's no * extra headers, so the whole page minus except for the standard page header * is used for the bitmap. */ #define MAPSIZE (BLCKSZ - SizeOfPageHeaderData) ISTM that we should MAXALIGN the

[HACKERS] visibility maps and heap_prune

2008-12-04 Thread Pavan Deolasee
ISTM that the PD_ALL_VISIBLE flag and the visibility map bit can be set at the end of pruning operation if we know that there are only tuples visible to all transactions left in the page. The way pruning is done, I think it would be straight forward to get this information. Thanks, Pavan --

[HACKERS] visibility maps

2008-12-04 Thread Pavan Deolasee
/* * We don't need to lock the page, as we're only looking at a single bit. */ result = (map[mapByte] (1 mapBit)) ? true : false; Isn't this a dangerous assumption to make ? I am not so sure that even a bit can be read atomically on all platforms. I think the only caller of

Re: [HACKERS] visibility maps

2008-12-04 Thread Pavan Deolasee
/* * Size of the bitmap on each visibility map page, in bytes. There's no * extra headers, so the whole page minus except for the standard page header * is used for the bitmap. */ #define MAPSIZE (BLCKSZ - SizeOfPageHeaderData) ISTM that we should MAXALIGN the SizeOfPageHeaderData to compute