Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Noah Misch
On Thu, Jun 16, 2011 at 12:02:47AM +0100, Simon Riggs wrote: On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch n...@leadboat.com wrote: On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 12, 2011 at

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Simon Riggs
On Thu, Jun 16, 2011 at 3:47 PM, Noah Misch n...@leadboat.com wrote: On Thu, Jun 16, 2011 at 12:02:47AM +0100, Simon Riggs wrote: On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch n...@leadboat.com wrote: On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: On Mon, Jun 13, 2011 at 3:11 AM,

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Simon Riggs
On Thu, Jun 16, 2011 at 4:53 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with your suggested fix. Please ignore the previous patch, which was sent in error. Here's the fix. I'll apply this tomorrow morning if we all still agree. --  Simon Riggs  

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-16 Thread Noah Misch
On Thu, Jun 16, 2011 at 04:53:41PM +0100, Simon Riggs wrote: On Thu, Jun 16, 2011 at 3:47 PM, Noah Misch n...@leadboat.com wrote: Thanks. ?We still hit a conflict when btpo.xact == RecentGlobalXmin and the standby has a transaction older than any master transaction. ?This happens because

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-15 Thread Simon Riggs
On Tue, Jun 14, 2011 at 5:28 AM, Noah Misch n...@leadboat.com wrote: On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch n...@leadboat.com wrote: Assuming that

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Simon Riggs
On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch n...@leadboat.com wrote: I fully agree.  That said, if this works on the standby, we may as well also use it opportunistically on the master, to throttle bloat. As long as

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Simon Riggs
On Thu, Jun 9, 2011 at 10:38 PM, Noah Misch n...@leadboat.com wrote: On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011 12:40, Noah Misch

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Noah Misch
On Mon, Jun 13, 2011 at 04:16:06PM +0100, Simon Riggs wrote: On Mon, Jun 13, 2011 at 3:11 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch n...@leadboat.com wrote: Assuming that conclusion, I do think it's worth starting with something simple, even if

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-13 Thread Noah Misch
On Mon, Jun 13, 2011 at 04:41:11PM +0100, Simon Riggs wrote: On Thu, Jun 9, 2011 at 10:38 PM, Noah Misch n...@leadboat.com wrote: On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: In an attempt to resuscitate this thread, here's my own shot at that. ?Apologies in advance if

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-12 Thread Noah Misch
On Sun, Jun 12, 2011 at 12:15:29AM -0400, Robert Haas wrote: On Sat, Jun 11, 2011 at 11:40 PM, Noah Misch n...@leadboat.com wrote: We currently achieve that wait-free by first marking the page with the next available xid and then reusing it when that mark (btpo.xact) predates the oldest

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-12 Thread Robert Haas
On Sun, Jun 12, 2011 at 3:01 PM, Noah Misch n...@leadboat.com wrote: I fully agree.  That said, if this works on the standby, we may as well also use it opportunistically on the master, to throttle bloat. As long as the performance cost is de minimis, I agree. At any rate, if taking a

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Robert Haas
On Fri, Apr 22, 2011 at 11:10 AM, Noah Misch n...@leadboat.com wrote: On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Noah Misch
Hi Robert, On Sat, Jun 11, 2011 at 08:55:28PM -0400, Robert Haas wrote: On Fri, Apr 22, 2011 at 11:10 AM, Noah Misch n...@leadboat.com wrote: On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-11 Thread Robert Haas
On Sat, Jun 11, 2011 at 11:40 PM, Noah Misch n...@leadboat.com wrote: We currently achieve that wait-free by first marking the page with the next available xid and then reusing it when that mark (btpo.xact) predates the oldest running xid (RecentXmin).  (At the moment, I'm failing to work out

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-06-09 Thread Noah Misch
On Fri, Apr 22, 2011 at 11:10:34AM -0400, Noah Misch wrote: On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report recently

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-04-22 Thread Noah Misch
On Tue, Mar 15, 2011 at 10:22:59PM -0400, Noah Misch wrote: On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report recently upgraded from 9.0.1 to 9.0.3, and your fix did significantly

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-15 Thread Noah Misch
On Mon, Mar 14, 2011 at 01:56:22PM +0200, Heikki Linnakangas wrote: On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report recently upgraded from 9.0.1 to 9.0.3, and your fix did significantly decrease its conflict frequency. The last several conflicts I

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-14 Thread Heikki Linnakangas
On 12.03.2011 12:40, Noah Misch wrote: The installation that inspired my original report recently upgraded from 9.0.1 to 9.0.3, and your fix did significantly decrease its conflict frequency. The last several conflicts I have captured involve XLOG_BTREE_REUSE_PAGE records. (FWIW, the index has

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-12 Thread Noah Misch
On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: On 29.11.2010 08:10, Noah Misch wrote: I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2011-03-10 Thread Heikki Linnakangas
On 10.12.2010 19:55, Noah Misch wrote: On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: Seems reasonable. HeapTupleHeaderAdvanceLatestRemovedXid() will need similar treatment. Actually,

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-10 Thread Noah Misch
On Thu, Dec 09, 2010 at 09:48:25AM +, Simon Riggs wrote: On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: On 29.11.2010 08:10, Noah Misch wrote: I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-09 Thread Simon Riggs
On Fri, 2010-12-03 at 21:43 +0200, Heikki Linnakangas wrote: On 29.11.2010 08:10, Noah Misch wrote: I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a

Re: [HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-12-03 Thread Heikki Linnakangas
On 29.11.2010 08:10, Noah Misch wrote: I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's

[HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-11-28 Thread Noah Misch
I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's transactions. Even so, I kept seeing recovery pause