Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-20 Thread Craig Ringer
On 01/18/2013 11:50 PM, Robert Haas wrote: On Fri, Jan 18, 2013 at 10:36 AM, Merlin Moncure mmonc...@gmail.com wrote: Any scenario that involves non-trivial amount of investigation or development should result in us pulling the patch for rework and resubmission in later 'festit's closing

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Craig Ringer
On 12/14/2012 09:57 PM, Amit Kapila wrote: I need to validate the vacuum results. It's possible that this is solvable by tweaking xmin check inside vacuum. Assuming that's fixed, the question stands: do the results justify the change? I'd argue 'maybe' We can try with change (assuming

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Merlin Moncure
On Fri, Jan 18, 2013 at 5:34 AM, Craig Ringer cr...@2ndquadrant.com wrote: On 12/14/2012 09:57 PM, Amit Kapila wrote: I need to validate the vacuum results. It's possible that this is solvable by tweaking xmin check inside vacuum. Assuming that's fixed, the question stands: do the results

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Atri Sharma
On 18-Jan-2013, at 17:04, Craig Ringer cr...@2ndquadrant.com wrote: On 12/14/2012 09:57 PM, Amit Kapila wrote: I need to validate the vacuum results. It's possible that this is solvable by tweaking xmin check inside vacuum. Assuming that's fixed, the question stands: do the results

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Merlin Moncure
On Fri, Jan 18, 2013 at 8:57 AM, Atri Sharma atri.j...@gmail.com wrote: Hello all, Sorry for the delay in updating the hackers list with the current status. I recently did some profiling using perf on PostgreSQL 9.2 with and without our patch. I noticed that maximum time is being spent

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Robert Haas
On Fri, Jan 18, 2013 at 10:36 AM, Merlin Moncure mmonc...@gmail.com wrote: Any scenario that involves non-trivial amount of investigation or development should result in us pulling the patch for rework and resubmission in later 'festit's closing time as they say :-). Amen. -- Robert Haas

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-14 Thread Amit Kapila
On Thursday, December 13, 2012 8:02 PM Merlin Moncure wrote: On Thu, Dec 13, 2012 at 7:06 AM, Hari Babu haribabu.ko...@huawei.com wrote: Please find the review of the patch. Thanks for detailed review! Basic stuff: - Patch applies with offsets. - Compiles cleanly

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-13 Thread Hari Babu
On Thu, Dec 7, 2012 at 7:56 PM, Hari babu haribabu(dot)kommi(at)Huawei(dot)com wrote: On Thu, Dec 6, 2012 at 8:52 PM, Merlin Moncure mmonc...@gmail.com wrote: Thanks for that -- that's fairly comprehensive I'd say. I'm quite interested in that benchmarking framework as well. Do you need

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-13 Thread Atri Sharma
On Thu, Dec 13, 2012 at 6:36 PM, Hari Babu haribabu.ko...@huawei.comwrote: On Thu, Dec 7, 2012 at 7:56 PM, Hari babu haribabu(dot)kommi(at)Huawei(dot)com wrote: On Thu, Dec 6, 2012 at 8:52 PM, Merlin Moncure mmonc...@gmail.com wrote: Thanks for that -- that's fairly comprehensive

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-13 Thread Merlin Moncure
On Thu, Dec 13, 2012 at 7:06 AM, Hari Babu haribabu.ko...@huawei.com wrote: Please find the review of the patch. Thanks for detailed review! Basic stuff: - Patch applies with offsets. - Compiles cleanly with no warnings - Regression Test pass. Code Review: -

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-07 Thread Hari Babu
On Thu, Dec 6, 2012 at 8:52 PM, Merlin Moncure mmonc...@gmail.com wrote: Thanks for that -- that's fairly comprehensive I'd say. I'm quite interested in that benchmarking framework as well. Do you need help setting up the scripts? Presently I am testing with pgbench custom query option taking

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-06 Thread Amit Kapila
On Thursday, November 22, 2012 3:00 AM Greg Smith wrote: On 11/16/12 9:03 AM, Merlin Moncure wrote: Atri ran some quick n dirty tests to see if there were any regressions. He benched a large scan followed by vacuum. So far, results are inconclusive so better testing methodologies will

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-06 Thread Merlin Moncure
On Thu, Dec 6, 2012 at 3:59 AM, Amit Kapila amit.kap...@huawei.com wrote: On Thursday, November 22, 2012 3:00 AM Greg Smith wrote: On 11/16/12 9:03 AM, Merlin Moncure wrote: Atri ran some quick n dirty tests to see if there were any regressions. He benched a large scan followed by vacuum.

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-28 Thread Alvaro Herrera
Merlin Moncure escribió: Maybe abstracting 'last xid cache' along with hint bit management out of both transam.c and tqual.c into something like 'hints.c' is appropriate, but that's a more invasive change. It would be good to have such a patch to measure/compare performance of both

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-21 Thread Greg Smith
On 11/16/12 9:03 AM, Merlin Moncure wrote: Atri ran some quick n dirty tests to see if there were any regressions. He benched a large scan followed by vacuum. So far, results are inconclusive so better testing methodologies will definitely be greatly appreciated. One of the challenges with

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-16 Thread Amit Kapila
On Thursday, November 15, 2012 10:19 PM Merlin Moncure wrote: On Thu, Nov 15, 2012 at 10:25 AM, Amit Kapila amit.kap...@huawei.com wrote: Sure, although in scans we are using ring buffer as well so in practical sense the results are pretty close. b. Considering sometimes people want

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-16 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 4:32 AM, Amit Kapila amit.kap...@huawei.com wrote: On Thursday, November 15, 2012 10:19 PM Merlin Moncure wrote: On Thu, Nov 15, 2012 at 10:25 AM, Amit Kapila amit.kap...@huawei.com wrote: Sure, although in scans we are using ring buffer as well so in practical sense

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-16 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 6:42 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2012-11-06 at 17:55 -0600, Merlin Moncure wrote: So given that -- the patch simple adds an extra check when/where hint bit status is checked in the visibility routines (currently, only HeapTupleSatisfiesMVCC is done

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-16 Thread Atri Sharma
On Fri, Nov 16, 2012 at 7:33 PM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, Nov 16, 2012 at 4:32 AM, Amit Kapila amit.kap...@huawei.com wrote: On Thursday, November 15, 2012 10:19 PM Merlin Moncure wrote: On Thu, Nov 15, 2012 at 10:25 AM, Amit Kapila amit.kap...@huawei.com wrote:

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-15 Thread Amit Kapila
On Thursday, November 15, 2012 2:02 AM Atri Sharma wrote: On Thu, Nov 15, 2012 at 12:42 AM, Atri Sharma atri.j...@gmail.com wrote: On Wed, Nov 7, 2012 at 9:47 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 4:39 AM, Amit Kapila amit.kap...@huawei.com wrote: In each visibility function (except HeapTupleSatisfiesVacuum() ), an addition check has been added to check if the commit status of Xmin or Xmax of a tuple can be retrieved from the cache. 1. From your

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-15 Thread Amit Kapila
On Thursday, November 15, 2012 9:27 PM Merlin Moncure wrote: On Thu, Nov 15, 2012 at 4:39 AM, Amit Kapila amit.kap...@huawei.com wrote: In each visibility function (except HeapTupleSatisfiesVacuum() ), an addition check has been added to check if the commit status of Xmin or Xmax of a

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 10:25 AM, Amit Kapila amit.kap...@huawei.com wrote: IMNSHO. deferring non-critical i/o from foreground process to background process is generally good. Yes, in regard of deferring you are right. However in this case may be when foreground process has to mark page dirty

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-15 Thread Jeff Davis
On Tue, 2012-11-06 at 17:55 -0600, Merlin Moncure wrote: So given that -- the patch simple adds an extra check when/where hint bit status is checked in the visibility routines (currently, only HeapTupleSatisfiesMVCC is done but all the applicable visibility routines should be done).

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-14 Thread Atri Sharma
On Wed, Nov 7, 2012 at 9:47 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the sig is a first cut at a patch (written by Atri) that attempts to mitigate hint bit i/o penalty when many pages worth of

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-14 Thread Atri Sharma
On Thu, Nov 15, 2012 at 12:42 AM, Atri Sharma atri.j...@gmail.com wrote: On Wed, Nov 7, 2012 at 9:47 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the sig is a first cut at a patch (written by Atri) that

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-07 Thread Amit Kapila
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Merlin Moncure Sent: Wednesday, November 07, 2012 5:26 AM To: PostgreSQL-development Cc: Atri Sharma Subject: [HACKERS] WIP patch for hint bit i/o mitigation Following the sig is a first cut

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-07 Thread Atri Sharma
On 07-Nov-2012, at 15:46, Amit Kapila amit.kap...@huawei.com wrote: From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Merlin Moncure Sent: Wednesday, November 07, 2012 5:26 AM To: PostgreSQL-development Cc: Atri Sharma Subject: [HACKERS] WIP

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-07 Thread Amit Kapila
-Original Message- From: Atri Sharma [mailto:atri.j...@gmail.com] Sent: Wednesday, November 07, 2012 4:02 PM To: Amit Kapila Cc: Merlin Moncure; PostgreSQL-development Subject: Re: [HACKERS] WIP patch for hint bit i/o mitigation On 07-Nov-2012, at 15:46, Amit Kapila amit.kap

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-07 Thread Atri Sharma
for hint bit i/o mitigation On 07-Nov-2012, at 15:46, Amit Kapila amit.kap...@huawei.com wrote: From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Merlin Moncure Sent: Wednesday, November 07, 2012 5:26 AM To: PostgreSQL-development

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-07 Thread Merlin Moncure
On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the sig is a first cut at a patch (written by Atri) that attempts to mitigate hint bit i/o penalty when many pages worth of tuples are sequentially written out with the same transaction id. There have been