Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-12 Thread Amit Kapila
On Wednesday, December 12, 2012 5:23 AM Greg Smith wrote: On 11/23/12 5:57 AM, Amit kapila wrote: Let us try to see by example: Total RAM - 22G Database size - 16G ... Case -2 (Shared Buffers - 10G) a. Load all the files in OS buffers. In best case OS buffers can contain10-12G data as

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-11 Thread Greg Smith
On 11/23/12 5:57 AM, Amit kapila wrote: Let us try to see by example: Total RAM - 22G Database size - 16G ... Case -2 (Shared Buffers - 10G) a. Load all the files in OS buffers. In best case OS buffers can contain10-12G data as OS has 12G of memory available. b. Try to load all in Shared

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-23 Thread Amit Kapila
Shouldn't that data be in the shared buffers if not the OS cache and hence approximately same IO will be required? I don't think so as the data in OS cache or PG Shared buffers doesn't have any direct relation, OS can flush its buffers based on its scheduler algorithm. Let us try to see by

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-23 Thread Amit kapila
On Friday, November 23, 2012 11:15 AM Pavan Deolasee wrote: On Thu, Nov 22, 2012 at 2:05 PM, Amit Kapila amit.kap...@huawei.com wrote: Sorry, I haven't followed this thread at all, but the numbers (43171 and 57920) in the last two runs of @mv-free-list for 32 clients look aberrations, no ? I

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-22 Thread Amit Kapila
From: Pavan Deolasee [mailto:pavan.deola...@gmail.com] Sent: Thursday, November 22, 2012 12:26 PM To: Amit kapila Cc: Jeff Janes; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management On Mon, Nov 19, 2012 at 8:52 PM, Amit

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-22 Thread Pavan Deolasee
On Thu, Nov 22, 2012 at 2:05 PM, Amit Kapila amit.kap...@huawei.com wrote: ** ** ** ** Sorry, I haven't followed this thread at all, but the numbers (43171 and 57920) in the last two runs of @mv-free-list for 32 clients look aberrations, no ? I wonder if **that's skewing the average.

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-21 Thread Pavan Deolasee
On Mon, Nov 19, 2012 at 8:52 PM, Amit kapila amit.kap...@huawei.com wrote: On Monday, November 19, 2012 5:53 AM Jeff Janes wrote: On Sun, Oct 21, 2012 at 12:59 AM, Amit kapila amit.kap...@huawei.com wrote: On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: Run the modes in

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-19 Thread Amit kapila
On Monday, November 19, 2012 6:05 AM Jeff Janes wrote: On Mon, Oct 22, 2012 at 10:51 AM, Amit kapila amit.kap...@huawei.com wrote: Today again I have again collected the data for configuration Shared_buffers = 7G along with vmstat. The data and vmstat information (bi) are attached with this

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
On Sun, Oct 21, 2012 at 12:59 AM, Amit kapila amit.kap...@huawei.com wrote: On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: Run the modes in reciprocating order? Sorry, I didn't understood this, What do you mean by modes in reciprocating order? Sorry for the long delay. In your

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
On Mon, Oct 22, 2012 at 10:51 AM, Amit kapila amit.kap...@huawei.com wrote: Today again I have again collected the data for configuration Shared_buffers = 7G along with vmstat. The data and vmstat information (bi) are attached with this mail. It is observed from vmstat info that I/O is

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-22 Thread Amit Kapila
On Saturday, October 20, 2012 11:07 PM Jeff Janes wrote: On Fri, Oct 19, 2012 at 11:00 PM, Amit kapila amit.kap...@huawei.com wrote: Robert wrote an accounting patch a while ago that tallied how often a buffer was cleaned but then reclaimed for the same page before being evicted. But

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-21 Thread Amit kapila
On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: On Fri, Sep 7, 2012 at 6:14 AM, Amit kapila amit.kap...@huawei.com wrote: On Thursday, September 06, 2012 2:38 PM Amit kapila wrote: On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: On Tuesday, September 04, 2012 12:42 AM Jeff

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-20 Thread Amit kapila
On Friday, October 19, 2012 9:15 PM Jeff Janes wrote: On Tue, Sep 4, 2012 at 6:25 AM, Amit kapila amit.kap...@huawei.com wrote: On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila amit.kap...@huawei.com wrote: This patch is based on below Todo

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-20 Thread Jeff Janes
On Fri, Sep 7, 2012 at 6:14 AM, Amit kapila amit.kap...@huawei.com wrote: On Thursday, September 06, 2012 2:38 PM Amit kapila wrote: On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-20 Thread Jeff Janes
On Fri, Oct 19, 2012 at 11:00 PM, Amit kapila amit.kap...@huawei.com wrote: Robert wrote an accounting patch a while ago that tallied how often a buffer was cleaned but then reclaimed for the same page before being evicted. But now I can't find it. If you can find that thread, there might

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-19 Thread Jeff Janes
On Tue, Sep 4, 2012 at 6:25 AM, Amit kapila amit.kap...@huawei.com wrote: On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila amit.kap...@huawei.com wrote: This patch is based on below Todo Item: Consider adding buffers the background writer

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-06 Thread Amit kapila
On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila amit.kap...@huawei.com wrote: This patch is based on below Todo Item: Consider adding buffers the background writer finds reusable to

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-04 Thread Amit kapila
On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila amit.kap...@huawei.com wrote: This patch is based on below Todo Item: Consider adding buffers the background writer finds reusable to the free list I have tried implementing it and taken

[HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-03 Thread Amit kapila
This patch is based on below Todo Item: Consider adding buffers the background writer finds reusable to the free list I have tried implementing it and taken the readings for Select when all the data is in either OS buffers or Shared Buffers. The Patch has simple implementation for

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-03 Thread Jeff Janes
On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila amit.kap...@huawei.com wrote: This patch is based on below Todo Item: Consider adding buffers the background writer finds reusable to the free list I have tried implementing it and taken the readings for Select when all the data is in either OS