Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-12 Thread Amit Kapila
On Wednesday, December 12, 2012 5:23 AM Greg Smith wrote: > On 11/23/12 5:57 AM, Amit kapila wrote: > > Let us try to see by example: > > Total RAM - 22G > > Database size - 16G > >... > > Case -2 (Shared Buffers - 10G) > > a. Load all the files in OS buffers. In best case OS buffers can > contain1

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-11 Thread Greg Smith
On 11/23/12 5:57 AM, Amit kapila wrote: Let us try to see by example: Total RAM - 22G Database size - 16G ... Case -2 (Shared Buffers - 10G) a. Load all the files in OS buffers. In best case OS buffers can contain10-12G data as OS has 12G of memory available. b. Try to load all in Shared buffers

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-23 Thread Amit kapila
On Friday, November 23, 2012 11:15 AM Pavan Deolasee wrote: On Thu, Nov 22, 2012 at 2:05 PM, Amit Kapila wrote: >>>Sorry, I haven't followed this thread at all, but the numbers (43171 and >>>57920) in the last two runs of @mv-free-list for 32 clients look >>>aberrations, no ? I wonder if >>>

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-23 Thread Amit Kapila
>>Shouldn't that data be in the shared buffers if not the OS cache and hence approximately same IO will be required? >I don't think so as the data in OS cache or PG Shared buffers doesn't have any direct relation, >OS can flush its buffers based on its scheduler algorithm. >Let us try to se

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-22 Thread Pavan Deolasee
On Thu, Nov 22, 2012 at 2:05 PM, Amit Kapila wrote: > ** ** > > > ** ** > > >Sorry, I haven't followed this thread at all, but the numbers (43171 and > 57920) in the last two runs of @mv-free-list for 32 clients look > aberrations, no ? I wonder if *>*that's skewing the average. > > ** ** >

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-22 Thread Amit Kapila
From: Pavan Deolasee [mailto:pavan.deola...@gmail.com] Sent: Thursday, November 22, 2012 12:26 PM To: Amit kapila Cc: Jeff Janes; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management On Mon, Nov 19, 2012 at 8:52 PM, Amit

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-21 Thread Pavan Deolasee
On Mon, Nov 19, 2012 at 8:52 PM, Amit kapila wrote: > On Monday, November 19, 2012 5:53 AM Jeff Janes wrote: > On Sun, Oct 21, 2012 at 12:59 AM, Amit kapila > wrote: > > On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: > > > >>Run the modes in reciprocating order? > >> Sorry, I didn't un

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-19 Thread Amit kapila
On Monday, November 19, 2012 6:05 AM Jeff Janes wrote: On Mon, Oct 22, 2012 at 10:51 AM, Amit kapila wrote: > >> Today again I have again collected the data for configuration Shared_buffers >> = 7G along with vmstat. >> The data and vmstat information (bi) are attached with this mail. It is >>

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
On Mon, Oct 22, 2012 at 10:51 AM, Amit kapila wrote: > > Today again I have again collected the data for configuration Shared_buffers > = 7G along with vmstat. > The data and vmstat information (bi) are attached with this mail. It is > observed from vmstat info that I/O is happening for both ca

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
On Sun, Oct 21, 2012 at 12:59 AM, Amit kapila wrote: > On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: > >>Run the modes in reciprocating order? > Sorry, I didn't understood this, What do you mean by modes in reciprocating > order? Sorry for the long delay. In your scripts, it looks li

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-22 Thread Amit Kapila
On Saturday, October 20, 2012 11:07 PM Jeff Janes wrote: > On Fri, Oct 19, 2012 at 11:00 PM, Amit kapila > wrote: > > > >> Robert wrote an accounting patch a while ago that tallied how often a > >> buffer was cleaned but then reclaimed for the same page before being > >> evicted. But now I can't

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-21 Thread Amit kapila
On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: On Fri, Sep 7, 2012 at 6:14 AM, Amit kapila wrote: > On Thursday, September 06, 2012 2:38 PM Amit kapila wrote: > On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: > On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: > On Mon,

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-20 Thread Jeff Janes
On Fri, Oct 19, 2012 at 11:00 PM, Amit kapila wrote: > >> Robert wrote an accounting patch a while ago that tallied how often a >> buffer was cleaned but then reclaimed for the same page before being >> evicted. But now I can't find it. If you can find that thread, there >> might be some benchma

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-20 Thread Jeff Janes
On Fri, Sep 7, 2012 at 6:14 AM, Amit kapila wrote: > On Thursday, September 06, 2012 2:38 PM Amit kapila wrote: > On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: > On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: > On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: This pa

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-19 Thread Amit kapila
On Friday, October 19, 2012 9:15 PM Jeff Janes wrote: On Tue, Sep 4, 2012 at 6:25 AM, Amit kapila wrote: > On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: > On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: >>> This patch is based on below Todo Item: >> >>> Consider adding buffers the

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-10-19 Thread Jeff Janes
On Tue, Sep 4, 2012 at 6:25 AM, Amit kapila wrote: > On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: > On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: >>> This patch is based on below Todo Item: >> >>> Consider adding buffers the background writer finds reusable to the free >>> list

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-06 Thread Amit kapila
On Tuesday, September 04, 2012 6:55 PM Amit kapila wrote: On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: >>> This patch is based on below Todo Item: > >>> Consider adding buffers the background writer finds reusable to the free >>> lis

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-04 Thread Amit kapila
On Tuesday, September 04, 2012 12:42 AM Jeff Janes wrote: On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: >> This patch is based on below Todo Item: > >> Consider adding buffers the background writer finds reusable to the free >> list > > > >> I have tried implementing it and taken the readings

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-03 Thread Jeff Janes
On Mon, Sep 3, 2012 at 7:15 AM, Amit kapila wrote: > This patch is based on below Todo Item: > > Consider adding buffers the background writer finds reusable to the free > list > > > > I have tried implementing it and taken the readings for Select when all the > data is in either OS buffers > > or

[HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-09-03 Thread Amit kapila
This patch is based on below Todo Item: Consider adding buffers the background writer finds reusable to the free list I have tried implementing it and taken the readings for Select when all the data is in either OS buffers or Shared Buffers. The Patch has simple implementation for "bgwrit