Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-08-21 Thread Robert Haas
On Mon, Aug 21, 2017 at 2:42 AM, Mithun Cy wrote: > Thanks for the patch, I have tested the above fix now it works as > described. From my test patch looks good, I did not find any other > issues. Considering the totality of the circumstances, it seemed appropriate to

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-08-21 Thread Mithun Cy
On Fri, Aug 18, 2017 at 9:43 PM, Robert Haas wrote: > On Fri, Aug 18, 2017 at 2:23 AM, Mithun Cy wrote: > Ah, good catch. While I agree that there is probably no great harm > from skipping the lock here, I think it would be better to just avoid

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-08-18 Thread Robert Haas
On Fri, Aug 18, 2017 at 2:23 AM, Mithun Cy wrote: > 1. Hang in apw_detach_shmem. > +/* > + * Clear our PID from autoprewarm shared state. > + */ > +static void > +apw_detach_shmem(int code, Datum arg) > +{ > + LWLockAcquire(_state->lock, LW_EXCLUSIVE); > + if

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-08-18 Thread Mithun Cy
On Wed, Aug 16, 2017 at 2:08 AM, Robert Haas wrote: > On Fri, Jul 14, 2017 at 8:17 AM, Mithun Cy wrote: >> [ new patch ] > It's quite possible that in making all of these changes I've > introduced some bugs, so I think this needs some testing

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-08-15 Thread Robert Haas
On Fri, Jul 14, 2017 at 8:17 AM, Mithun Cy wrote: > [ new patch ] I spent some time going over this patch. I initially thought it only needed minor cosmetic tweaking but the more I poked at it the more things I found that seemed like they should be changed, so the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-14 Thread Mithun Cy
On Thu, Jul 6, 2017 at 10:52 AM, Amit Kapila wrote: >>> 3. >> -- I do not think that is true pages of the unlogged table are also >> read into buffers for read-only purpose. So if we miss to dump them >> while we shut down then the previous dump should be used. >> > > I

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-06 Thread Mithun Cy
On Thu, Jul 6, 2017 at 10:52 AM, Amit Kapila wrote: > I am not able to understand what you want to say. Unlogged tables > should be empty in case of crash recovery. Also, we never flush > unlogged buffers except for shutdown checkpoint, refer BufferAlloc and > in

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-06 Thread Amit Kapila
On Wed, Jul 5, 2017 at 6:25 PM, Mithun Cy wrote: > On Mon, Jul 3, 2017 at 11:58 AM, Amit Kapila wrote: >> On Sun, Jul 2, 2017 at 10:32 PM, Mithun Cy >> wrote: >>> On Tue, Jun 27, 2017 at 11:41 AM, Mithun Cy

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-05 Thread Amit Kapila
On Wed, Jul 5, 2017 at 6:25 PM, Mithun Cy wrote: > On Mon, Jul 3, 2017 at 3:34 PM, Amit Kapila wrote: >> >> Few comments on the latest patch: >> >> 1. >> + LWLockRelease(_state->lock); >> + if (!is_bgworker) >> + ereport(ERROR, >> +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-05 Thread Mithun Cy
On Mon, Jul 3, 2017 at 3:34 PM, Amit Kapila wrote: > > Few comments on the latest patch: > > 1. > + LWLockRelease(_state->lock); > + if (!is_bgworker) > + ereport(ERROR, > + (errmsg("could not perform block dump because dump file is being > used by PID %d", > +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-05 Thread Mithun Cy
On Mon, Jul 3, 2017 at 11:58 AM, Amit Kapila wrote: > On Sun, Jul 2, 2017 at 10:32 PM, Mithun Cy wrote: >> On Tue, Jun 27, 2017 at 11:41 AM, Mithun Cy >> wrote: >>> On Fri, Jun 23, 2017 at 5:45 AM, Thom Brown

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-05 Thread Mithun Cy
On Mon, Jul 3, 2017 at 3:55 PM, Amit Kapila wrote: > On Fri, Jun 23, 2017 at 3:22 AM, Robert Haas wrote: >> On Thu, Jun 15, 2017 at 12:35 AM, Mithun Cy >> wrote: >> >> * Instead of creating our own buffering system via

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-03 Thread Amit Kapila
On Fri, Jun 23, 2017 at 3:22 AM, Robert Haas wrote: > On Thu, Jun 15, 2017 at 12:35 AM, Mithun Cy > wrote: > > * Instead of creating our own buffering system via buffer_file_write() > and buffer_file_flush(), why not just use the facilities

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-03 Thread Amit Kapila
On Sun, Jul 2, 2017 at 10:32 PM, Mithun Cy wrote: > On Tue, Jun 27, 2017 at 11:41 AM, Mithun Cy > wrote: >> On Fri, Jun 23, 2017 at 5:45 AM, Thom Brown wrote: >>> >>> Also, I find it a bit messy that

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-03 Thread Amit Kapila
On Sun, Jul 2, 2017 at 10:32 PM, Mithun Cy wrote: > On Tue, Jun 27, 2017 at 11:41 AM, Mithun Cy > wrote: >> On Fri, Jun 23, 2017 at 5:45 AM, Thom Brown wrote: >>> >>> Also, I find it a bit messy that

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-02 Thread Mithun Cy
On Tue, Jun 27, 2017 at 11:41 AM, Mithun Cy wrote: > On Fri, Jun 23, 2017 at 5:45 AM, Thom Brown wrote: >> >> Also, I find it a bit messy that launch_autoprewarm_dump() doesn't >> detect an autoprewarm process already running. I'd want this to >>

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-27 Thread Mithun Cy
On Fri, Jun 23, 2017 at 5:45 AM, Thom Brown wrote: > > I also think pg_prewarm.dump_interval should be renamed to > pg_prewarm.autoprewarm_interval. Thanks, I have changed it to pg_prewarm.autoprewarm_interval. >> >> * In the documentation, don't say "This is a SQL callable

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-27 Thread Mithun Cy
Thanks, Robert, I have tried to fix all of you comments and merged to fixes suggested by Thom in patch 15. On Fri, Jun 23, 2017 at 3:22 AM, Robert Haas wrote: > > * I suggest renaming launch_autoprewarm_dump() to > autoprewarm_start_worker(). I think that will be clearer.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-22 Thread Thom Brown
On 22 June 2017 at 22:52, Robert Haas wrote: > On Thu, Jun 15, 2017 at 12:35 AM, Mithun Cy > wrote: >> [ new patch ] > > I think this is looking better. I have some suggestions: > > * I suggest renaming launch_autoprewarm_dump() to >

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-22 Thread Robert Haas
On Thu, Jun 15, 2017 at 12:35 AM, Mithun Cy wrote: > [ new patch ] I think this is looking better. I have some suggestions: * I suggest renaming launch_autoprewarm_dump() to autoprewarm_start_worker(). I think that will be clearer. Remember that user-visible

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-14 Thread Mithun Cy
On Mon, Jun 12, 2017 at 7:34 PM, Amit Kapila wrote: > On Mon, Jun 12, 2017 at 6:31 AM, Mithun Cy > wrote: > > Thanks, Amit, > > > > + /* Perform autoprewarm's task. */ > + if (todo_task == TASK_PREWARM_BUFFERPOOL && > +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-12 Thread Amit Kapila
On Mon, Jun 12, 2017 at 6:31 AM, Mithun Cy wrote: > Thanks, Amit, > + /* Perform autoprewarm's task. */ + if (todo_task == TASK_PREWARM_BUFFERPOOL && + !state->skip_prewarm_on_restart) Why have you removed above comment in the new patch? I am not pointing this

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-11 Thread Mithun Cy
Thanks, Amit, On Fri, Jun 9, 2017 at 8:07 PM, Amit Kapila wrote: > > Few comments on the latest patch: > + > + /* Prewarm buffer. */ > + buf = ReadBufferExtended(rel, blk->forknum, blk->blocknum, RBM_NORMAL, > + NULL); > + if (BufferIsValid(buf)) > + ReleaseBuffer(buf);

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-09 Thread Amit Kapila
On Fri, Jun 9, 2017 at 10:01 AM, Mithun Cy wrote: > I have merged Rafia's patch for cosmetic changes. I have also fixed > some of the changes you have recommended over that. But kept few as it > is since Rafia's opinion was needed on that. > Few comments on the latest

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-08 Thread Mithun Cy
I have merged Rafia's patch for cosmetic changes. I have also fixed some of the changes you have recommended over that. But kept few as it is since Rafia's opinion was needed on that. On Wed, Jun 7, 2017 at 5:57 PM, Robert Haas wrote: > My experience is the opposite. If I

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-08 Thread Mithun Cy
On Tue, Jun 6, 2017 at 3:48 PM, Neha Sharma wrote: > Hi, > > I have been testing this feature for a while and below are the observations > for few scenarios. > > Observation: > scenario 1: If we set pg_prewarm.dump_interval = -1.0,we get an additional > warning

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-07 Thread Robert Haas
On Tue, Jun 6, 2017 at 6:29 AM, Rafia Sabih wrote: > On Mon, Jun 5, 2017 at 8:06 PM, Robert Haas wrote: >> Many of these seem worse, like these ones: >> >> - * Quit if we've reached records for another database. Unless the >> +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-06 Thread Rafia Sabih
On Mon, Jun 5, 2017 at 8:06 PM, Robert Haas wrote: > > Many of these seem worse, like these ones: > > - * Quit if we've reached records for another database. Unless the > + * Quit if we've reached records of another database. Unless the > Why is it worse?

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-06 Thread Neha Sharma
Hi, I have been testing this feature for a while and below are the observations for few scenarios. *Observation:* scenario 1: If we set pg_prewarm.dump_interval = -1.0,we get an additional warning message in logfile and instead of ending the task of auto-dump it executes successfully.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-05 Thread Robert Haas
On Mon, Jun 5, 2017 at 7:58 AM, Rafia Sabih wrote: > I had a look at the patch from stylistic/formatting point of view, > please find the attached patch for the suggested modifications. Many of these seem worse, like these ones: - * Quit if we've reached

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-05 Thread Rafia Sabih
On Sun, Jun 4, 2017 at 12:45 PM, Mithun Cy wrote: > On Wed, May 31, 2017 at 10:18 PM, Robert Haas wrote: >> + *contrib/autoprewarm.c >> >> Wrong. > > -- Oops Sorry fixed. > >> +Oiddatabase;/* database */ >> +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-06-04 Thread Mithun Cy
On Wed, May 31, 2017 at 10:18 PM, Robert Haas wrote: > + *contrib/autoprewarm.c > > Wrong. -- Oops Sorry fixed. > +Oiddatabase;/* database */ > +Oidspcnode;/* tablespace */ > +Oidfilenode;/*

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-31 Thread Robert Haas
On Tue, May 30, 2017 at 8:15 AM, Mithun Cy wrote: > On Tue, May 30, 2017 at 10:16 AM, Mithun Cy > wrote: >> Thanks Robert, > > Sorry, there was one more mistake ( a typo) in dump_now() instead of > using pfree I used free corrected same in

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-30 Thread Amit Kapila
On Tue, May 30, 2017 at 12:36 PM, Konstantin Knizhnik wrote: > On 27.10.2016 14:39, Mithun Cy wrote: > > > I wonder if you considered parallel prewarming of a table? > Right now either with pg_prewarm, either with pg_autoprewarm, preloading > table's data is performed

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-30 Thread Mithun Cy
On Tue, May 30, 2017 at 12:36 PM, Konstantin Knizhnik wrote: > On 27.10.2016 14:39, Mithun Cy wrote: > And as far as I understand pg_autoprewarm has all necessary infrastructure > to do parallel load. We just need to spawn more than one background worker > and specify >

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-30 Thread Mithun Cy
On Tue, May 30, 2017 at 10:16 AM, Mithun Cy wrote: > Thanks Robert, Sorry, there was one more mistake ( a typo) in dump_now() instead of using pfree I used free corrected same in the new patch v10. -- Thanks and Regards Mithun C Y EnterpriseDB:

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-30 Thread Konstantin Knizhnik
On 27.10.2016 14:39, Mithun Cy wrote: # pg_autoprewarm. This a PostgreSQL contrib module which automatically dump all of the blocknums present in buffer pool at the time of server shutdown(smart and fast mode only, to be enhanced to dump at regular interval.) and load these blocks when

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-29 Thread Mithun Cy
Thanks Robert, On Wed, May 24, 2017 at 5:41 PM, Robert Haas wrote: > + * > + * Once the "autoprewarm" bgworker has completed its prewarm task, it will > + * start a new task to periodically dump the BlockInfoRecords related to > blocks > + * which are currently in shared

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-24 Thread Robert Haas
On Wed, May 24, 2017 at 6:28 AM, Mithun Cy wrote: > On Tue, May 23, 2017 at 7:06 PM, Mithun Cy wrote: >> Thanks, Andres, >> >> I have tried to fix all of your comments. > > There was a typo issue in previous patch 07 where instead of == I

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-24 Thread Mithun Cy
On Tue, May 23, 2017 at 7:06 PM, Mithun Cy wrote: > Thanks, Andres, > > I have tried to fix all of your comments. There was a typo issue in previous patch 07 where instead of == I have used !=. And, a mistake in comments. I have corrected same now. -- Thanks and

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-05-23 Thread Mithun Cy
Thanks, Andres, I have tried to fix all of your comments. One important change has happened with this patch is previously we used to read one block info structure at a time and load it. But now we read all of them together and load it into as DSA and then we distribute the block infos to the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-04-07 Thread Mithun Cy
On Thu, Apr 6, 2017 at 4:12 AM, Andres Freund wrote: > On 2017-03-13 18:45:00 +0530, Mithun Cy wrote: >> I have implemented a similar logic now. The prewarm bgworker will >> launch a sub-worker per database in the dump file. And, each >> sub-worker will load its database block

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-04-05 Thread Andres Freund
On 2017-03-13 18:45:00 +0530, Mithun Cy wrote: > I have implemented a similar logic now. The prewarm bgworker will > launch a sub-worker per database in the dump file. And, each > sub-worker will load its database block info. The sub-workers will be > launched only after previous one is finished.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-25 Thread Mithun Cy
On Sat, Mar 25, 2017 at 11:46 PM, Peter Eisentraut wrote: > On 3/13/17 09:15, Mithun Cy wrote: >> A. launch_autoprewarm_dump() RETURNS int4 >> This is a SQL callable function to launch the autoprewarm worker to >> dump the buffer pool information at regular

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-25 Thread Peter Eisentraut
On 3/13/17 09:15, Mithun Cy wrote: > A. launch_autoprewarm_dump() RETURNS int4 > This is a SQL callable function to launch the autoprewarm worker to > dump the buffer pool information at regular interval. In a server, we > can only run one autoprewarm worker so if a worker sees another > existing

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-13 Thread Mithun Cy
>> - The error handling loop around load_block() suggests that you're >> expecting some reads to fail, which I guess is because you could be >> trying to read blocks from a relation that's been rewritten under a >> different relfilenode, or partially or entirely truncated. But I >> don't think

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-13 Thread Mithun Cy
Hi all, thanks for the feedback. Based on your recent comments I have implemented a new patch which is attached below, On Sun, Feb 26, 2017 at 10:16 PM, Robert Haas wrote: > This is not easy to fix. The lock has to be taken based on the > relation OID, not the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-03 Thread Robert Haas
On Thu, Mar 2, 2017 at 7:14 PM, Amit Kapila wrote: > So we should move this loading of blocks once the recovery reaches a > consistent state so that we can connect to a database. To allow > worker, to take a lock, we need to dump relation oid as well. Is that > what you

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-03-02 Thread Amit Kapila
On Sun, Feb 26, 2017 at 10:16 PM, Robert Haas wrote: > On Wed, Feb 22, 2017 at 11:49 PM, Mithun Cy > wrote: >> Hi all thanks, >> I have tried to fix all of the comments given above with some more >> code cleanups. > > While reading this patch

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-27 Thread Robert Haas
On Mon, Feb 27, 2017 at 7:18 PM, Peter Eisentraut wrote: > On 2/26/17 11:46, Robert Haas wrote: >> I don't see >> a solution other than launching a separate worker for each database, >> which seems like it could be extremely expensive if there are many >>

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-27 Thread Peter Eisentraut
On 2/26/17 11:46, Robert Haas wrote: > I don't see > a solution other than launching a separate worker for each database, > which seems like it could be extremely expensive if there are many > databases. You don't have to start all these workers at once. Starting one and not starting the next

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-26 Thread Robert Haas
On Wed, Feb 22, 2017 at 11:49 PM, Mithun Cy wrote: > Hi all thanks, > I have tried to fix all of the comments given above with some more > code cleanups. While reading this patch tonight, I realized a serious problem with the entire approach, which is that this patch

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-22 Thread Mithun Cy
Hi all thanks, I have tried to fix all of the comments given above with some more code cleanups. On Wed, Feb 22, 2017 at 6:28 AM, Robert Haas wrote: > I think it's OK to treat that as something of a corner case. There's > nothing to keep you from doing that today: just

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-21 Thread Robert Haas
On Wed, Feb 22, 2017 at 4:06 AM, Peter Eisentraut wrote: > On 2/10/17 15:12, Robert Haas wrote: >> Right. I can't see why you'd want to be able to separately control >> those two things. If you're not dumping, you don't want to load; if >> you're not loading,

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-21 Thread Peter Eisentraut
On 2/10/17 15:12, Robert Haas wrote: > Right. I can't see why you'd want to be able to separately control > those two things. If you're not dumping, you don't want to load; if > you're not loading, you don't want to dump. What about the case where you want to prewarm a standby from the info

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-10 Thread Robert Haas
On Thu, Feb 9, 2017 at 1:47 AM, Beena Emerson wrote: >> Won't it be simple if you consider -1 as a value to just load library? >> For *_interval = -1, it will neither load nor dump. >> > +1 > That is what I thought was the behaviour we decided upon for -1. Right. I

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-10 Thread Robert Haas
On Tue, Feb 7, 2017 at 2:04 AM, Amit Kapila wrote: > On Tue, Feb 7, 2017 at 10:44 AM, Mithun Cy wrote: >> >> == >> One problem now I have kept it open is multiple "auto pg_prewarm dump" >> can be launched even if already a

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-09 Thread Amit Kapila
On Thu, Feb 9, 2017 at 12:36 PM, Peter Geoghegan wrote: > On Tue, Feb 7, 2017 at 1:31 AM, Mithun Cy wrote: >> SEGFAULT was the coding mistake I have called the C-language function >> directly without initializing the functioncallinfo. Thanks for >>

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-08 Thread Peter Geoghegan
On Tue, Feb 7, 2017 at 1:31 AM, Mithun Cy wrote: > SEGFAULT was the coding mistake I have called the C-language function > directly without initializing the functioncallinfo. Thanks for > raising. Below patch fixes same. It would be nice if this had an option to

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-08 Thread Beena Emerson
Hello, On Wed, Feb 8, 2017 at 3:40 PM, Amit Kapila wrote: > On Tue, Feb 7, 2017 at 5:14 PM, Mithun Cy > wrote: > > On Tue, Feb 7, 2017 at 12:24 PM, Amit Kapila > wrote: > >> On Tue, Feb 7, 2017 at 11:53 AM, Beena

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-08 Thread Amit Kapila
On Tue, Feb 7, 2017 at 5:14 PM, Mithun Cy wrote: > On Tue, Feb 7, 2017 at 12:24 PM, Amit Kapila wrote: >> On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson >> wrote: >>> Are 2 workers required? >>> >> >> I think in the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Mithun Cy
On Tue, Feb 7, 2017 at 6:11 PM, Beena Emerson wrote: > Yes it would be better to have only one pg_prewarm worker as the loader is > idle for the entire server run time after the initial load activity of few > secs. Sorry, that is again a bug in the code. The code to

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Beena Emerson
Hello, On Tue, Feb 7, 2017 at 5:14 PM, Mithun Cy wrote: > On Tue, Feb 7, 2017 at 12:24 PM, Amit Kapila > wrote: > > On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson > wrote: > >> Are 2 workers required? > >> > > > > I

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Beena Emerson
Hello, On Tue, Feb 7, 2017 at 5:52 PM, Mithun Cy wrote: > Thanks Beena, > > On Tue, Feb 7, 2017 at 4:46 PM, Beena Emerson > wrote: > > Few more comments: > > > > = Background worker messages: > > > > - Workers when launched, show messages

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Mithun Cy
Thanks Beena, On Tue, Feb 7, 2017 at 4:46 PM, Beena Emerson wrote: > Few more comments: > > = Background worker messages: > > - Workers when launched, show messages like: "logical replication launcher > started”, "autovacuum launcher started”. We should probably have a

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Mithun Cy
On Tue, Feb 7, 2017 at 12:24 PM, Amit Kapila wrote: > On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson > wrote: >> Are 2 workers required? >> > > I think in the new design there is a provision of launching the worker > dynamically to dump the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Beena Emerson
On Tue, Feb 7, 2017 at 3:01 PM, Mithun Cy wrote: > On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson > wrote: > > launched by other applications. Also with max_worker_processes = 2 and > > restart, the system crashes when the 2nd worker is not

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-07 Thread Mithun Cy
On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson wrote: > launched by other applications. Also with max_worker_processes = 2 and > restart, the system crashes when the 2nd worker is not launched: > 2017-02-07 11:36:39.132 IST [20573] LOG: auto pg_prewarm load : number of

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Amit Kapila
On Tue, Feb 7, 2017 at 10:44 AM, Mithun Cy wrote: > > == > One problem now I have kept it open is multiple "auto pg_prewarm dump" > can be launched even if already a dump/load worker is running by > calling launch_pg_prewarm_dump. I can avoid this by

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Amit Kapila
On Tue, Feb 7, 2017 at 11:53 AM, Beena Emerson wrote: > Hello, > > Thank you for the updated patch. > > On Tue, Feb 7, 2017 at 10:44 AM, Mithun Cy > wrote: >> >> Hi all, >> Here is the new patch which fixes all of above comments, I changed the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Beena Emerson
Hello, Thank you for the updated patch. On Tue, Feb 7, 2017 at 10:44 AM, Mithun Cy wrote: > Hi all, > Here is the new patch which fixes all of above comments, I changed the > design a bit now as below > > What is it? > === > A pair of bgwrokers one which

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-02-06 Thread Mithun Cy
Hi all, Here is the new patch which fixes all of above comments, I changed the design a bit now as below What is it? === A pair of bgwrokers one which automatically dumps buffer pool's block info at a given interval and another which loads those block into buffer pool when the server

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-31 Thread Michael Paquier
On Wed, Feb 1, 2017 at 1:17 AM, Robert Haas wrote: > Well, the question even for that case is whether it really costs > anything. My bet is that it is nearly free when it doesn't help, but > that could be wrong. My experience running pgbench tests is that > prewarming all

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-31 Thread Mithun Cy
On Tue, Jan 31, 2017 at 9:47 PM, Robert Haas wrote: > Now, I assume that this patch sorts the I/O (although I haven't > checked that) and therefore I expect that the prewarm completes really > fast. If that's not the case, then that's bad. But if it is the > case, then

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-31 Thread Robert Haas
On Tue, Jan 31, 2017 at 1:48 AM, Michael Paquier wrote: > I partially agree with this paragraph, at least there are advantages > to do so for cases where the data fits in shared buffers. Even for > data sets fitting in RAM that can be an advantage as the buffers would >

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-30 Thread Michael Paquier
On Tue, Jan 31, 2017 at 3:02 AM, Robert Haas wrote: > On Fri, Jan 27, 2017 at 5:34 PM, Jim Nasby wrote: >> On 1/26/17 10:11 PM, Beena Emerson wrote: >>> In that case, we could add the file location parameter. By default it >>> would store in the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-30 Thread Robert Haas
On Fri, Jan 27, 2017 at 5:34 PM, Jim Nasby wrote: > On 1/26/17 10:11 PM, Beena Emerson wrote: >> In that case, we could add the file location parameter. By default it >> would store in the cluster directory else in the location provided. We >> can update this parameter

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-27 Thread Jim Nasby
On 1/26/17 10:11 PM, Beena Emerson wrote: In that case, we could add the file location parameter. By default it would store in the cluster directory else in the location provided. We can update this parameter in standby for it to access the file. I don't see file location being as useful in

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-27 Thread Robert Haas
On Fri, Jan 27, 2017 at 3:18 PM, Peter Eisentraut wrote: > On 1/26/17 11:11 PM, Beena Emerson wrote: >> In that case, we could add the file location parameter. By default it >> would store in the cluster directory else in the location provided. We >> can update

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-27 Thread Peter Eisentraut
On 1/26/17 11:11 PM, Beena Emerson wrote: > In that case, we could add the file location parameter. By default it > would store in the cluster directory else in the location provided. We > can update this parameter in standby for it to access the file. I don't see the need for that. -- Peter

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-26 Thread Mithun Cy
On Thu, Jan 26, 2017 at 8:45 PM, Peter Eisentraut wrote: > Just a thought with an additional use case: If I want to set up a > standby for offloading queries, could I take the dump file from the > primary or another existing standby, copy it to the new standby,

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-26 Thread Beena Emerson
On Fri, Jan 27, 2017 at 8:14 AM, Amit Kapila wrote: > On Thu, Jan 26, 2017 at 8:45 PM, Peter Eisentraut > wrote: > > On 1/24/17 3:26 AM, Mithun Cy wrote: > >> In my code by default, we only dump at shutdown time. If we want to > >> dump

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-26 Thread Amit Kapila
On Thu, Jan 26, 2017 at 8:45 PM, Peter Eisentraut wrote: > On 1/24/17 3:26 AM, Mithun Cy wrote: >> In my code by default, we only dump at shutdown time. If we want to >> dump at regular interval then we need to set the GUC >> pg_autoprewarm.buff_dump_interval to

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-26 Thread Peter Eisentraut
On 1/24/17 3:26 AM, Mithun Cy wrote: > In my code by default, we only dump at shutdown time. If we want to > dump at regular interval then we need to set the GUC > pg_autoprewarm.buff_dump_interval to > 0. Just a thought with an additional use case: If I want to set up a standby for offloading

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:46 PM, Jim Nasby wrote: > I think the two need to be integrated much better than they are right now. > They should certainly be in the same .so, and as others have mentioned the > docs need to be fixed. For consistency, I think the name should

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/25/17 1:46 PM, Jim Nasby wrote: Based on that and other feedback I'm going to mark this as returned with feedback, though if you're able to get a revised patch in the next few days please do. Actually, based on the message that popped up when I went to do that I guess it's better not to

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/24/17 11:13 PM, Beena Emerson wrote: On Wed, Jan 25, 2017 at 10:36 AM, Jim Nasby > wrote: On 1/24/17 2:26 AM, Mithun Cy wrote: Thanks for looking into this patch, I just downloaded the patch and applied same

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Robert Haas
On Mon, Jan 23, 2017 at 6:37 PM, Jim Nasby wrote: > I'm not sure the default GUC setting of 0 makes sense. If you've loaded the > module, presumably you want it to be running. I think it'd be nice if the GUC > had a -1 setting that meant to use checkpoint_timeout.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Amit Kapila
On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby wrote: > I took a look at this again, and it doesn't appear to be working for me. The > library is being loaded during startup, but I don't see any further activity > in the log, and I don't see an autoprewarm file in $PGDATA.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Beena Emerson
On Wed, Jan 25, 2017 at 10:36 AM, Jim Nasby wrote: > On 1/24/17 2:26 AM, Mithun Cy wrote: > >> Thanks for looking into this patch, I just downloaded the patch and >> applied same to the latest code, I can see file " autoprewarm.save" in >> $PGDATA which is created and

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Jim Nasby
On 1/24/17 2:26 AM, Mithun Cy wrote: Thanks for looking into this patch, I just downloaded the patch and applied same to the latest code, I can see file " autoprewarm.save" in $PGDATA which is created and dumped at shutdown time and an activity is logged as below 2017-01-24 13:22:25.012 IST

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Jim Nasby
On 1/24/17 4:56 AM, Beena Emerson wrote: 2. buff_dump_interval could be renamed to just dump_interval or buffer_dump_interval. Also, since it is now part of pg_prewarm. I think it makes sense to have the conf parameter be: pg_prewarm.xxx instead of pg_autoprewarm.xxx I'd really like to find

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Mithun Cy
On Fri, Oct 28, 2016 at 6:36 AM, Tsunakawa, Takayuki wrote: > I welcome this feature! I remember pg_hibernate did this. I wonder what > happened to pg_hibernate. Did you check it? Thanks, when I checked with pg_hibernate there were two things people complained

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Beena Emerson
On Tue, Jan 24, 2017 at 1:56 PM, Mithun Cy wrote: > On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby > wrote: > > I took a look at this again, and it doesn't appear to be working for me. > The library is being loaded during startup, but I don't

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Mithun Cy
On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby wrote: > I took a look at this again, and it doesn't appear to be working for me. The > library is being loaded during startup, but I don't see any further activity > in the log, and I don't see an autoprewarm file in $PGDATA.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-23 Thread Jim Nasby
I took a look at this again, and it doesn't appear to be working for me. The library is being loaded during startup, but I don't see any further activity in the log, and I don't see an autoprewarm file in $PGDATA. There needs to be some kind of documentation change as part of this patch. I'm

Re: [HACKERS] Proposal : For Auto-Prewarm.

2016-12-02 Thread Haribabu Kommi
On Tue, Nov 29, 2016 at 4:26 PM, Mithun Cy wrote: > Sorry I took some time on this please find latest patch with addressed > review comments. Apart from fixes for comments I have introduced a new GUC > variable for the pg_autoprewarm "buff_dump_interval". So now we

Re: [HACKERS] Proposal : For Auto-Prewarm.

2016-11-28 Thread Mithun Cy
Sorry I took some time on this please find latest patch with addressed review comments. Apart from fixes for comments I have introduced a new GUC variable for the pg_autoprewarm "buff_dump_interval". So now we dump the buffer pool metadata at every buff_dump_interval secs. Currently

Re: [HACKERS] Proposal : For Auto-Prewarm.

2016-11-22 Thread Ashutosh Bapat
> you assigned as reviewer to the current patch in the 11-2016 commitfest. > But you haven't shared your review yet. Please share your views about > the patch. This will help us in smoother operation of commitfest. > Thanks for the reminder. Mithun has not provided a patch addressing the

Re: [HACKERS] Proposal : For Auto-Prewarm.

2016-11-21 Thread Haribabu Kommi
Hi Ashutosh, This is a gentle reminder. you assigned as reviewer to the current patch in the 11-2016 commitfest. But you haven't shared your review yet. Please share your views about the patch. This will help us in smoother operation of commitfest. Please Ignore if you already shared your

  1   2   >