Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

At Fri, 23 Dec 2016 11:02:11 +0900, Michael Paquier wrote in > On Fri, Dec 23, 2016 at 8:13 AM, Robert Haas wrote: > > On Thu, Dec 22, 2016 at 2:34 PM, Andres Freund

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Dec 23, 2016 at 8:13 AM, Robert Haas wrote: > On Thu, Dec 22, 2016 at 2:34 PM, Andres Freund wrote: >> On 2016-12-22 08:32:56 -0800, Andres Freund wrote: >>> I plan to commit this later today. Hope I got the reviewers roughly right. >> >> And

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, Dec 22, 2016 at 2:34 PM, Andres Freund wrote: > On 2016-12-22 08:32:56 -0800, Andres Freund wrote: >> I plan to commit this later today. Hope I got the reviewers roughly right. > > And pushed. Thanks for the work on this everyone. Cool. Also, +1 for the

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi, On 2016-12-22 08:32:56 -0800, Andres Freund wrote: > I plan to commit this later today. Hope I got the reviewers roughly right. And pushed. Thanks for the work on this everyone. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi, On 2016-12-21 13:28:54 -0800, Andres Freund wrote: > A mime-type of invalid/octet-stream? That's an, uh, odd choice. > > Working on committing this (tomorrow morning, not tonight). There's > some relatively minor things I want to change: > > - I don't like the name XLogSetFlags() - it's

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, Dec 22, 2016 at 3:10 AM, Andres Freund wrote: > On 2016-12-21 16:35:28 -0500, Robert Haas wrote: >> On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund wrote: >> > - Similarly I don't like the name "progress LSN" much. What does >> > "progress" really

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, Dec 22, 2016 at 6:41 AM, David Steele wrote: > On 12/21/16 4:28 PM, Andres Freund wrote: > >> Working on committing this (tomorrow morning, not tonight). There's >> some relatively minor things I want to change: Thanks for looking at this patch. >> - I don't like

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, Dec 22, 2016 at 6:28 AM, Andres Freund wrote: > A mime-type of invalid/octet-stream? That's an, uh, odd choice. Indeed. I am not sure what kind of accident happened here. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 12/21/16 4:40 PM, Andres Freund wrote: On 2016-12-21 16:35:28 -0500, Robert Haas wrote: What I think "progress LSN" is getting at -- actually fairly well -- is whether we're getting anything *important* done, not whether we are consistent. I don't mind changing the name, but not to

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Wed, Dec 21, 2016 at 2:40 PM, Andres Freund wrote: > That's imo pretty much what progress LSN currently describes. Have there > been any records which are important for durability/consistency and > hence need to be archived and such. > The above, to me, describes a

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi Andres, On 12/21/16 4:28 PM, Andres Freund wrote: Working on committing this (tomorrow morning, not tonight). There's some relatively minor things I want to change: - I don't like the name XLogSetFlags() - it's completely unclear what that those flags refer to - it could just as well be

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 2016-12-21 16:35:28 -0500, Robert Haas wrote: > On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund wrote: > > - Similarly I don't like the name "progress LSN" much. What does > > "progress" really mean in that". Maybe "consistency LSN"? > > Whoa. -1 from me for "consistency

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund wrote: > - Similarly I don't like the name "progress LSN" much. What does > "progress" really mean in that". Maybe "consistency LSN"? Whoa. -1 from me for "consistency LSN". Consistency has to with whether the cluster has

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi, A mime-type of invalid/octet-stream? That's an, uh, odd choice. Working on committing this (tomorrow morning, not tonight). There's some relatively minor things I want to change: - I don't like the name XLogSetFlags() - it's completely unclear what that those flags refer to - it could

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Dec 2, 2016 at 9:50 AM, Michael Paquier wrote: > On Wed, Nov 30, 2016 at 7:53 PM, Amit Kapila wrote: >>> + * Switch segment only when WAL has done some progress since the >> + * > last time a segment has switched because of a timeout.

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Wed, Nov 30, 2016 at 7:53 PM, Amit Kapila wrote: > On Mon, Nov 21, 2016 at 11:08 AM, Michael Paquier > wrote: >> On Tue, Nov 15, 2016 at 9:59 PM, Amit Kapila wrote: >>> On Tue, Nov 15, 2016 at 9:27 AM, Kyotaro

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 21, 2016 at 11:08 AM, Michael Paquier wrote: > On Tue, Nov 15, 2016 at 9:59 PM, Amit Kapila wrote: >> On Tue, Nov 15, 2016 at 9:27 AM, Kyotaro HORIGUCHI >> wrote: >>> The term "WAL activity' is used

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 22, 2016 at 6:27 PM, Kyotaro HORIGUCHI wrote: > I have marked this as ready for committer again. And moved to next CF for now. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

I almost forgot this. At Mon, 21 Nov 2016 15:44:08 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161121.154408.47398334.horiguchi.kyot...@lab.ntt.co.jp> > Hello, > > At Mon, 21 Nov 2016 14:41:27 +0900, Michael Paquier >

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hello, At Mon, 21 Nov 2016 14:41:27 +0900, Michael Paquier wrote in > On Mon, Nov 21, 2016 at 1:31 PM, Kyotaro HORIGUCHI > wrote: > > So, all my original concern

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 21, 2016 at 1:31 PM, Kyotaro HORIGUCHI wrote: > So, all my original concern were cleared. Cool. Perhaps this could be marked as ready for committer then? > The last one is > resetting by a checkpointer restart.. I'd like to remove that if > Andres

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 15, 2016 at 9:59 PM, Amit Kapila wrote: > On Tue, Nov 15, 2016 at 9:27 AM, Kyotaro HORIGUCHI > wrote: >> The term "WAL activity' is used in the comment for >> GetProgressRecPtr. Its meaning is not clear but not well >>

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Thank you very much for the testing on the nice machine. At Fri, 18 Nov 2016 20:35:43 -0800, Michael Paquier wrote in

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 11/18/16 12:38 PM, David Steele wrote: > On 11/14/16 4:29 AM, Michael Paquier wrote: >> On Mon, Nov 14, 2016 at 6:10 PM, Kyotaro HORIGUCHI >>> If I'm not missing something, at the worst we have a checkpoint >>> after a checkpointer restart that should have been supressed. Is >>> it worth

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Nov 18, 2016 at 7:00 PM, Amit Kapila wrote: > Okay, I have done some performance tests with this patch and found that it > doesn't have any noticeable impact which is good. Details of performance > tests is below: > Machine configuration: > 2 sockets, 28 cores

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 14, 2016 at 9:33 AM, Michael Paquier wrote: > On Mon, Nov 14, 2016 at 12:49 PM, Kyotaro HORIGUCHI > wrote: >> At Sat, 12 Nov 2016 10:28:56 +0530, Amit Kapila wrote in

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 11/14/16 4:29 AM, Michael Paquier wrote: > On Mon, Nov 14, 2016 at 6:10 PM, Kyotaro HORIGUCHI > wrote: >> It applies the master and compiled cleanly and no error by >> regtest. (I didn't confirmed that the problem is still fixed but >> seemingly no problem) >

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 15, 2016 at 9:27 AM, Kyotaro HORIGUCHI wrote: > Hello, > > At Mon, 14 Nov 2016 16:53:35 +0530, Amit Kapila > wrote in >> > >> >> The progress parameter is used not only for checkpoint activity but by >> bgwriter as well for

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hello, At Mon, 14 Nov 2016 16:53:35 +0530, Amit Kapila wrote in > On Mon, Nov 14, 2016 at 9:33 AM, Michael Paquier > >>> Is it right to set XLOG_NO_PROGRESS flag in LogAccessExclusiveLocks? > >>> This

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 14, 2016 at 9:33 AM, Michael Paquier wrote: > On Mon, Nov 14, 2016 at 12:49 PM, Kyotaro HORIGUCHI > wrote: >> At Sat, 12 Nov 2016 10:28:56 +0530, Amit Kapila >> wrote in

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 14, 2016 at 6:10 PM, Kyotaro HORIGUCHI wrote: > It applies the master and compiled cleanly and no error by > regtest. (I didn't confirmed that the problem is still fixed but > seemingly no problem) Thanks for double-checking. > If I'm not missing

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hello, It applies the master and compiled cleanly and no error by regtest. (I didn't confirmed that the problem is still fixed but seemingly no problem) At Mon, 14 Nov 2016 15:09:09 +0900, Michael Paquier wrote in

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Sat, Nov 12, 2016 at 9:01 PM, Andres Freund wrote: > On 2016-11-11 16:42:43 +0900, Michael Paquier wrote: > >> + * This takes also >> + * advantage to avoid 8-byte torn reads on some platforms by using the >> + * fact that each insert lock is located on the same cache line.

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Mon, Nov 14, 2016 at 12:49 PM, Kyotaro HORIGUCHI wrote: > At Sat, 12 Nov 2016 10:28:56 +0530, Amit Kapila > wrote in >> I think it is good to check the performance

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hello, thank you for the comment. At Sat, 12 Nov 2016 10:28:56 +0530, Amit Kapila wrote in > On Tue, Nov 8, 2016 at 5:18 PM, Kyotaro HORIGUCHI > wrote: > > Hello, > >

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi, On 2016-11-11 16:42:43 +0900, Michael Paquier wrote: > + * This takes also > + * advantage to avoid 8-byte torn reads on some platforms by using the > + * fact that each insert lock is located on the same cache line. Something residing on the same cache line doens't provide that guarantee

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 8, 2016 at 5:18 PM, Kyotaro HORIGUCHI wrote: > Hello, > >> on something else than LW_EXCLUSIVE compared to now. To keep things >> more simple I' would still favor using WALInsertLocks for this patch, >> that looks more consistent, and also because

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Thank you for the new patch. At Fri, 11 Nov 2016 16:42:43 +0900, Michael Paquier wrote in > On Fri, Nov 11, 2016 at 12:28 AM, Stephen Frost wrote: > > We should probably include

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Nov 11, 2016 at 12:28 AM, Stephen Frost wrote: > * Michael Paquier (michael.paqu...@gmail.com) wrote: >> Thanks for the review! Waiting for a couple of days more is fine for >> me. This won't change much. Attached is v15 with the fixes you >> mentioned. > > I figured

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 11/10/16 1:03 PM, Stephen Frost wrote: > On Thursday, November 10, 2016, Joshua D. Drake > wrote: > > On 11/10/2016 09:33 AM, David Steele wrote: > > On 11/10/16 10:28 AM, Stephen Frost wrote: > > diff --git

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thursday, November 10, 2016, Joshua D. Drake wrote: > On 11/10/2016 09:33 AM, David Steele wrote: > >> On 11/10/16 10:28 AM, Stephen Frost wrote: >> >> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c >>> [...] >>> +

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 11/10/2016 09:33 AM, David Steele wrote: On 11/10/16 10:28 AM, Stephen Frost wrote: diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c [...] + if (log_checkpoints) + ereport(LOG, (errmsg("checkpoint

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 11/10/16 10:28 AM, Stephen Frost wrote: >> diff --git a/src/backend/access/transam/xlog.c >> b/src/backend/access/transam/xlog.c > [...] >> +if (log_checkpoints) >> +ereport(LOG, (errmsg("checkpoint skipped"))); > > Do we really need to log

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > Thanks for the review! Waiting for a couple of days more is fine for > me. This won't change much. Attached is v15 with the fixes you > mentioned. I figured I'd go ahead and start looking into this (and it's pretty easy for me to

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 8, 2016 at 9:32 PM, David Steele wrote: > I had a bit of trouble parsing this paragraph: > > [...] > > So I did a little reworking: > > [...] > > If that still says what you think it should, then I believe it is clearer. Thanks! I have included your suggestion.

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 10/5/16 7:18 AM, Michael Paquier wrote: Note: I am moving this patch to next CF. And I am back on it more seriously... And I am taking back what I said upthread. I looked at the v12 that Horiguchi-san has written, and that seems correct to me. So I have squashed everything into a single

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hello, At Tue, 8 Nov 2016 14:45:59 +0900, Michael Paquier wrote in > On Tue, Nov 1, 2016 at 8:31 PM, Kyotaro HORIGUCHI > wrote: > > Could you let me struggle a bit

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Nov 1, 2016 at 8:31 PM, Kyotaro HORIGUCHI wrote: > Could you let me struggle a bit more to avoid LWLocks in > GetProgressRecPtr? Be my guest :) > I considered two alternatives for updating logic of progressAt > more seriously. One is, as Amit suggested,

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Thanks for merging. It still applies on the current master with some displacements. At Wed, 5 Oct 2016 15:18:53 +0900, Michael Paquier wrote in

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

(Squashing replies) On Fri, Sep 30, 2016 at 6:13 PM, Michael Paquier wrote: > On Fri, Sep 30, 2016 at 2:51 PM, Michael Paquier > wrote: >> On Fri, Sep 30, 2016 at 2:05 PM, Kyotaro HORIGUCHI >> wrote: >>> At

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Sep 30, 2016 at 2:51 PM, Michael Paquier wrote: > On Fri, Sep 30, 2016 at 2:05 PM, Kyotaro HORIGUCHI > wrote: >> At Fri, 30 Sep 2016 14:00:15 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI >>

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Fri, Sep 30, 2016 at 2:05 PM, Kyotaro HORIGUCHI wrote: > At Fri, 30 Sep 2016 14:00:15 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20160930.140015.150178454.horiguchi.kyot...@lab.ntt.co.jp> >> I don't see no

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Sorry, it wrote wrong thing. At Fri, 30 Sep 2016 14:00:15 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20160930.140015.150178454.horiguchi.kyot...@lab.ntt.co.jp> > Sorry, I might have torn off this thread somehow.. > > At Thu, 29 Sep 2016 11:26:29

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Sorry, I might have torn off this thread somehow.. At Thu, 29 Sep 2016 11:26:29 -0400, David Steele wrote in <30095aea-3910-dbb7-1790-a579fb93f...@pgmasters.net> > On 9/28/16 10:32 PM, Michael Paquier wrote: > > On Thu, Sep 29, 2016 at 7:45 AM, David Steele

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 9/28/16 10:32 PM, Michael Paquier wrote: On Thu, Sep 29, 2016 at 7:45 AM, David Steele wrote: In general I agree with the other comments that this could end up being a problem. On the other hand, since the additional locks are only taken at checkpoint or

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, Sep 29, 2016 at 7:45 AM, David Steele wrote: > OK, I've done functional testing and this patch seems to work as > specified (including the caveat noted above). Some comments: Thanks! > * [PATCH 1/3] hs-checkpoints-v12-1 > > +++ b/src/backend/access/transam/xlog.c >

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 9/28/16 3:35 AM, Michael Paquier wrote: > On Wed, Sep 28, 2016 at 6:12 AM, David Steele wrote: >> I tried the attached patch set and noticed an interesting behavior. With >> archive_timeout=5 whenever I made a change I would get a WAL segment within >> a few seconds as

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Wed, Sep 28, 2016 at 6:12 AM, David Steele wrote: > I tried the attached patch set and noticed an interesting behavior. With > archive_timeout=5 whenever I made a change I would get a WAL segment within > a few seconds as expected then another one would follow a few

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Tue, Sep 27, 2016 at 7:16 PM, Kyotaro HORIGUCHI wrote: > I apologize in advance that the comments in this message might > one of the ideas discarded in the past thread.. I might not grasp > the discussion completely X( No problem. > At Wed, 18 May 2016

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On 9/27/16 6:16 AM, Kyotaro HORIGUCHI wrote: I apologize in advance that the comments in this message might one of the ideas discarded in the past thread.. I might not grasp the discussion completely X( The attached patches are rebased to the master and additional one mentioned below. I

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi, I apologize in advance that the comments in this message might one of the ideas discarded in the past thread.. I might not grasp the discussion completely X( The attached patches are rebased to the master and additional one mentioned below. At Wed, 18 May 2016 17:57:49 -0400, Michael

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

On Thu, May 19, 2016 at 6:57 AM, Michael Paquier wrote: > I am adding that to the commit fest of September. And a lot of activity has happened here since. Attached are refreshed patches based on da6c4f6. v11 still applies correctly but it's always better to avoid hunks

[HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

Hi all, A couple of months back is has been reported to pgsql-bugs that WAL segments were always switched with a low value of archive_timeout even if a system is completely idle: http://www.postgresql.org/message-id/20151016203031.3019.72...@wrigleys.postgresql.org In short, a closer look at the