Re: Triggering non-integrity writeback from userspace

2015-10-29 Thread Dave Chinner
On Thu, Oct 29, 2015 at 05:23:56PM +0100, Andres Freund wrote: > On 2015-10-29 12:54:22 +1100, Dave Chinner wrote: > > On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > > > By calling sync_file_range() over small ranges of pages shortly after > > > they've been written we make it

Re: Triggering non-integrity writeback from userspace

2015-10-29 Thread Andres Freund
On 2015-10-29 12:54:22 +1100, Dave Chinner wrote: > On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > > The blocking/latency of the fsync doesn't actually matter at all *for > > this callsite*. It's called from a dedicated background process - if > > it's slowed down by a couple

Re: Triggering non-integrity writeback from userspace

2015-10-29 Thread Andres Freund
On 2015-10-29 12:54:22 +1100, Dave Chinner wrote: > On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > > The blocking/latency of the fsync doesn't actually matter at all *for > > this callsite*. It's called from a dedicated background process - if > > it's slowed down by a couple

Re: Triggering non-integrity writeback from userspace

2015-10-29 Thread Dave Chinner
On Thu, Oct 29, 2015 at 05:23:56PM +0100, Andres Freund wrote: > On 2015-10-29 12:54:22 +1100, Dave Chinner wrote: > > On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > > > By calling sync_file_range() over small ranges of pages shortly after > > > they've been written we make it

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > Hi, > > On 2015-10-29 07:48:34 +1100, Dave Chinner wrote: > > > The idea of using SYNC_FILE_RANGE_WRITE beforehand is that > > > the fsync() will only have to do very little work. The language in > > > sync_file_range(2) doesn't

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
On Thu, Oct 29, 2015 at 07:48:34AM +1100, Dave Chinner wrote: > Hi Andres, > > On Wed, Oct 28, 2015 at 10:27:52AM +0100, Andres Freund wrote: > > On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > > > > Data integrity operations require related file metadata (e.g. block > > > allocation

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Andres Freund
Hi, On 2015-10-29 07:48:34 +1100, Dave Chinner wrote: > > The idea of using SYNC_FILE_RANGE_WRITE beforehand is that > > the fsync() will only have to do very little work. The language in > > sync_file_range(2) doesn't inspire enough confidence for using it as an > > actual integrity operation :/

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
Hi Andres, On Wed, Oct 28, 2015 at 10:27:52AM +0100, Andres Freund wrote: > On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > > Data integrity operations require related file metadata (e.g. block > > allocation trnascations) to be forced to the journal/disk, and a > > device cache flush

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Andres Freund
Hi, Thanks for looking into this. On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > WB_SYNC_ALL is simply a method of saying "writeback all dirty pages > and don't skip any". That's part of a data integrity operation, but > it's not what results in data integrity being provided. It may cause >

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
On Thu, Oct 29, 2015 at 12:23:12AM +0100, Andres Freund wrote: > Hi, > > On 2015-10-29 07:48:34 +1100, Dave Chinner wrote: > > > The idea of using SYNC_FILE_RANGE_WRITE beforehand is that > > > the fsync() will only have to do very little work. The language in > > > sync_file_range(2) doesn't

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
Hi Andres, On Wed, Oct 28, 2015 at 10:27:52AM +0100, Andres Freund wrote: > On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > > Data integrity operations require related file metadata (e.g. block > > allocation trnascations) to be forced to the journal/disk, and a > > device cache flush

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Andres Freund
Hi, On 2015-10-29 07:48:34 +1100, Dave Chinner wrote: > > The idea of using SYNC_FILE_RANGE_WRITE beforehand is that > > the fsync() will only have to do very little work. The language in > > sync_file_range(2) doesn't inspire enough confidence for using it as an > > actual integrity operation :/

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Dave Chinner
On Thu, Oct 29, 2015 at 07:48:34AM +1100, Dave Chinner wrote: > Hi Andres, > > On Wed, Oct 28, 2015 at 10:27:52AM +0100, Andres Freund wrote: > > On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > > > > Data integrity operations require related file metadata (e.g. block > > > allocation

Re: Triggering non-integrity writeback from userspace

2015-10-28 Thread Andres Freund
Hi, Thanks for looking into this. On 2015-10-25 08:39:12 +1100, Dave Chinner wrote: > WB_SYNC_ALL is simply a method of saying "writeback all dirty pages > and don't skip any". That's part of a data integrity operation, but > it's not what results in data integrity being provided. It may cause >

Re: Triggering non-integrity writeback from userspace

2015-10-24 Thread Jan Kara
Hi, On Thu 22-10-15 15:15:55, Andres Freund wrote: > postgres regularly has to checkpoint data to disk to be able to free > data from its journal. We currently use buffered IO and that's not > going to change short term. > > In a busy database this checkpointing process can write out a lot of

Re: Triggering non-integrity writeback from userspace

2015-10-24 Thread Dave Chinner
On Thu, Oct 22, 2015 at 03:15:55PM +0200, Andres Freund wrote: > Hi, > > postgres regularly has to checkpoint data to disk to be able to free > data from its journal. We currently use buffered IO and that's not > going to change short term. > > In a busy database this checkpointing process can

Re: Triggering non-integrity writeback from userspace

2015-10-24 Thread Dave Chinner
On Thu, Oct 22, 2015 at 03:15:55PM +0200, Andres Freund wrote: > Hi, > > postgres regularly has to checkpoint data to disk to be able to free > data from its journal. We currently use buffered IO and that's not > going to change short term. > > In a busy database this checkpointing process can

Re: Triggering non-integrity writeback from userspace

2015-10-24 Thread Jan Kara
Hi, On Thu 22-10-15 15:15:55, Andres Freund wrote: > postgres regularly has to checkpoint data to disk to be able to free > data from its journal. We currently use buffered IO and that's not > going to change short term. > > In a busy database this checkpointing process can write out a lot of

Triggering non-integrity writeback from userspace

2015-10-22 Thread Andres Freund
Hi, postgres regularly has to checkpoint data to disk to be able to free data from its journal. We currently use buffered IO and that's not going to change short term. In a busy database this checkpointing process can write out a lot of data. Currently that frequently leads to massive latency

Triggering non-integrity writeback from userspace

2015-10-22 Thread Andres Freund
Hi, postgres regularly has to checkpoint data to disk to be able to free data from its journal. We currently use buffered IO and that's not going to change short term. In a busy database this checkpointing process can write out a lot of data. Currently that frequently leads to massive latency