Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-04 Thread Jan Kara
On Tue 03-06-14 15:37:39, Daniel Phillips wrote: > On Tuesday, June 3, 2014 8:21:55 AM PDT, Jan Kara wrote: > >On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: > >>On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: > >And I agree we went for per-bdi > >flushing to avoid two threads

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-04 Thread Jan Kara
On Tue 03-06-14 15:37:39, Daniel Phillips wrote: On Tuesday, June 3, 2014 8:21:55 AM PDT, Jan Kara wrote: On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: And I agree we went for per-bdi flushing to avoid two threads congesting a

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
On Tuesday, June 3, 2014 8:21:55 AM PDT, Jan Kara wrote: On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: ... So I agree per-bdi / per-sb matters only in simple setups but machines with single rotating disk with several partitions

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Christian Stroetmann
On the 3rd of June 2014 16:57, Theodore Ts'o wrote: On Tue, Jun 03, 2014 at 07:30:32AM +0200, Christian Stroetmann wrote: In general, I do not believe that the complexity problems of soft updates, atomic writes, and related techniques can be solved by hand/manually. So my suggestion is to

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Jan Kara
On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: > On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: > > So we currently flush inodes in first dirtied first written back order when > > superblock is not specified in writeback work. That completely ignores the > > fact to which superblock

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Theodore Ts'o
On Tue, Jun 03, 2014 at 07:30:32AM +0200, Christian Stroetmann wrote: > In general, I do not believe that the complexity problems of soft updates, > atomic writes, and related techniques can be solved by hand/manually. So my > suggestion is to automatically handle the complexity problem of e.g. >

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Theodore Ts'o
On Tue, Jun 03, 2014 at 07:14:44AM -0700, Christoph Hellwig wrote: > Honestly I think doing per-bdi writeback has been a major mistake. As > you said it only even matters when we have filesystems on multiple > partitions on a single device, and even then only in a simple setup, > as soon as we

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Christoph Hellwig
On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: > So we currently flush inodes in first dirtied first written back order when > superblock is not specified in writeback work. That completely ignores the > fact to which superblock inode belongs but I don't see per-sb fairness to >

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Jan Kara
On Tue 03-06-14 17:52:09, Dave Chinner wrote: > On Tue, Jun 03, 2014 at 12:01:11AM -0700, Daniel Phillips wrote: > > > However, we already avoid the VFS writeback lists for certain > > > filesystems for pure metadata. e.g. XFS does not use the VFS dirty > > > inode lists for inode metadata

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread OGAWA Hirofumi
Dave Chinner writes: >> It doesn't move inode to end of the dirty if wb.b_dirty is empty >> (I.e. it can move from wb.b_io to wb.b_dirty too). > > Um, really? What code are you reading? From 3.15-rc8: > > static void redirty_tail(struct inode *inode, struct bdi_writeback *wb) > { >

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Dave Chinner
On Tue, Jun 03, 2014 at 04:47:52PM +0900, OGAWA Hirofumi wrote: > Daniel Phillips writes: > > > Hi Dave, > > On 06/02/2014 08:33 PM, Dave Chinner wrote: > >> On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: > >>> > >>> Redirty_tail nearly works, but "if (!list_empty(>b_dirty))"

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Dave Chinner
On Tue, Jun 03, 2014 at 12:01:11AM -0700, Daniel Phillips wrote: > Hi Dave, > On 06/02/2014 08:33 PM, Dave Chinner wrote: > > On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: > >>> H - this is using the wb dirty lists and locks, but you > >>> don't pass the wb structure to the

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread OGAWA Hirofumi
Daniel Phillips writes: > Hi Dave, > On 06/02/2014 08:33 PM, Dave Chinner wrote: >> On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: >>> >>> Redirty_tail nearly works, but "if (!list_empty(>b_dirty))" is >>> not correct because the inode needs to end up on the dirty list >>>

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
Hi Dave, Here is a non-incremental patch. This implements your suggestion from yesterday, except that the wb list lock is dropped before calling ->writeback(). Regards, Daniel >From d030d328757b160b39b252e82811a94843513cfc Mon Sep 17 00:00:00 2001 From: Daniel Phillips Date: Tue, 3 Jun 2014

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
Hi Dave, On 06/02/2014 08:33 PM, Dave Chinner wrote: > On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: >> >> Redirty_tail nearly works, but "if (!list_empty(>b_dirty))" is >> not correct because the inode needs to end up on the dirty list >> whether it was already there or not. >

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
Hi Dave, On 06/02/2014 08:33 PM, Dave Chinner wrote: On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: Redirty_tail nearly works, but if (!list_empty(wb-b_dirty)) is not correct because the inode needs to end up on the dirty list whether it was already there or not.

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
Hi Dave, Here is a non-incremental patch. This implements your suggestion from yesterday, except that the wb list lock is dropped before calling -writeback(). Regards, Daniel From d030d328757b160b39b252e82811a94843513cfc Mon Sep 17 00:00:00 2001 From: Daniel Phillips dan...@tux3.org Date: Tue,

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread OGAWA Hirofumi
Daniel Phillips dan...@phunq.net writes: Hi Dave, On 06/02/2014 08:33 PM, Dave Chinner wrote: On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: Redirty_tail nearly works, but if (!list_empty(wb-b_dirty)) is not correct because the inode needs to end up on the dirty list

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Dave Chinner
On Tue, Jun 03, 2014 at 12:01:11AM -0700, Daniel Phillips wrote: Hi Dave, On 06/02/2014 08:33 PM, Dave Chinner wrote: On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: H - this is using the wb dirty lists and locks, but you don't pass the wb structure to the writeback

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Dave Chinner
On Tue, Jun 03, 2014 at 04:47:52PM +0900, OGAWA Hirofumi wrote: Daniel Phillips dan...@phunq.net writes: Hi Dave, On 06/02/2014 08:33 PM, Dave Chinner wrote: On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: Redirty_tail nearly works, but if (!list_empty(wb-b_dirty)) is

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread OGAWA Hirofumi
Dave Chinner da...@fromorbit.com writes: It doesn't move inode to end of the dirty if wb.b_dirty is empty (I.e. it can move from wb.b_io to wb.b_dirty too). Um, really? What code are you reading? From 3.15-rc8: static void redirty_tail(struct inode *inode, struct bdi_writeback *wb) {

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Jan Kara
On Tue 03-06-14 17:52:09, Dave Chinner wrote: On Tue, Jun 03, 2014 at 12:01:11AM -0700, Daniel Phillips wrote: However, we already avoid the VFS writeback lists for certain filesystems for pure metadata. e.g. XFS does not use the VFS dirty inode lists for inode metadata changes. They

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Christoph Hellwig
On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: So we currently flush inodes in first dirtied first written back order when superblock is not specified in writeback work. That completely ignores the fact to which superblock inode belongs but I don't see per-sb fairness to actually

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Theodore Ts'o
On Tue, Jun 03, 2014 at 07:14:44AM -0700, Christoph Hellwig wrote: Honestly I think doing per-bdi writeback has been a major mistake. As you said it only even matters when we have filesystems on multiple partitions on a single device, and even then only in a simple setup, as soon as we use

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Theodore Ts'o
On Tue, Jun 03, 2014 at 07:30:32AM +0200, Christian Stroetmann wrote: In general, I do not believe that the complexity problems of soft updates, atomic writes, and related techniques can be solved by hand/manually. So my suggestion is to automatically handle the complexity problem of e.g.

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Jan Kara
On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: So we currently flush inodes in first dirtied first written back order when superblock is not specified in writeback work. That completely ignores the fact to which superblock inode

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Christian Stroetmann
On the 3rd of June 2014 16:57, Theodore Ts'o wrote: On Tue, Jun 03, 2014 at 07:30:32AM +0200, Christian Stroetmann wrote: In general, I do not believe that the complexity problems of soft updates, atomic writes, and related techniques can be solved by hand/manually. So my suggestion is to

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-03 Thread Daniel Phillips
On Tuesday, June 3, 2014 8:21:55 AM PDT, Jan Kara wrote: On Tue 03-06-14 07:14:44, Christoph Hellwig wrote: On Tue, Jun 03, 2014 at 04:05:31PM +0200, Jan Kara wrote: ... So I agree per-bdi / per-sb matters only in simple setups but machines with single rotating disk with several partitions

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Christian Stroetmann
On the 3rd of June 2014 05:39, Dave Chinner wrote: On Mon, Jun 02, 2014 at 10:30:07AM +0200, Christian Stroetmann wrote: When I followed the advice of Dave Chinner: "We're not going to merge that page forking stuff (like you were told at LSF 2013 more than a year ago:

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Dave Chinner
On Mon, Jun 02, 2014 at 10:30:07AM +0200, Christian Stroetmann wrote: > When I followed the advice of Dave Chinner: > "We're not going to merge that page forking stuff (like you were > told at LSF 2013 more than a year ago: > http://lwn.net/Articles/548091/) without rigorous design review and > a

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Dave Chinner
[ please line wrap at something sane like 68 columns ] On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: > On 06/01/2014 08:15 PM, Dave Chinner wrote: > > On Sun, Jun 01, 2014 at 02:41:02PM -0700, I wrote: > >> + > >> +/* > >> + * Add inode to writeback dirty list with current

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Daniel Phillips
On 06/01/2014 08:15 PM, Dave Chinner wrote: > On Sun, Jun 01, 2014 at 02:41:02PM -0700, I wrote: >> + >> +/* >> + * Add inode to writeback dirty list with current time. >> + */ >> +void inode_writeback_touch(struct inode *inode) >> +{ >> +struct backing_dev_info *bdi = inode->i_sb->s_bdi; >>

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Christian Stroetmann
On the 1st of June 2014 23:41, Daniel Phillips wrote: Hi, This is the first of four core changes we would like for Tux3. We start with a hard one and suggest a simple solution. The first patch in this series adds a new super operation to write back multiple inodes in a single call. The second

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Christian Stroetmann
On the 1st of June 2014 23:41, Daniel Phillips wrote: Hi, This is the first of four core changes we would like for Tux3. We start with a hard one and suggest a simple solution. The first patch in this series adds a new super operation to write back multiple inodes in a single call. The second

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Daniel Phillips
On 06/01/2014 08:15 PM, Dave Chinner wrote: On Sun, Jun 01, 2014 at 02:41:02PM -0700, I wrote: + +/* + * Add inode to writeback dirty list with current time. + */ +void inode_writeback_touch(struct inode *inode) +{ +struct backing_dev_info *bdi = inode-i_sb-s_bdi; +

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Dave Chinner
[ please line wrap at something sane like 68 columns ] On Mon, Jun 02, 2014 at 01:02:29PM -0700, Daniel Phillips wrote: On 06/01/2014 08:15 PM, Dave Chinner wrote: On Sun, Jun 01, 2014 at 02:41:02PM -0700, I wrote: + +/* + * Add inode to writeback dirty list with current time. + */

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Dave Chinner
On Mon, Jun 02, 2014 at 10:30:07AM +0200, Christian Stroetmann wrote: When I followed the advice of Dave Chinner: We're not going to merge that page forking stuff (like you were told at LSF 2013 more than a year ago: http://lwn.net/Articles/548091/) without rigorous design review and a

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-02 Thread Christian Stroetmann
On the 3rd of June 2014 05:39, Dave Chinner wrote: On Mon, Jun 02, 2014 at 10:30:07AM +0200, Christian Stroetmann wrote: When I followed the advice of Dave Chinner: We're not going to merge that page forking stuff (like you were told at LSF 2013 more than a year ago:

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-01 Thread Dave Chinner
On Sun, Jun 01, 2014 at 02:41:02PM -0700, Daniel Phillips wrote: > --- > From: Daniel Phillips > Subject: [PATCH] Add a super operation for writeback > > Add a "writeback" super operation to be called in the > form: > > progress = sb->s_op->writeback(sb, , ); > > The filesystem is

[RFC][PATCH 1/2] Add a super operation for writeback

2014-06-01 Thread Daniel Phillips
Hi, This is the first of four core changes we would like for Tux3. We start with a hard one and suggest a simple solution. The first patch in this series adds a new super operation to write back multiple inodes in a single call. The second patch applies to our linux-tux3 repository at

[RFC][PATCH 1/2] Add a super operation for writeback

2014-06-01 Thread Daniel Phillips
Hi, This is the first of four core changes we would like for Tux3. We start with a hard one and suggest a simple solution. The first patch in this series adds a new super operation to write back multiple inodes in a single call. The second patch applies to our linux-tux3 repository at

Re: [RFC][PATCH 1/2] Add a super operation for writeback

2014-06-01 Thread Dave Chinner
On Sun, Jun 01, 2014 at 02:41:02PM -0700, Daniel Phillips wrote: --- From: Daniel Phillips dan...@tux3.org Subject: [PATCH] Add a super operation for writeback Add a writeback super operation to be called in the form: progress = sb-s_op-writeback(sb, wbc, pages); The