Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
On Wed, Apr 13, 2022 at 10:09:40AM -0700, Dan Williams wrote: > Yes, sounds like we're on the same page. I had mistakenly interpreted > "Hence these notifications need to be delayed until after the > filesystem is mounted" as something the producer would need to handle, > but yes, consumer is free to drop if the notification arrives at an > inopportune time. A SB_BORN check might be all that we need.
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
On Tue, Apr 12, 2022 at 11:10 PM Dave Chinner wrote: > > On Tue, Apr 12, 2022 at 07:06:40PM -0700, Dan Williams wrote: > > On Tue, Apr 12, 2022 at 5:04 PM Dave Chinner wrote: > > > On Mon, Apr 11, 2022 at 12:09:03AM +0800, Shiyang Ruan wrote: > > > > Introduce xfs_notify_failure.c to handle failure related works, such as > > > > implement ->notify_failure(), register/unregister dax holder in xfs, and > > > > so on. > > > > > > > > If the rmap feature of XFS enabled, we can query it to find files and > > > > metadata which are associated with the corrupt data. For now all we do > > > > is kill processes with that file mapped into their address spaces, but > > > > future patches could actually do something about corrupt metadata. > > > > > > > > After that, the memory failure needs to notify the processes who are > > > > using those files. > ... > > > > @@ -1964,8 +1965,8 @@ xfs_alloc_buftarg( > > > > btp->bt_mount = mp; > > > > btp->bt_dev = bdev->bd_dev; > > > > btp->bt_bdev = bdev; > > > > - btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, > > > > NULL, > > > > - NULL); > > > > + btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, > > > > mp, > > > > + _dax_holder_operations); > > > > > > I see a problem with this: we are setting up notify callbacks before > > > we've even read in the superblock during mount. i.e. we don't even > > > kow yet if we've got an XFS filesystem on this block device. > > > Hence these notifications need to be delayed until after the > > > filesystem is mounted, all the internal structures have been set up > > > and log recovery has completed. > > > > So I think this gets back to the fact that there will eventually be 2 > > paths into this notifier. > > I'm not really concerned by how the notifications are generated; > my concern is purely that notifications can be handled safely. > > > All that to say, I think it is ok / expected for the filesystem to > > drop notifications on the floor when it is not ready to handle them. > > Well, yes. The whole point of notifications is the consumer makes > the decision on what to do with the notification it receives - the > producer of the notification does not (and can not) dictate what > policy the consumer(s) implement... > > > For example there are no processes to send SIGBUS to if the filesystem > > has not even finished mount. > > There may be not processes to send SIGBUS to even if the filesystem > has finished mount. But we still want the notifications to be > delivered and we still need to handle them safely. > > IOWs, while we might start by avoiding notifications during mount, > this doesn't mean we will never have reason to process events during > mount. What we do with this notification is going to evolve over > time as we add new and adapt existing functionality Yes, sounds like we're on the same page. I had mistakenly interpreted "Hence these notifications need to be delayed until after the filesystem is mounted" as something the producer would need to handle, but yes, consumer is free to drop if the notification arrives at an inopportune time.
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
On Tue, Apr 12, 2022 at 07:06:40PM -0700, Dan Williams wrote: > On Tue, Apr 12, 2022 at 5:04 PM Dave Chinner wrote: > > On Mon, Apr 11, 2022 at 12:09:03AM +0800, Shiyang Ruan wrote: > > > Introduce xfs_notify_failure.c to handle failure related works, such as > > > implement ->notify_failure(), register/unregister dax holder in xfs, and > > > so on. > > > > > > If the rmap feature of XFS enabled, we can query it to find files and > > > metadata which are associated with the corrupt data. For now all we do > > > is kill processes with that file mapped into their address spaces, but > > > future patches could actually do something about corrupt metadata. > > > > > > After that, the memory failure needs to notify the processes who are > > > using those files. ... > > > @@ -1964,8 +1965,8 @@ xfs_alloc_buftarg( > > > btp->bt_mount = mp; > > > btp->bt_dev = bdev->bd_dev; > > > btp->bt_bdev = bdev; > > > - btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, > > > NULL, > > > - NULL); > > > + btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, mp, > > > + _dax_holder_operations); > > > > I see a problem with this: we are setting up notify callbacks before > > we've even read in the superblock during mount. i.e. we don't even > > kow yet if we've got an XFS filesystem on this block device. > > Hence these notifications need to be delayed until after the > > filesystem is mounted, all the internal structures have been set up > > and log recovery has completed. > > So I think this gets back to the fact that there will eventually be 2 > paths into this notifier. I'm not really concerned by how the notifications are generated; my concern is purely that notifications can be handled safely. > All that to say, I think it is ok / expected for the filesystem to > drop notifications on the floor when it is not ready to handle them. Well, yes. The whole point of notifications is the consumer makes the decision on what to do with the notification it receives - the producer of the notification does not (and can not) dictate what policy the consumer(s) implement... > For example there are no processes to send SIGBUS to if the filesystem > has not even finished mount. There may be not processes to send SIGBUS to even if the filesystem has finished mount. But we still want the notifications to be delivered and we still need to handle them safely. IOWs, while we might start by avoiding notifications during mount, this doesn't mean we will never have reason to process events during mount. What we do with this notification is going to evolve over time as we add new and adapt existing functionality Cheers, Dave. -- Dave Chinner da...@fromorbit.com
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
On Tue, Apr 12, 2022 at 5:04 PM Dave Chinner wrote:
>
> On Mon, Apr 11, 2022 at 12:09:03AM +0800, Shiyang Ruan wrote:
> > Introduce xfs_notify_failure.c to handle failure related works, such as
> > implement ->notify_failure(), register/unregister dax holder in xfs, and
> > so on.
> >
> > If the rmap feature of XFS enabled, we can query it to find files and
> > metadata which are associated with the corrupt data. For now all we do
> > is kill processes with that file mapped into their address spaces, but
> > future patches could actually do something about corrupt metadata.
> >
> > After that, the memory failure needs to notify the processes who are
> > using those files.
> >
> > Signed-off-by: Shiyang Ruan
> > ---
> > fs/xfs/Makefile | 5 +
> > fs/xfs/xfs_buf.c| 7 +-
> > fs/xfs/xfs_fsops.c | 3 +
> > fs/xfs/xfs_mount.h | 1 +
> > fs/xfs/xfs_notify_failure.c | 219
> > fs/xfs/xfs_super.h | 1 +
> > 6 files changed, 233 insertions(+), 3 deletions(-)
> > create mode 100644 fs/xfs/xfs_notify_failure.c
> >
> > diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile
> > index 04611a1068b4..09f5560e29f2 100644
> > --- a/fs/xfs/Makefile
> > +++ b/fs/xfs/Makefile
> > @@ -128,6 +128,11 @@ xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o
> > xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o
> > xfs-$(CONFIG_EXPORTFS_BLOCK_OPS) += xfs_pnfs.o
> >
> > +# notify failure
> > +ifeq ($(CONFIG_MEMORY_FAILURE),y)
> > +xfs-$(CONFIG_FS_DAX) += xfs_notify_failure.o
> > +endif
> > +
> > # online scrub/repair
> > ifeq ($(CONFIG_XFS_ONLINE_SCRUB),y)
> >
> > diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
> > index f9ca08398d32..9064b8dfbc66 100644
> > --- a/fs/xfs/xfs_buf.c
> > +++ b/fs/xfs/xfs_buf.c
> > @@ -5,6 +5,7 @@
> > */
> > #include "xfs.h"
> > #include
> > +#include
> >
> > #include "xfs_shared.h"
> > #include "xfs_format.h"
> > @@ -1911,7 +1912,7 @@ xfs_free_buftarg(
> > list_lru_destroy(>bt_lru);
> >
> > blkdev_issue_flush(btp->bt_bdev);
> > - fs_put_dax(btp->bt_daxdev, NULL);
> > + fs_put_dax(btp->bt_daxdev, btp->bt_mount);
> >
> > kmem_free(btp);
> > }
> > @@ -1964,8 +1965,8 @@ xfs_alloc_buftarg(
> > btp->bt_mount = mp;
> > btp->bt_dev = bdev->bd_dev;
> > btp->bt_bdev = bdev;
> > - btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, NULL,
> > - NULL);
> > + btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, mp,
> > + _dax_holder_operations);
>
> I see a problem with this: we are setting up notify callbacks before
> we've even read in the superblock during mount. i.e. we don't even
> kow yet if we've got an XFS filesystem on this block device.
>
> Hence if we get a notification immediately after registering this
> notification callback
>
> [...]
>
> > +
> > +static int
> > +xfs_dax_notify_ddev_failure(
> > + struct xfs_mount*mp,
> > + xfs_daddr_t daddr,
> > + xfs_daddr_t bblen,
> > + int mf_flags)
> > +{
> > + struct xfs_trans*tp = NULL;
> > + struct xfs_btree_cur*cur = NULL;
> > + struct xfs_buf *agf_bp = NULL;
> > + int error = 0;
> > + xfs_fsblock_t fsbno = XFS_DADDR_TO_FSB(mp, daddr);
> > + xfs_agnumber_t agno = XFS_FSB_TO_AGNO(mp, fsbno);
> > + xfs_fsblock_t end_fsbno = XFS_DADDR_TO_FSB(mp, daddr +
> > bblen);
> > + xfs_agnumber_t end_agno = XFS_FSB_TO_AGNO(mp, end_fsbno);
>
> none of this code is going to function correctly because it
> is dependent on the superblock having been read, validated and
> copied to the in-memory superblock.
>
> > + error = xfs_trans_alloc_empty(mp, );
> > + if (error)
> > + return error;
>
> ... and it's not valid to use transactions (even empty ones) before
> log recovery has completed and set the log up correctly.
>
> > +
> > + for (; agno <= end_agno; agno++) {
> > + struct xfs_rmap_irecri_low = { };
> > + struct xfs_rmap_irecri_high;
> > + struct failure_info notify;
> > + struct xfs_agf *agf;
> > + xfs_agblock_t agend;
> > +
> > + error = xfs_alloc_read_agf(mp, tp, agno, 0, _bp);
> > + if (error)
> > + break;
> > +
> > + cur = xfs_rmapbt_init_cursor(mp, tp, agf_bp, agf_bp->b_pag);
>
> ... and none of the structures this rmapbt walk is dependent on
> (e.g. perag structures) have been initialised yet so there's null
> pointer dereferences going to happen here.
>
> Perhaps even worse is that the rmapbt is not guaranteed to be in
> consistent state until after log recovery has completed, so this
> walk could get stuck forever in a stale on-disk cycle that
> recovery
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
On Mon, Apr 11, 2022 at 12:09:03AM +0800, Shiyang Ruan wrote:
> Introduce xfs_notify_failure.c to handle failure related works, such as
> implement ->notify_failure(), register/unregister dax holder in xfs, and
> so on.
>
> If the rmap feature of XFS enabled, we can query it to find files and
> metadata which are associated with the corrupt data. For now all we do
> is kill processes with that file mapped into their address spaces, but
> future patches could actually do something about corrupt metadata.
>
> After that, the memory failure needs to notify the processes who are
> using those files.
>
> Signed-off-by: Shiyang Ruan
> ---
> fs/xfs/Makefile | 5 +
> fs/xfs/xfs_buf.c| 7 +-
> fs/xfs/xfs_fsops.c | 3 +
> fs/xfs/xfs_mount.h | 1 +
> fs/xfs/xfs_notify_failure.c | 219
> fs/xfs/xfs_super.h | 1 +
> 6 files changed, 233 insertions(+), 3 deletions(-)
> create mode 100644 fs/xfs/xfs_notify_failure.c
>
> diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile
> index 04611a1068b4..09f5560e29f2 100644
> --- a/fs/xfs/Makefile
> +++ b/fs/xfs/Makefile
> @@ -128,6 +128,11 @@ xfs-$(CONFIG_SYSCTL) += xfs_sysctl.o
> xfs-$(CONFIG_COMPAT) += xfs_ioctl32.o
> xfs-$(CONFIG_EXPORTFS_BLOCK_OPS) += xfs_pnfs.o
>
> +# notify failure
> +ifeq ($(CONFIG_MEMORY_FAILURE),y)
> +xfs-$(CONFIG_FS_DAX) += xfs_notify_failure.o
> +endif
> +
> # online scrub/repair
> ifeq ($(CONFIG_XFS_ONLINE_SCRUB),y)
>
> diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
> index f9ca08398d32..9064b8dfbc66 100644
> --- a/fs/xfs/xfs_buf.c
> +++ b/fs/xfs/xfs_buf.c
> @@ -5,6 +5,7 @@
> */
> #include "xfs.h"
> #include
> +#include
>
> #include "xfs_shared.h"
> #include "xfs_format.h"
> @@ -1911,7 +1912,7 @@ xfs_free_buftarg(
> list_lru_destroy(>bt_lru);
>
> blkdev_issue_flush(btp->bt_bdev);
> - fs_put_dax(btp->bt_daxdev, NULL);
> + fs_put_dax(btp->bt_daxdev, btp->bt_mount);
>
> kmem_free(btp);
> }
> @@ -1964,8 +1965,8 @@ xfs_alloc_buftarg(
> btp->bt_mount = mp;
> btp->bt_dev = bdev->bd_dev;
> btp->bt_bdev = bdev;
> - btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, NULL,
> - NULL);
> + btp->bt_daxdev = fs_dax_get_by_bdev(bdev, >bt_dax_part_off, mp,
> + _dax_holder_operations);
I see a problem with this: we are setting up notify callbacks before
we've even read in the superblock during mount. i.e. we don't even
kow yet if we've got an XFS filesystem on this block device.
Hence if we get a notification immediately after registering this
notification callback
[...]
> +
> +static int
> +xfs_dax_notify_ddev_failure(
> + struct xfs_mount*mp,
> + xfs_daddr_t daddr,
> + xfs_daddr_t bblen,
> + int mf_flags)
> +{
> + struct xfs_trans*tp = NULL;
> + struct xfs_btree_cur*cur = NULL;
> + struct xfs_buf *agf_bp = NULL;
> + int error = 0;
> + xfs_fsblock_t fsbno = XFS_DADDR_TO_FSB(mp, daddr);
> + xfs_agnumber_t agno = XFS_FSB_TO_AGNO(mp, fsbno);
> + xfs_fsblock_t end_fsbno = XFS_DADDR_TO_FSB(mp, daddr + bblen);
> + xfs_agnumber_t end_agno = XFS_FSB_TO_AGNO(mp, end_fsbno);
none of this code is going to function correctly because it
is dependent on the superblock having been read, validated and
copied to the in-memory superblock.
> + error = xfs_trans_alloc_empty(mp, );
> + if (error)
> + return error;
... and it's not valid to use transactions (even empty ones) before
log recovery has completed and set the log up correctly.
> +
> + for (; agno <= end_agno; agno++) {
> + struct xfs_rmap_irecri_low = { };
> + struct xfs_rmap_irecri_high;
> + struct failure_info notify;
> + struct xfs_agf *agf;
> + xfs_agblock_t agend;
> +
> + error = xfs_alloc_read_agf(mp, tp, agno, 0, _bp);
> + if (error)
> + break;
> +
> + cur = xfs_rmapbt_init_cursor(mp, tp, agf_bp, agf_bp->b_pag);
... and none of the structures this rmapbt walk is dependent on
(e.g. perag structures) have been initialised yet so there's null
pointer dereferences going to happen here.
Perhaps even worse is that the rmapbt is not guaranteed to be in
consistent state until after log recovery has completed, so this
walk could get stuck forever in a stale on-disk cycle that
recovery would have corrected
Hence these notifications need to be delayed until after the
filesystem is mounted, all the internal structures have been set up
and log recovery has completed.
Cheers,
Dave.
--
Dave Chinner
da...@fromorbit.com
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
> --- a/fs/xfs/xfs_super.h > +++ b/fs/xfs/xfs_super.h > @@ -93,6 +93,7 @@ extern xfs_agnumber_t xfs_set_inode_alloc(struct xfs_mount > *, > extern const struct export_operations xfs_export_operations; > extern const struct xattr_handler *xfs_xattr_handlers[]; > extern const struct quotactl_ops xfs_quotactl_operations; > +extern const struct dax_holder_operations xfs_dax_holder_operations; This needs to be defined to NULL if at least one of CONFIG_FS_DAX or CONFIG_MEMORY_FAILURE is not set. Otherwise looks good: Reviewed-by: Christoph Hellwig
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
Hi Shiyang, Thank you for the patch! Yet something to improve: [auto build test ERROR on hnaz-mm/master] [also build test ERROR on next-20220408] [cannot apply to xfs-linux/for-next linus/master linux/master v5.18-rc1] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/intel-lab-lkp/linux/commits/Shiyang-Ruan/fsdax-introduce-fs-query-to-support-reflink/20220411-001048 base: https://github.com/hnaz/linux-mm master config: s390-allyesconfig (https://download.01.org/0day-ci/archive/20220411/202204110700.66eh1xzg-...@intel.com/config) compiler: s390-linux-gcc (GCC) 11.2.0 reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # https://github.com/intel-lab-lkp/linux/commit/bf68be0c39b8ecc4223b948a9ee126af167d74f0 git remote add linux-review https://github.com/intel-lab-lkp/linux git fetch --no-tags linux-review Shiyang-Ruan/fsdax-introduce-fs-query-to-support-reflink/20220411-001048 git checkout bf68be0c39b8ecc4223b948a9ee126af167d74f0 # save the config file to linux build tree mkdir build_dir COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=s390 SHELL=/bin/bash If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot All errors (new ones prefixed by >>): s390-linux-ld: fs/xfs/xfs_buf.o: in function `xfs_alloc_buftarg': >> xfs_buf.c:(.text+0x9920): undefined reference to `xfs_dax_holder_operations' -- 0-DAY CI Kernel Test Service https://01.org/lkp
Re: [PATCH v12 6/7] xfs: Implement ->notify_failure() for XFS
Hi Shiyang,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on hnaz-mm/master]
[also build test ERROR on next-20220408]
[cannot apply to xfs-linux/for-next linus/master linux/master v5.18-rc1]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url:
https://github.com/intel-lab-lkp/linux/commits/Shiyang-Ruan/fsdax-introduce-fs-query-to-support-reflink/20220411-001048
base: https://github.com/hnaz/linux-mm master
config: s390-defconfig
(https://download.01.org/0day-ci/archive/20220411/202204110240.oa3g7lsw-...@intel.com/config)
compiler: s390-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget
https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O
~/bin/make.cross
chmod +x ~/bin/make.cross
#
https://github.com/intel-lab-lkp/linux/commit/bf68be0c39b8ecc4223b948a9ee126af167d74f0
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review
Shiyang-Ruan/fsdax-introduce-fs-query-to-support-reflink/20220411-001048
git checkout bf68be0c39b8ecc4223b948a9ee126af167d74f0
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross
O=build_dir ARCH=s390 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot
All errors (new ones prefixed by >>):
s390-linux-ld: fs/xfs/xfs_buf.o: in function `xfs_alloc_buftarg':
>> fs/xfs/xfs_buf.c:1968: undefined reference to `xfs_dax_holder_operations'
pahole: .tmp_vmlinux.btf: No such file or directory
.btf.vmlinux.bin.o: file not recognized: file format not recognized
vim +1968 fs/xfs/xfs_buf.c
1955
1956 struct xfs_buftarg *
1957 xfs_alloc_buftarg(
1958 struct xfs_mount*mp,
1959 struct block_device *bdev)
1960 {
1961 xfs_buftarg_t *btp;
1962
1963 btp = kmem_zalloc(sizeof(*btp), KM_NOFS);
1964
1965 btp->bt_mount = mp;
1966 btp->bt_dev = bdev->bd_dev;
1967 btp->bt_bdev = bdev;
> 1968 btp->bt_daxdev = fs_dax_get_by_bdev(bdev,
> >bt_dax_part_off, mp,
1969 _dax_holder_operations);
1970
1971 /*
1972 * Buffer IO error rate limiting. Limit it to no more than 10
messages
1973 * per 30 seconds so as to not spam logs too much on repeated
errors.
1974 */
1975 ratelimit_state_init(>bt_ioerror_rl, 30 * HZ,
1976 DEFAULT_RATELIMIT_BURST);
1977
1978 if (xfs_setsize_buftarg_early(btp, bdev))
1979 goto error_free;
1980
1981 if (list_lru_init(>bt_lru))
1982 goto error_free;
1983
1984 if (percpu_counter_init(>bt_io_count, 0, GFP_KERNEL))
1985 goto error_lru;
1986
1987 btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count;
1988 btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan;
1989 btp->bt_shrinker.seeks = DEFAULT_SEEKS;
1990 btp->bt_shrinker.flags = SHRINKER_NUMA_AWARE;
1991 if (register_shrinker(>bt_shrinker))
1992 goto error_pcpu;
1993 return btp;
1994
1995 error_pcpu:
1996 percpu_counter_destroy(>bt_io_count);
1997 error_lru:
1998 list_lru_destroy(>bt_lru);
1999 error_free:
2000 kmem_free(btp);
2001 return NULL;
2002 }
2003
--
0-DAY CI Kernel Test Service
https://01.org/lkp
