Re: nfs subvolume access?
On Wed 2021-03-10 (08:46), Ulli Horlacher wrote: > When I try to access a btrfs filesystem via nfs, I get the error: > > root@tsmsrvi:~# mount tsmsrvj:/data/fex /nfs/tsmsrvj/fex > root@tsmsrvi:~# time find /nfs/tsmsrvj/fex | wc -l > find: File system loop detected; '/nfs/tsmsrvj/fex/spool' is part of the same > file system loop as '/nfs/tsmsrvj/fex'. It is even worse: root@tsmsrvj:# grep localhost /etc/exports /data/fex localhost(rw,async,no_subtree_check,no_root_squash) root@tsmsrvj:# mount localhost:/data/fex /nfs/localhost/fex root@tsmsrvj:# du -s /data/fex 64282240/data/fex root@tsmsrvj:# du -s /nfs/localhost/fex du: WARNING: Circular directory structure. This almost certainly means that you have a corrupted file system. NOTIFY YOUR SYSTEM MANAGER. The following directory is part of the cycle: /nfs/localhost/fex/spool 0 /nfs/localhost/fex root@tsmsrvj:# btrfs subvolume list /data ID 257 gen 42 top level 5 path fex ID 270 gen 42 top level 257 path fex/spool ID 271 gen 21 top level 270 path fex/spool/.snapshot/2021-03-07_1453.test ID 272 gen 23 top level 270 path fex/spool/.snapshot/2021-03-07_1531.test ID 273 gen 25 top level 270 path fex/spool/.snapshot/2021-03-07_1532.test ID 274 gen 27 top level 270 path fex/spool/.snapshot/2021-03-07_1718.test root@tsmsrvj:# uname -a Linux tsmsrvj 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux root@tsmsrvj:# btrfs version btrfs-progs v5.4.1 root@tsmsrvj:# dpkg -l | grep nfs- ii nfs-common 1:1.3.4-2.5ubuntu3.3 amd64NFS support files common to client and server ii nfs-kernel-server 1:1.3.4-2.5ubuntu3.3 amd64support for NFS kernel server The same bug appears if nfs server and client are different hosts or the client is an older Ubuntu 18.04 system. -- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de Allmandring 30aTel:++49-711-68565868 70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/ REF:<20210310074620.ga2...@tik.uni-stuttgart.de>
Re: nfs subvolume access?
On Wed 2021-03-10 (18:29), Forza wrote:
> Did you try the fsid on the export?
Yes:
root@tsmsrvj:/etc# grep tsm exports
/data/fex tsmsrvi(rw,async,no_subtree_check,no_root_squash,fsid=0x0011)
root@tsmsrvj:/etc# exportfs -va
exporting fex.rus.uni-stuttgart.de:/data/fex
exporting tsmsrvi.rus.uni-stuttgart.de:/data/fex
root@tsmsrvi:~# umount /nfs/tsmsrvj/fex
root@tsmsrvi:~# mount -o nfsvers=3,proto=tcp tsmsrvj:/data/fex /nfs/tsmsrvj/fex
root@tsmsrvi:~# find /nfs/tsmsrvj/fex
/nfs/tsmsrvj/fex
find: File system loop detected; '/nfs/tsmsrvj/fex/spool' is part of the same
file system loop as '/nfs/tsmsrvj/fex'.
> You may want to enable debug logging on your server.
> https://wiki.tnonline.net/w/Blog/NFS_Server_Logging
root@tsmsrvj:/etc# rpcdebug -m nfsd all
nfsd sock fh export svc proc fileop auth repcache xdr lockd
root@tsmsrvj:/var/log# tailf kern.log
2021-03-10 18:45:17 [106259.649850] nfsd_dispatch: vers 3 proc 1
2021-03-10 18:45:17 [106259.649854] nfsd: GETATTR(3) 8: 00010001 0011
2021-03-10 18:45:17 [106259.649856] nfsd: fh_verify(8: 00010001 0011
)
2021-03-10 18:45:17 [106259.650306] nfsd_dispatch: vers 3 proc 4
2021-03-10 18:45:17 [106259.650310] nfsd: ACCESS(3) 8: 00010001 0011
0x1f
2021-03-10 18:45:17 [106259.650313] nfsd: fh_verify(8: 00010001 0011
)
2021-03-10 18:45:17 [106259.650869] nfsd_dispatch: vers 3 proc 17
2021-03-10 18:45:17 [106259.650874] nfsd: READDIR+(3) 8: 00010001 0011
32768 bytes at 0
2021-03-10 18:45:17 [106259.650877] nfsd: fh_verify(8: 00010001 0011
)
2021-03-10 18:45:17 [106259.650883] nfsd: fh_verify(8: 00010001 0011
)
2021-03-10 18:45:17 [106259.650903] nfsd: fh_compose(exp 00:31/256 /fex,
ino=256)
2021-03-10 18:45:17 [106259.650907] nfsd: fh_compose(exp 00:31/256 /, ino=256)
2021-03-10 18:45:17 [106259.651454] nfsd_dispatch: vers 3 proc 3
2021-03-10 18:45:17 [106259.651459] nfsd: LOOKUP(3) 8: 00010001 0011
spool
2021-03-10 18:45:17 [106259.651463] nfsd: fh_verify(8: 00010001 0011
)
2021-03-10 18:45:17 [106259.651471] nfsd: nfsd_lookup(fh 8: 00010001 0011
, spool)
2021-03-10 18:45:17 [106259.651477] nfsd: fh_compose(exp 00:31/256 fex/spool,
ino=256)
Hmmm... and now?
--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de
Allmandring 30aTel:++49-711-68565868
70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/
REF:<55bb7f3.9ce44d1.1781d2fe...@tnonline.net>
Re: nfs subvolume access?
From: Ulli Horlacher -- Sent: 2021-03-10 - 16:55 > On Wed 2021-03-10 (09:35), Graham Cobb wrote: > >> >>> root@tsmsrvj:~# find /data/fex | wc -l >> >>> 489887 >> > >> >>I can't remember if this is why, but I've had to put a distinct >> >> fsid field in each separate subvolume being exported: >> >> >> >> /srv/nfs/home -rw,async,fsid=0x1730,no_subtree_check,no_root_squash >> > >> > I must export EACH subvolume?! >> >> I have had similar problems. I *think* the current case is that modern >> NFS, using NFS V4, can cope with the whole disk being accessible without >> giving each subvolume its own FSID (which I have stopped doing). > > I cannot use NFS4 (for several reasons). I must use NFS3 > > >> > The snapshots are generated automatically (via cron)! >> > I cannot add them to /etc/exports >> >> Well, you could write some scripts... but I don't think it is necessary. >> I *think* it is only necessary if you want `find` to be able to cross >> between subvolumes on the NFS mounted disks. > > It is not only a find problem: > > root@fex:/nfs/tsmsrvj/fex# ls -R > : > spool > ls: ./spool: not listing already-listed directory > > > And as I wrote: there is no such problem with Ubuntu 18.04! > So, is it a btrfs or a nfs bug? > > Did you try the fsid on the export? (not separate exports for all subvols) Without it the NFS server tries to enumerate it from the filesystem itself, which can cause weird issues. It is good practice to always use fsid on all exports in any case. At least with NFS4 server on my Ubuntu NFS servers at work, there are no issues with subvols for clients the mount with vers=3 You may want to enable debug logging on your server. https://wiki.tnonline.net/w/Blog/NFS_Server_Logging /Forza
Re: nfs subvolume access?
On Wed 2021-03-10 (09:35), Graham Cobb wrote: > >>> root@tsmsrvj:~# find /data/fex | wc -l > >>> 489887 > > > >>I can't remember if this is why, but I've had to put a distinct > >> fsid field in each separate subvolume being exported: > >> > >> /srv/nfs/home -rw,async,fsid=0x1730,no_subtree_check,no_root_squash > > > > I must export EACH subvolume?! > > I have had similar problems. I *think* the current case is that modern > NFS, using NFS V4, can cope with the whole disk being accessible without > giving each subvolume its own FSID (which I have stopped doing). I cannot use NFS4 (for several reasons). I must use NFS3 > > The snapshots are generated automatically (via cron)! > > I cannot add them to /etc/exports > > Well, you could write some scripts... but I don't think it is necessary. > I *think* it is only necessary if you want `find` to be able to cross > between subvolumes on the NFS mounted disks. It is not only a find problem: root@fex:/nfs/tsmsrvj/fex# ls -R : spool ls: ./spool: not listing already-listed directory And as I wrote: there is no such problem with Ubuntu 18.04! So, is it a btrfs or a nfs bug? -- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de Allmandring 30aTel:++49-711-68565868 70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/ REF:<5bded122-8adf-e5e7-dceb-37a3875f1...@cobb.uk.net>
Re: Re: nfs subvolume access?
On 10/03/2021 08:09, Ulli Horlacher wrote: > On Wed 2021-03-10 (07:59), Hugo Mills wrote: > >>> On tsmsrvj I have in /etc/exports: >>> >>> /data/fex tsmsrvi(rw,async,no_subtree_check,no_root_squash) >>> >>> This is a btrfs subvolume with snapshots: >>> >>> root@tsmsrvj:~# btrfs subvolume list /data >>> ID 257 gen 35 top level 5 path fex >>> ID 270 gen 36 top level 257 path fex/spool >>> ID 271 gen 21 top level 270 path fex/spool/.snapshot/2021-03-07_1453.test >>> ID 272 gen 23 top level 270 path fex/spool/.snapshot/2021-03-07_1531.test >>> ID 273 gen 25 top level 270 path fex/spool/.snapshot/2021-03-07_1532.test >>> ID 274 gen 27 top level 270 path fex/spool/.snapshot/2021-03-07_1718.test >>> >>> root@tsmsrvj:~# find /data/fex | wc -l >>> 489887 > >>I can't remember if this is why, but I've had to put a distinct >> fsid field in each separate subvolume being exported: >> >> /srv/nfs/home -rw,async,fsid=0x1730,no_subtree_check,no_root_squash > > I must export EACH subvolume?! I have had similar problems. I *think* the current case is that modern NFS, using NFS V4, can cope with the whole disk being accessible without giving each subvolume its own FSID (which I have stopped doing). HOWEVER, I think that find (and anything else which uses fsids and inode numbers) will see subvolumes as having duplicated inodes. > The snapshots are generated automatically (via cron)! > I cannot add them to /etc/exports Well, you could write some scripts... but I don't think it is necessary. I *think* it is only necessary if you want `find` to be able to cross between subvolumes on the NFS mounted disks. However, I am NOT an NFS expert, nor have I done a lot of work on this. I might be wrong. But I do NFS mount my snapshots disk remotely and use it. And I do see occasional complaints from find, but I live with it.
Re: nfs subvolume access?
On Wed 2021-03-10 (08:46), Ulli Horlacher wrote: > When I try to access a btrfs filesystem via nfs, I get the error: > > root@tsmsrvi:~# mount tsmsrvj:/data/fex /nfs/tsmsrvj/fex > root@tsmsrvi:~# time find /nfs/tsmsrvj/fex | wc -l > find: File system loop detected; '/nfs/tsmsrvj/fex/spool' is part of the same > file system loop as '/nfs/tsmsrvj/fex'. > 1 > tsmsrvi and tsmsrvj (nfs client and server) both run Ubuntu 20.04 with > btrfs-progs v5.4.1 On Ubuntu 18.04 this setup works without errors: root@mutter:/backup/rsync# grep tandem /etc/exports /backup/rsync/tandem 176.9.135.138(rw,async,no_subtree_check,no_root_squash) root@mutter:/backup/rsync# btrfs subvolume list /backup/rsync | grep tandem ID 257 gen 62652 top level 5 path tandem ID 5898 gen 62284 top level 257 path tandem/.snapshot/2021-03-01_0300.rsync ID 5906 gen 62284 top level 257 path tandem/.snapshot/2021-03-02_0300.rsync ID 5914 gen 62284 top level 257 path tandem/.snapshot/2021-03-03_0300.rsync ID 5924 gen 62284 top level 257 path tandem/.snapshot/2021-03-04_0300.rsync ID 5932 gen 62284 top level 257 path tandem/.snapshot/2021-03-05_0300.rsync ID 5941 gen 62284 top level 257 path tandem/.snapshot/2021-03-06_0300.rsync ID 5950 gen 62284 top level 257 path tandem/.snapshot/2021-03-07_0300.rsync ID 5962 gen 62413 top level 257 path tandem/.snapshot/2021-03-08_0300.rsync ID 5970 gen 62522 top level 257 path tandem/.snapshot/2021-03-09_0300.rsync ID 5978 gen 62626 top level 257 path tandem/.snapshot/2021-03-10_0300.rsync root@mutter:/backup/rsync# btrfs version btrfs-progs v4.15.1 root@tandem:/backup# mount | grep backup mutter:/backup/rsync/tandem on /backup type nfs (ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=1,sec=sys,mountaddr=176.9.68.251,mountvers=3,mountport=52943,mountproto=tcp,local_lock=none,addr=176.9.68.251) root@tandem:/backup# ls -l .snapshot/ total 0 drwxr-xr-x 1 root root 392 Mar 1 03:00 2021-03-01_0300.rsync drwxr-xr-x 1 root root 392 Mar 2 03:00 2021-03-02_0300.rsync drwxr-xr-x 1 root root 392 Mar 3 03:00 2021-03-03_0300.rsync drwxr-xr-x 1 root root 392 Mar 4 03:00 2021-03-04_0300.rsync drwxr-xr-x 1 root root 392 Mar 5 03:00 2021-03-05_0300.rsync drwxr-xr-x 1 root root 392 Mar 6 03:00 2021-03-06_0300.rsync drwxr-xr-x 1 root root 392 Mar 7 03:00 2021-03-07_0300.rsync drwxr-xr-x 1 root root 392 Mar 8 03:00 2021-03-08_0300.rsync drwxr-xr-x 1 root root 392 Mar 9 03:00 2021-03-09_0300.rsync drwxr-xr-x 1 root root 392 Mar 10 03:00 2021-03-10_0300.rsync So, it is an issue with the newer btrfs version on Ubuntu 20.04? -- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de Allmandring 30aTel:++49-711-68565868 70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/ REF:<20210310074620.ga2...@tik.uni-stuttgart.de>
Re: nfs subvolume access?
On Wed 2021-03-10 (07:59), Hugo Mills wrote: > > On tsmsrvj I have in /etc/exports: > > > > /data/fex tsmsrvi(rw,async,no_subtree_check,no_root_squash) > > > > This is a btrfs subvolume with snapshots: > > > > root@tsmsrvj:~# btrfs subvolume list /data > > ID 257 gen 35 top level 5 path fex > > ID 270 gen 36 top level 257 path fex/spool > > ID 271 gen 21 top level 270 path fex/spool/.snapshot/2021-03-07_1453.test > > ID 272 gen 23 top level 270 path fex/spool/.snapshot/2021-03-07_1531.test > > ID 273 gen 25 top level 270 path fex/spool/.snapshot/2021-03-07_1532.test > > ID 274 gen 27 top level 270 path fex/spool/.snapshot/2021-03-07_1718.test > > > > root@tsmsrvj:~# find /data/fex | wc -l > > 489887 >I can't remember if this is why, but I've had to put a distinct > fsid field in each separate subvolume being exported: > > /srv/nfs/home -rw,async,fsid=0x1730,no_subtree_check,no_root_squash I must export EACH subvolume?! The snapshots are generated automatically (via cron)! I cannot add them to /etc/exports -- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de Allmandring 30aTel:++49-711-68565868 70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/ REF:<20210310075957.gg22...@savella.carfax.org.uk>
Re: nfs subvolume access?
On Wed, Mar 10, 2021 at 08:46:20AM +0100, Ulli Horlacher wrote: > When I try to access a btrfs filesystem via nfs, I get the error: > > root@tsmsrvi:~# mount tsmsrvj:/data/fex /nfs/tsmsrvj/fex > root@tsmsrvi:~# time find /nfs/tsmsrvj/fex | wc -l > find: File system loop detected; '/nfs/tsmsrvj/fex/spool' is part of the same > file system loop as '/nfs/tsmsrvj/fex'. > 1 > root@tsmsrvi:~# > > > > On tsmsrvj I have in /etc/exports: > > /data/fex tsmsrvi(rw,async,no_subtree_check,no_root_squash) > > This is a btrfs subvolume with snapshots: > > root@tsmsrvj:~# btrfs subvolume list /data > ID 257 gen 35 top level 5 path fex > ID 270 gen 36 top level 257 path fex/spool > ID 271 gen 21 top level 270 path fex/spool/.snapshot/2021-03-07_1453.test > ID 272 gen 23 top level 270 path fex/spool/.snapshot/2021-03-07_1531.test > ID 273 gen 25 top level 270 path fex/spool/.snapshot/2021-03-07_1532.test > ID 274 gen 27 top level 270 path fex/spool/.snapshot/2021-03-07_1718.test > > root@tsmsrvj:~# find /data/fex | wc -l > 489887 > root@tsmsrvj:~# > > What must I add to /etc/exports to enable subvolume access for the nfs > client? > > tsmsrvi and tsmsrvj (nfs client and server) both run Ubuntu 20.04 with > btrfs-progs v5.4.1 I can't remember if this is why, but I've had to put a distinct fsid field in each separate subvolume being exported: /srv/nfs/home -rw,async,fsid=0x1730,no_subtree_check,no_root_squash It doesn't matter what value you use, as long as each one's different. Hugo. -- Hugo Mills | Alert status mauve ocelot: Slight chance of hugo@... carfax.org.uk | brimstone. Be prepared to make a nice cup of tea. http://carfax.org.uk/ | PGP: E2AB1DE4 |
nfs subvolume access?
When I try to access a btrfs filesystem via nfs, I get the error: root@tsmsrvi:~# mount tsmsrvj:/data/fex /nfs/tsmsrvj/fex root@tsmsrvi:~# time find /nfs/tsmsrvj/fex | wc -l find: File system loop detected; '/nfs/tsmsrvj/fex/spool' is part of the same file system loop as '/nfs/tsmsrvj/fex'. 1 root@tsmsrvi:~# On tsmsrvj I have in /etc/exports: /data/fex tsmsrvi(rw,async,no_subtree_check,no_root_squash) This is a btrfs subvolume with snapshots: root@tsmsrvj:~# btrfs subvolume list /data ID 257 gen 35 top level 5 path fex ID 270 gen 36 top level 257 path fex/spool ID 271 gen 21 top level 270 path fex/spool/.snapshot/2021-03-07_1453.test ID 272 gen 23 top level 270 path fex/spool/.snapshot/2021-03-07_1531.test ID 273 gen 25 top level 270 path fex/spool/.snapshot/2021-03-07_1532.test ID 274 gen 27 top level 270 path fex/spool/.snapshot/2021-03-07_1718.test root@tsmsrvj:~# find /data/fex | wc -l 489887 root@tsmsrvj:~# What must I add to /etc/exports to enable subvolume access for the nfs client? tsmsrvi and tsmsrvj (nfs client and server) both run Ubuntu 20.04 with btrfs-progs v5.4.1 -- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: horlac...@tik.uni-stuttgart.de Allmandring 30aTel:++49-711-68565868 70569 Stuttgart (Germany) WWW:http://www.tik.uni-stuttgart.de/ REF:<20210310074620.ga2...@tik.uni-stuttgart.de>
