Re: Stale NFS file handle when using aufs on top of btrfs?
Sorry it's been a bit since I reported back, but I got a chance to apply your patch and I have mixed results to report. Basically, it looks like the patch may resolve the issues, but not entirely. After building the new kernel, I was still able to reproduce the problem initially in a sub directory, but not on the top level. When it was happening, this is what showed up in the logs: Feb  8 15:09:57 aufs kernel: [  998.810169] [ cut here ] Feb  8 15:09:57 aufs kernel: [  998.810177] WARNING: CPU: 0 PID: 2076 at /build/buildd/linux-3.16.0/fs/inode.c:282 drop_nlink+0x41/0x50() Feb  8 15:09:57 aufs kernel: [  998.810178] Modules linked in: aufs serio_raw snd_hda_codec_generic kvm_amd kvm qxl crct10dif_pclmul ttm drm_kms_helper crc32_pclmul ghash_clmulni_intel aesni_intel drm aes_x86_64 lrw ppdev snd_hda_intel snd_hda_controller i2c_piix4 snd_hda_codec gf128mul glue_helper snd_hwdep snd_pcm snd_timer ablk_helper cryptd snd parport_pc soundcore pvpanic parport mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  8 15:09:57 aufs kernel: [  998.810200] CPU: 0 PID: 2076 Comm: rm Tainted: G     W   3.16.0-29-generic #39-Ubuntu Feb  8 15:09:57 aufs kernel: [  998.810202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  8 15:09:57 aufs kernel: [  998.810204]  0009 88003c7f3cc8 8178218a Feb  8 15:09:57 aufs kernel: [  998.810206]  88003c7f3d00 8106fedd 88003bc1a658 88003bc1a658 Feb  8 15:09:57 aufs kernel: [  998.810208]  0001 88003cbcede0 880039d91c00 88003c7f3d10 Feb  8 15:09:57 aufs kernel: [  998.810210] Call Trace: Feb  8 15:09:57 aufs kernel: [  998.810216]  [8178218a] dump_stack+0x45/0x56 Feb  8 15:09:57 aufs kernel: [  998.810219]  [8106fedd] warn_slowpath_common+0x7d/0xa0 Feb  8 15:09:57 aufs kernel: [  998.810221]  [8106ffba] warn_slowpath_null+0x1a/0x20 Feb  8 15:09:57 aufs kernel: [  998.810224]  [811fcc31] drop_nlink+0x41/0x50 Feb  8 15:09:57 aufs kernel: [  998.810231]  [c03d6c1d] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810236]  [c03d5e5e] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810241]  [c03e52e7] aufs_rmdir+0x2b7/0x430 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810244]  [811f8390] ? prepend.constprop.25+0x30/0x30 Feb  8 15:09:57 aufs kernel: [  998.810246]  [811eede7] vfs_rmdir+0xa7/0x100 Feb  8 15:09:57 aufs kernel: [  998.810248]  [811f20b9] do_rmdir+0x1d9/0x1f0 Feb  8 15:09:57 aufs kernel: [  998.810250]  [811e2dbe] ? fput+0xe/0x10 Feb  8 15:09:57 aufs kernel: [  998.810253]  [81091afc] ? task_work_run+0xbc/0xf0 Feb  8 15:09:57 aufs kernel: [  998.810256]  [810131e7] ? do_notify_resume+0x97/0xb0 Feb  8 15:09:57 aufs kernel: [  998.810258]  [811f3365] SyS_unlinkat+0x25/0x40 Feb  8 15:09:57 aufs kernel: [  998.810261]  [8178a1ad] system_call_fastpath+0x1a/0x1f Feb  8 15:09:57 aufs kernel: [  998.810263] ---[ end trace 7e60ac9f449b2a85 ]--- Once I saw that it was still happening, I attempted to clean things up and present a more complete example of how I was getting this to happen. I unmounted the aufs filesystem and removed the contents of the underlying filesystems (including the .wh* bits) and remounted. Once I saw what the aufs mountpoint was correctly reflecting that everything was removed, I tried to reproduce the error again, and I have not been able to. I can only assume that something with the old .wh* files was causing some sort of issue. Ultimately, at this time I cannot reproduce the problem, so I guess we call it fixed? On Wed, Feb 4, 2015 at 5:41 AM, [1]sf...@users.sourceforge.net wrote: Michael Johnson - MJ: I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs.  Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related:     ::: Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at
Re: Stale NFS file handle when using aufs on top of btrfs?
I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs.  Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; The end result is always a stale file handle. If you 'cd .' following this set of commands the error clears, unless you are in the top level of the aufs filesystem. If you are in the top level, you must umount/mount the aufs file system to recover. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related: Feb  3 23:54:38 aufs kernel: [ 3577.351197] [ cut here ] Feb  3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at /build/buildd/linux-3.16.0/fs /inode.c:282 drop_nlink+0x41/0x50() Feb  3 23:54:38 aufs kernel: [ 3577.351208] Modules linked in: aufs snd_hda_codec_generic kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller snd_hda_codec qxl snd_hwdep snd_pcm ttm drm_kms_helper snd_timer serio_raw drm snd soundcore parport_pc pvpanic parport i2c_piix4 mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  3 23:54:38 aufs kernel: [ 3577.351233] CPU: 0 PID: 3725 Comm: rm Tainted: G     W   3.16.0-23-generic #31-Ubuntu Feb  3 23:54:38 aufs kernel: [ 3577.351234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  3 23:54:38 aufs kernel: [ 3577.351236]  0009 88003a07fcc8 8177fcbc Feb  3 23:54:38 aufs kernel: [ 3577.351238]  88003a07fd00 8106fd8d 88003a742058 88003a742058 Feb  3 23:54:38 aufs kernel: [ 3577.351240]  88003b3ac240 88003b92b100 88003a07fd10 Feb  3 23:54:38 aufs kernel: [ 3577.351251] Call Trace: Feb  3 23:54:38 aufs kernel: [ 3577.351257]  [8177fcbc] dump_stack+0x45/0x56 Feb  3 23:54:38 aufs kernel: [ 3577.351261]  [8106fd8d] warn_slowpath_common+0x7d/0xa0 Feb  3 23:54:38 aufs kernel: [ 3577.351263]  [8106fe6a] warn_slowpath_null+0x1a/0x20 Feb  3 23:54:38 aufs kernel: [ 3577.351265]  [811fc5f1] drop_nlink+0x41/0x50 Feb  3 23:54:38 aufs kernel: [ 3577.351273]  [c0510c1d] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351278]  [c050fe5e] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351283]  [c051f2e7] aufs_rmdir+0x2b7/0x430 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351285]  [811f7d50] ? prepend.constprop.25+0x30/0x30 Feb  3 23:54:38 aufs kernel: [ 3577.351288]  [811eef57] vfs_rmdir+0xa7/0x100 Feb  3 23:54:38 aufs kernel: [ 3577.351290]  [811f1a79] do_rmdir+0x1d9/0x1f0 Feb  3 23:54:38 aufs kernel: [ 3577.351293]  [811e285e] ? fput+0xe/0x10 Feb  3 23:54:38 aufs kernel: [ 3577.351295]  [810919ac] ? task_work_run+0xbc/0xf0 Feb  3 23:54:38 aufs kernel: [ 3577.351299]  [810130f7] ? do_notify_resume+0x97/0xb0 Feb  3 23:54:38 aufs kernel: [ 3577.351301]  [811f2d25] SyS_unlinkat+0x25/0x40 Feb  3 23:54:38 aufs kernel: [ 3577.351304]  [81787ced] system_call_fastpath+0x1a/0x1f Feb  3 23:54:38 aufs kernel: [ 3577.351391] ---[ end trace d717630435aab60d ]--- I could not get this to occur on a regular basis, but it popped up at some point. Hopefully that is of some help. Here are all the aufs config options for my current build: CONFIG_AUFS_FS=m CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set CONFIG_AUFS_EXPORT=y CONFIG_AUFS_INO_T_64=y # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SP_IATTR is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set # CONFIG_AUFS_BR_HFSPLUS is not set CONFIG_AUFS_BDEV_LOOP=y CONFIG_AUFS_DEBUG=y CONFIG_AUFS_MAGIC_SYSRQ=y I have not yet applied the patch, but I will give it a try tomorrow to see if it makes a difference in behavior or log messages. On Tue, Feb 3, 2015 at 8:06 PM, [1]sf...@users.sourceforge.net wrote: MJ, Dan, I appriciate your efforts. Dan Kegel: That failed with no config found, so I repeated it after doing $
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs. Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related: ::: Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at /build/buildd/linux-3.16.0/fs /inode.c:282 drop_nlink+0x41/0x50() It is one of good news. It is caused by a strange link count of a dir on btrfs. At least this problem will be solved by the patch I sent previously. But we may meet another problem later. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. On Feb 1, 2015 10:37 PM, [1]sf...@users.sourceforge.net wrote: [2]sf...@users.sourceforge.net: I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration   (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the   problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs  # mkdir rw ro u  # sudo mount -t aufs -o br=/rw:/ro none /u  # cd /u  # sudo touch fileA  # sudo mkdir -p dir1/dir2  # sudo rm -fr dir1  # ls  and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module  which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by  ls -l $(modinfo -F filename aufs) and see the timestamp. - make sure that you can receive the kern.debug logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just  before rm, run echo 1 /sys/module/aufs/parameters/debug. and just  after ls, run echo 0 /sys/module/aufs/parameters/debug.  Setting 1 enables aufs debugging feature and produces many kernel  debug log. To stop the debug log, set 0 to  /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. mailto:sf...@users.sourceforge.net -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs # mkdir rw ro u # sudo mount -t aufs -o br=/rw:/ro none /u # cd /u # sudo touch fileA # sudo mkdir -p dir1/dir2 # sudo rm -fr dir1 # ls and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by ls -l $(modinfo -F filename aufs) and see the timestamp. - make sure that you can receive the kern.debug logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just before rm, run echo 1 /sys/module/aufs/parameters/debug. and just after ls, run echo 0 /sys/module/aufs/parameters/debug. Setting 1 enables aufs debugging feature and produces many kernel debug log. To stop the debug log, set 0 to /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: Anyway I will try ubuntu kernel by myself, although it will take long time... I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# mkdir -p dir1/dir2 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# rm -rf dir1 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# ls ls: cannot open directory .: Stale file handle Hmm, I don't understand why I cannot reproduce... I may be able to try building the Ubuntu kernel, but I'm a bit slammed now, not sure when I can try it. But if your patch works it'd save us some effort. Ok, when you can, please try the attached patch. Here's the other info you requested: Ok, thanx. I understand your branches and options. You don't have compress=zlib. Oh but you have noplink. Anyway I will try ubuntu kernel by myself, although it will take long time... J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I see very similar behavior (and always have with AUFS) and it is easily reproducible. Here is how to reproduce 100% of the time: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle To mke the error go away: $ cd . And now an ls works flawlessly. Basically, any time you have a directory with one or more files or directories inside of it, and then you recursively remove it while the working directory is the parent, subsequent attempts to access files inside of the parent will fail with the 'Stale file handle' error. For the most part this does not impact me becuase most things are not working of files/directories that are a immediate descent of the current working directory. But it would be nice if this problem did not occur. Hope this helps to at least understand the problem better and possible provide you with a workaround. On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel [1]d...@kegel.com wrote: I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: [2]http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per [3]http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. [4]http://goparallel.sourceforge.net/ -- Michael Johnson - MJ References 1. mailto:d...@kegel.com 2. http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 3. http://aufs.sourceforge.net/ 4. http://goparallel.sourceforge.net/ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
... although if it's bug #1, I'm not sure why it just showed up for me with btrfs and not ext4... On Tue, Jan 27, 2015 at 4:03 PM, Dan Kegel d...@kegel.com wrote: Thanks for the short repro script. This prevents debian packaging tools from working; seems like it'd be good to fix it in aufs rather than adding a workaround into dpkg and friends. I believe this is bug #1: http://sourceforge.net/p/aufs/bugs/1/ - Dan On Tue, Jan 27, 2015 at 3:47 PM, Michael Johnson - MJ m...@revmj.com wrote: I see very similar behavior (and always have with AUFS) and it is easily reproducible. Here is how to reproduce 100% of the time: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle To mke the error go away: $ cd . And now an ls works flawlessly. Basically, any time you have a directory with one or more files or directories inside of it, and then you recursively remove it while the working directory is the parent, subsequent attempts to access files inside of the parent will fail with the 'Stale file handle' error. For the most part this does not impact me becuase most things are not working of files/directories that are a immediate descent of the current working directory. But it would be nice if this problem did not occur. Hope this helps to at least understand the problem better and possible provide you with a workaround. On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel d...@kegel.com wrote: I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ -- Michael Johnson - MJ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle At least, these steps succeeded on my test machine. $ mkdir -p dir1/dir2 $ rm -ir dir1 rm: descend into directory `dir1'? y rm: remove directory `dir1/dir2'? y aufs au_new_inode:423:rm[4413]: Warning: Un-notified UDBA or repeatedly renamed dir, b0, btrfs, dir1, hi274, i11. rm: remove directory `dir1'? y $ ls b_dst empty f_src mv501_a/ p_src| sleep* ::: (shows everything) - latest aufs3.14 - u = rw + ro /dev/ram1 /run/shm/ro ext2 ro,relatime,errors=continue,user_xattr,acl 0 0 /dev/ram0 /run/shm/rw btrfs rw,relatime,space_cache 0 0 none /run/shm/u aufs rw,relatime,si=a8fd9d47e41c7918 0 0 An interesting thing is, a warning about UDBA was produced. Which means that by removing dir2, the something about dir1 was changed unexpectedly and aufs detects it (and produced a warning). But I am not sure this is related to your problem. Anyway it will be a good help for me to investigate the problem if you post these info. - /proc/mounts (instead of the output of mount(8)) - /sys/module/aufs/* - /sys/fs/aufs/* (if you have them) - /debug/aufs/* (if you have them) - linux kernel version if your kernel is not plain, for example modified by distributor, the url where i can download its source is necessary too. - aufs version which was printed at loading the module or booting the system, instead of the date you downloaded. - configuration (define/undefine CONFIG_AUFS_xxx) - kernel configuration or /proc/config.gz (if you have it) By the way, I have posted about some unusual behaviours of btrfs. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg02430.html Again I begin thinking btrfs is not usable still as aufs branch. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer Is this your Ubuntu 14.04? 2d22fc7 2014-10-09 UBUNTU: Ubuntu-3.13.0-38.65 According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses aufs3.13-20140303. As always, ubuntu uses old version enough to make it very hard for me suppport. But I am trying, as always, to support and help users. Anyway I use MJ's short steps to test this problem because it looks like same to yours. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses aufs3.13-20140303. As always, ubuntu uses old version enough to make it very hard for me suppport. But I am trying, as always, to support and help users. Ah, I might be confused, sorry. If ubuntu-trusty.git is release on Apr 2014, then aufs3.13-20140303 is not bad. It should be new at that time. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
