Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I think I know why you were unable to recreate the problem. It appears to > only occur in attempt the test case from a directory contained within the > aufs mount point, and not the base of the mount itself. In fact, if the > directory you are in exists in all branches, the problem does not occur. I > have verified that this problem does not occur when the underlying > filesystes are all ext4, but it does occur when they are all btrfs. Hmm... Still I cannot reproduce the problem. Linux jrofr 3.13.11.6aufsD #8 SMP Mon Feb 2 13:56:32 JST 2015 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro=rw none /u [ 51.057477] aufs 3.13-20140303 + sudo mkdir -p /u/dir1/dir2/dir3 + cd /u/dir1 + sudo rm -fr dir2 + ls (no error) > I can try building a new aufs module, but I will need some directions to > get that started. - get all source files of ubuntu trusty and build as it is $ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git $ cd ubuntu-trusty $ fakeroot make-kpkg --uc --us --config silentoldconfig configure $ fakeroot make-kpkg --uc --us --append_to_version -aufsD --revision `date +%y%m%d%H%M` kernel_image (lots of time and space are necessary) - install the built kernel pkg $ dpkg -i ../linux-image-deb - reboot - make sure you can receive "kern.debug" logs $ logger -p kern.debug testing check the kernel log - reproduce the problem, in order to make sure check the kernel log - enable debugging $ cd ubuntu-trusty $ make menuconfig enable CONFIG_AUFS_DEBUG (menu) "Ubuntu Supplied Third-Party Device Drivers" --> "Aufs (Advanced multi layered unification filesystem) support" --> "Debug aufs" - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem check the kernel log - apply the patch in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html - re-run "make-kpkg ... kernel_image" and re-install the pkg - reboot - reproduce the problem check the kernel log > If it would be helpful, I can figure out have to give you ssh access to > this VM if you want to provide me with an SSH public to add. Can I reboot the system and check the log anytime via ssh? If so, I will send my public key privately. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I think I know why you were unable to recreate the problem. It appears to only occur in attempt the test case from a directory contained within the aufs mount point, and not the base of the mount itself. In fact, if the directory you are in exists in all branches, the problem does not occur. I have verified that this problem does not occur when the underlying filesystes are all ext4, but it does occur when they are all btrfs. So to reproduce the problem, I do this: mount -t aufs -o br:/mnt/vol0=rw:/mnt/vol1=rw /z mkdir -p /z/foo/bar/baz cd /z/foo rm -rf bar ls The ls results in 'ls: cannot open directory .: Stale file handle' This is using a fresh install of Ubuntu 14.10 in a VM. I can try building a new aufs module, but I will need some directions to get that started. If it would be helpful, I can figure out have to give you ssh access to this VM if you want to provide me with an SSH public to add. On Mon, Feb 2, 2015 at 7:57 AM, <[1]sf...@users.sourceforge.net> wrote: Michael Johnson - MJ: > I will see if I can reproduce in a vm and generate very specific > instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in [2]http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.h tml J. R. Okajima -- Michael Johnson - MJ References 1. mailto:sf...@users.sourceforge.net 2. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: > I will see if I can reproduce in a vm and generate very specific > instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. On Feb 1, 2015 10:37 PM, <[1]sf...@users.sourceforge.net> wrote: [2]sf...@users.sourceforge.net: > I have tried but could not reproduce the problem. > - got full trusty kernel source, built it with the default configuration >  (as MJ posted) --> no disk space > - disabled unnecessary drivers, built again --> succeeded > - rebooted with the new kernel, tried, but could not reproduce the >  problem > > Now I am preparing a parition to install full ubuntu trusty by > debootstrap. I am going to make btrfs as its root partition. Do I need > some options for btrfs? Is it ok to mkfs.btrfs without any options > simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs  # mkdir rw ro u  # sudo mount -t aufs -o br=/rw:/ro none /u  # cd /u  # sudo touch fileA  # sudo mkdir -p dir1/dir2  # sudo rm -fr dir1  # ls  and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module  which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by  "ls -l $(modinfo -F filename aufs)" and see the timestamp. - make sure that you can receive the "kern.debug" logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just  before rm, run "echo 1 > /sys/module/aufs/parameters/debug". and just  after ls, run "echo 0 > /sys/module/aufs/parameters/debug".  Setting 1 enables aufs debugging feature and produces many kernel  debug log. To stop the debug log, set 0 to  /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. mailto:sf...@users.sourceforge.net -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
