Re: Stale NFS file handle - latency issue? - SOLVED (so I hope...)
Hello, J. R. Okajima List I think I have encircled the issue: The Problem disappears when I boot the server into Kernel 3.2 instead of 3.16. So your indication given below was right. It does not disappear when I change nfs export options from sync to async, as mentioned in some (older) internet threads. I'll try to file a bug report with debian, I think. Wolfgang Rosner Am Montag, 9. Februar 2015 18:26:14 schrieben Sie: it's all on a single ext4, ordinary consumer type SATA HD root@.# mount . /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered) On my debian-3.16.7-ckt2-1~bpo70+1 kernel, I can see ext4 via nfs4 blocked a long time. It blocked at open(O_TRUNC) and sleep_on_page(). While I don't investigate deeper yet, this version of ext4 or io-scheduler is suspcious. But what I saw might be different from your problem. If you can, try another version of kernel or ext4. For example, - another debian kernel - use ext2 instead of ext4 I will try other tests more. -- -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle - latency issue?
Dear J. R. Okajima Am Samstag, 7. Februar 2015 03:14:25 schrieben Sie: Wolfgang Rosner: Do you mean - you specified =ro+wh as a branch permission - but /sys/fs/aufs/si_*/br2 shows ro right? yes. Thats what I did. I cannot reproduce this problem. I think this was a bug in aufs, and already fixed by these commits. 7a39204 2014-10-16 aufs: bugfix, fix the returning size of the branch attr 5001e71 2014-06-25 aufs: simply handing attribute string f7292d3 2014-06-25 aufs: introduce au_br_perm_str_t If you can, try updating aufs. J. R. Okajima I'm afraid I cannot avoid to drop into the build process. What I did now: Thanks for the valuable hint to wireshark. As an old-console-day-guy I usually am reluctant to use GUI-tools. But the interactive filtering and deep protocol specific analysis really helps against the feeling of sitting in the crankcase and watch the pistons going up and down I remember from tcpdump. wiresharking on the server, I could pin down the moment when things hang: It is when gzip is run on /var/lib/dmesg.0 Before, I see older log files being renamed. I can watch success of these operation in both the aufs branches and the aufs mount. The last signs of life of my NFS client look like 41090 122.518015 192.168.130.2 192.168.130.250 NFS 3146V4 Call (Reply In 41427) WRITE StateID: 0xdcdd Offset: 0 Len: 17341 - 41427 122.543403 192.168.130.250 192.168.130.2 NFS 202 V4 Reply (Call In 41090) WRITE Status: NFS4_OK (0) 41565 122.549393 192.168.130.2 192.168.130.250 NFS 270 V4 Call (Reply In 42087) CLOSE StateID: 0xdcdd ... 41589 122.549842 192.168.130.2 192.168.130.250 TCP 66 781→2049 [ACK] Seq=637669 Ack=19538241 Win=949888 Len=0 TSval=4294896938 TSecr=474615 42086 122.564122 192.168.130.2 192.168.130.250 TCP 66 781→2049 [ACK] Seq=637669 Ack=21219681 Win=949888 Len=0 TSval=4294896942 TSecr=474618 42087 122.564131 192.168.130.250 192.168.130.2 NFS 47382 V4 Reply (Call In 41559) READ 42179 122.603758 192.168.130.2 192.168.130.250 TCP 66 781→2049 [ACK] Seq=637669 Ack=21531981 Win=949888 Len=0 TSval=4294896952 TSecr=474618 (the dots are only TCP fragments, which I think are related to large file transfer) Thats it. After that, I only read DHCP renewal and TCP keepalives at sparse intervals. I can ping but not login nor do anything - neither at the console nor by ssh. I think the client NFS has gone and took the root mount with it. I don't know whether it's a pure NFS problem or the NFS client is insulted by some misbehaviour of aufs. So, the plan was to capture both network interfaces, rpcdebug -m nfs on both sides , aufs debug and strace /usr/bin/savelog (which is calling gzip) Already attached a HD on one of the client machines, since neiter echoing all trace back over the network nor storing it into ramdisk which is gone after system hags seems a good idea to me. I see, I cannot avoid to drop into the build process - to enable debugging - to exclude errors already fixed - to solve the rowh permission display problem (although this might be a minor issue) I haven't build a kernel for a decade or so, and back in good-old-SuSE-times things (distribution specific patching) worked quite different than in debian now. I'm afraid, this will cost me some other days to get it working. I was quite glad to read in http://aufs.sourceforge.net/README.aufs2 of the module only method, and equally disappoited to figure out that in aufs3, this obviusly has changed again :-( Nevertheless, can I avoid complete kernel rebuild, when I build a module for the 3.16 kernel, where aufs still is included? below is what ships with debian (testing) Is there a way to build aufs module without going through the whole kernel rebuild, starting from there? Thank You Wolfgang Rosner == ...# uname -v #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) ...$ grep AUFS /boot/config-3.16.0-4-amd64 CONFIG_AUFS_FS=m CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set CONFIG_AUFS_EXPORT=y CONFIG_AUFS_INO_T_64=y # CONFIG_AUFS_FHSM is not set # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set CONFIG_AUFS_BR_HFSPLUS=y CONFIG_AUFS_BDEV_LOOP=y # CONFIG_AUFS_DEBUG is not set # modinfo aufs filename: /lib/modules/3.16.0-4-amd64/kernel/fs/aufs/aufs.ko staging:Y alias: fs-aufs version:3.16-20140908 description:aufs -- Advanced multi layered unification filesystem author: Junjiro R. Okajima aufs-users@lists.sourceforge.net license:
Re: Stale NFS file handle - latency issue?
Wolfgang Rosner: The last signs of life of my NFS client look like ::: It shows these sequences. - nfs client sent WRITE + nfs server returned OK - nfs client sent CLOSE + nfs server didn't return anything - nfs client sent READ + nfs server didn't return anything According to the protocol of nfsv4, the server should return OK or error for CLOSE, which looks like the beginning of this problem. Still I don't know why, but I agree aufs is the first candidate of this problem. I haven't build a kernel for a decade or so, and back in good-old-SuSE-tim= es=20 things (distribution specific patching) worked quite different than in debi= an=20 now. I'm afraid, this will cost me some other days to get it working. Recently I've post the steps how to re-build and re-install ubuntu kernel. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg05005.html As you might know, the commands on ubuntu is very similar to debian, so these steps will help you hopefully. And people who followed this sequence posted about the errors in it, so you should read the follow-ups too. And what is the fs-type of your branches? - /cluster/nfs/nfsroot/wheezy_cow/cow_* - /cluster/nfs/nfsroot/wheezy_root_config - /cluster/nfs/nfsroot/wheezy_root_mask - /cluster/nfs/nfsroot/wheezy J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Sorry it's been a bit since I reported back, but I got a chance to apply your patch and I have mixed results to report. Basically, it looks like the patch may resolve the issues, but not entirely. After building the new kernel, I was still able to reproduce the problem initially in a sub directory, but not on the top level. When it was happening, this is what showed up in the logs: Feb  8 15:09:57 aufs kernel: [  998.810169] [ cut here ] Feb  8 15:09:57 aufs kernel: [  998.810177] WARNING: CPU: 0 PID: 2076 at /build/buildd/linux-3.16.0/fs/inode.c:282 drop_nlink+0x41/0x50() Feb  8 15:09:57 aufs kernel: [  998.810178] Modules linked in: aufs serio_raw snd_hda_codec_generic kvm_amd kvm qxl crct10dif_pclmul ttm drm_kms_helper crc32_pclmul ghash_clmulni_intel aesni_intel drm aes_x86_64 lrw ppdev snd_hda_intel snd_hda_controller i2c_piix4 snd_hda_codec gf128mul glue_helper snd_hwdep snd_pcm snd_timer ablk_helper cryptd snd parport_pc soundcore pvpanic parport mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  8 15:09:57 aufs kernel: [  998.810200] CPU: 0 PID: 2076 Comm: rm Tainted: G     W   3.16.0-29-generic #39-Ubuntu Feb  8 15:09:57 aufs kernel: [  998.810202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  8 15:09:57 aufs kernel: [  998.810204]  0009 88003c7f3cc8 8178218a Feb  8 15:09:57 aufs kernel: [  998.810206]  88003c7f3d00 8106fedd 88003bc1a658 88003bc1a658 Feb  8 15:09:57 aufs kernel: [  998.810208]  0001 88003cbcede0 880039d91c00 88003c7f3d10 Feb  8 15:09:57 aufs kernel: [  998.810210] Call Trace: Feb  8 15:09:57 aufs kernel: [  998.810216]  [8178218a] dump_stack+0x45/0x56 Feb  8 15:09:57 aufs kernel: [  998.810219]  [8106fedd] warn_slowpath_common+0x7d/0xa0 Feb  8 15:09:57 aufs kernel: [  998.810221]  [8106ffba] warn_slowpath_null+0x1a/0x20 Feb  8 15:09:57 aufs kernel: [  998.810224]  [811fcc31] drop_nlink+0x41/0x50 Feb  8 15:09:57 aufs kernel: [  998.810231]  [c03d6c1d] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810236]  [c03d5e5e] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810241]  [c03e52e7] aufs_rmdir+0x2b7/0x430 [aufs] Feb  8 15:09:57 aufs kernel: [  998.810244]  [811f8390] ? prepend.constprop.25+0x30/0x30 Feb  8 15:09:57 aufs kernel: [  998.810246]  [811eede7] vfs_rmdir+0xa7/0x100 Feb  8 15:09:57 aufs kernel: [  998.810248]  [811f20b9] do_rmdir+0x1d9/0x1f0 Feb  8 15:09:57 aufs kernel: [  998.810250]  [811e2dbe] ? fput+0xe/0x10 Feb  8 15:09:57 aufs kernel: [  998.810253]  [81091afc] ? task_work_run+0xbc/0xf0 Feb  8 15:09:57 aufs kernel: [  998.810256]  [810131e7] ? do_notify_resume+0x97/0xb0 Feb  8 15:09:57 aufs kernel: [  998.810258]  [811f3365] SyS_unlinkat+0x25/0x40 Feb  8 15:09:57 aufs kernel: [  998.810261]  [8178a1ad] system_call_fastpath+0x1a/0x1f Feb  8 15:09:57 aufs kernel: [  998.810263] ---[ end trace 7e60ac9f449b2a85 ]--- Once I saw that it was still happening, I attempted to clean things up and present a more complete example of how I was getting this to happen. I unmounted the aufs filesystem and removed the contents of the underlying filesystems (including the .wh* bits) and remounted. Once I saw what the aufs mountpoint was correctly reflecting that everything was removed, I tried to reproduce the error again, and I have not been able to. I can only assume that something with the old .wh* files was causing some sort of issue. Ultimately, at this time I cannot reproduce the problem, so I guess we call it fixed? On Wed, Feb 4, 2015 at 5:41 AM, [1]sf...@users.sourceforge.net wrote: Michael Johnson - MJ: I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs.  Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related:     ::: Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at
Re: Stale NFS file handle - latency issue?
Hello Wolfgang, Wolfgang Rosner: short story before aufs on server, nfs export as nfsroot for clients the symptom on the client (first login after boot): root@blade-008:~# la ls: cannot open directory .: Stale NFS file handle root@blade-008:~# cd . root@blade-008:~# la ::: I'd rather suspect a latency problem, since aufs mount and nfs eports are generated within the same perl scripts. May it be that aufs mount is forked from the script and not completed, before nfs export is called? How can I find out? how can I avoid? Other explanations? Is the client correctly booted? And prompted for user login? If so, I don't think the problem shoule not be latency you call, because the booted client is a strong evidence of that aufs is mounted correctly and is exported correctly. The mount procedure should be done synchronously and nfsd should be able to use it just after the completion of mount. But it is totally up to your mount(8) command. Is it ordinary one from linux-utils? If ESTALE happend in the very early stage of mounting nfsroot, then your guess (latency problem) might be possible. As a first step, I'd suggest you to see what was done between the completion of system boot and ls (where ESTALE happened). In other words, is there something unusual around getty, login or ~user/.profile? This is a story on your nfs-client. And, just to make sure, the story is same to your nfs-server. Is there something unusual on your nfs-server after the completion of client boot? To investigate more, you need to find out which systemcall and which module returns ESTALE. The candidates are - open(2) in ls(1) - nfs client - nfs server - aufs on the server - branch fs in aufs The debugging method for those will be - strace - wireshark - aufs module parameter debug With these tools and feature, you will see the behaviour of these modules. - layer 2 is masking among others /root/some.files and /var/log it's mounted as ro+wh (not visible in /sys/fs/aufs/) Do you mean - you specified =ro+wh as a branch permission - but /sys/fs/aufs/si_*/br2 shows ro right? If so, there is something wrong. But I don't know it is related to your latency problem. root@cruncher:/cluster/etc/scripts/available# exportfs -v /cluster/mp/nfsr/aufs_008192.168.130.8 (rw,wdelay,crossmnt,no_root_squash,no_subtree_check,fsid=158,sec=sys,rw,no_root_squash,no_all_squash) Are you setting fsid for each client with different values? Such as - fsid=158 for /cluster/mp/nfsr/aufs_008 - fsid=159 for /cluster/mp/nfsr/aufs_009 ::: J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs.  Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; The end result is always a stale file handle. If you 'cd .' following this set of commands the error clears, unless you are in the top level of the aufs filesystem. If you are in the top level, you must umount/mount the aufs file system to recover. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related: Feb  3 23:54:38 aufs kernel: [ 3577.351197] [ cut here ] Feb  3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at /build/buildd/linux-3.16.0/fs /inode.c:282 drop_nlink+0x41/0x50() Feb  3 23:54:38 aufs kernel: [ 3577.351208] Modules linked in: aufs snd_hda_codec_generic kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller snd_hda_codec qxl snd_hwdep snd_pcm ttm drm_kms_helper snd_timer serio_raw drm snd soundcore parport_pc pvpanic parport i2c_piix4 mac_hid btrfs xor raid6_pq psmouse floppy pata_acpi Feb  3 23:54:38 aufs kernel: [ 3577.351233] CPU: 0 PID: 3725 Comm: rm Tainted: G     W   3.16.0-23-generic #31-Ubuntu Feb  3 23:54:38 aufs kernel: [ 3577.351234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014 Feb  3 23:54:38 aufs kernel: [ 3577.351236]  0009 88003a07fcc8 8177fcbc Feb  3 23:54:38 aufs kernel: [ 3577.351238]  88003a07fd00 8106fd8d 88003a742058 88003a742058 Feb  3 23:54:38 aufs kernel: [ 3577.351240]  88003b3ac240 88003b92b100 88003a07fd10 Feb  3 23:54:38 aufs kernel: [ 3577.351251] Call Trace: Feb  3 23:54:38 aufs kernel: [ 3577.351257]  [8177fcbc] dump_stack+0x45/0x56 Feb  3 23:54:38 aufs kernel: [ 3577.351261]  [8106fd8d] warn_slowpath_common+0x7d/0xa0 Feb  3 23:54:38 aufs kernel: [ 3577.351263]  [8106fe6a] warn_slowpath_null+0x1a/0x20 Feb  3 23:54:38 aufs kernel: [ 3577.351265]  [811fc5f1] drop_nlink+0x41/0x50 Feb  3 23:54:38 aufs kernel: [ 3577.351273]  [c0510c1d] au_whtmp_rmdir+0x19d/0x1b0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351278]  [c050fe5e] ? au_whtmp_ren+0x6e/0xe0 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351283]  [c051f2e7] aufs_rmdir+0x2b7/0x430 [aufs] Feb  3 23:54:38 aufs kernel: [ 3577.351285]  [811f7d50] ? prepend.constprop.25+0x30/0x30 Feb  3 23:54:38 aufs kernel: [ 3577.351288]  [811eef57] vfs_rmdir+0xa7/0x100 Feb  3 23:54:38 aufs kernel: [ 3577.351290]  [811f1a79] do_rmdir+0x1d9/0x1f0 Feb  3 23:54:38 aufs kernel: [ 3577.351293]  [811e285e] ? fput+0xe/0x10 Feb  3 23:54:38 aufs kernel: [ 3577.351295]  [810919ac] ? task_work_run+0xbc/0xf0 Feb  3 23:54:38 aufs kernel: [ 3577.351299]  [810130f7] ? do_notify_resume+0x97/0xb0 Feb  3 23:54:38 aufs kernel: [ 3577.351301]  [811f2d25] SyS_unlinkat+0x25/0x40 Feb  3 23:54:38 aufs kernel: [ 3577.351304]  [81787ced] system_call_fastpath+0x1a/0x1f Feb  3 23:54:38 aufs kernel: [ 3577.351391] ---[ end trace d717630435aab60d ]--- I could not get this to occur on a regular basis, but it popped up at some point. Hopefully that is of some help. Here are all the aufs config options for my current build: CONFIG_AUFS_FS=m CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set CONFIG_AUFS_EXPORT=y CONFIG_AUFS_INO_T_64=y # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SP_IATTR is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set # CONFIG_AUFS_BR_HFSPLUS is not set CONFIG_AUFS_BDEV_LOOP=y CONFIG_AUFS_DEBUG=y CONFIG_AUFS_MAGIC_SYSRQ=y I have not yet applied the patch, but I will give it a try tomorrow to see if it makes a difference in behavior or log messages. On Tue, Feb 3, 2015 at 8:06 PM, [1]sf...@users.sourceforge.net wrote: MJ, Dan, I appriciate your efforts. Dan Kegel: That failed with no config found, so I repeated it after doing $
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: I've made it through the initial build and the problem does in fact still occur. I completed the build with CONFIG_AUFS_DEBUG=y and as one would expect, the problem still occurs. Interestingly, I cannot reproduce 100% of the time in the way previously described. Here is what does work to reproduce for me 100% of the time currently: mkdir -p a/b; cd .; rm -rf a; ls; Still I cannot reproduce... But I will try more. For the most part I did not get anything in my kernel debug logs, but after hammering on it a bit I did get the following in my logs which appears to be related: ::: Feb 3 23:54:38 aufs kernel: [ 3577.351206] WARNING: CPU: 0 PID: 3725 at /build/buildd/linux-3.16.0/fs /inode.c:282 drop_nlink+0x41/0x50() It is one of good news. It is caused by a strange link count of a dir on btrfs. At least this problem will be solved by the patch I sent previously. But we may meet another problem later. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. On Feb 1, 2015 10:37 PM, [1]sf...@users.sourceforge.net wrote: [2]sf...@users.sourceforge.net: I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration   (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the   problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs  # mkdir rw ro u  # sudo mount -t aufs -o br=/rw:/ro none /u  # cd /u  # sudo touch fileA  # sudo mkdir -p dir1/dir2  # sudo rm -fr dir1  # ls  and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module  which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by  ls -l $(modinfo -F filename aufs) and see the timestamp. - make sure that you can receive the kern.debug logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just  before rm, run echo 1 /sys/module/aufs/parameters/debug. and just  after ls, run echo 0 /sys/module/aufs/parameters/debug.  Setting 1 enables aufs debugging feature and produces many kernel  debug log. To stop the debug log, set 0 to  /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. mailto:sf...@users.sourceforge.net -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: I will see if I can reproduce in a vm and generate very specific instructions to reproduce. I think I will have time to do this later today. Thank you very much. Other testers are also welcome. The first debug patch I will ask will be the one in http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04991.html J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? Totally I cannot reproduce the problem. I have to give up it on my side. Linux jrofr 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux /dev/sda15 / btrfs rw,relatime,space_cache 0 0 + sudo mount -t aufs -o br=/rw:/ro none /u + cd /u + sudo touch fileA + sudo mkdir -p dir1/dir2/dir3 + sudo rm -fr dir1 + ls fileA Anyone who can reproduce the problem, could you try these? - use ubuntu trusty - use btrfs and aufs # mkdir rw ro u # sudo mount -t aufs -o br=/rw:/ro none /u # cd /u # sudo touch fileA # sudo mkdir -p dir1/dir2 # sudo rm -fr dir1 # ls and see ESTALE happens or not. If you got ESTALE, then I'd ask you to - check the kernel log - enable CONFIG_AUFS_DEBUG and rebuild aufs module which eats lots of disk spaces, about 10GB, and takes a long time. - make sure that the module is correctly installed by ls -l $(modinfo -F filename aufs) and see the timestamp. - make sure that you can receive the kern.debug logs - repeat the same commands, cd + mkdir + rm + ls. But this time, just before rm, run echo 1 /sys/module/aufs/parameters/debug. and just after ls, run echo 0 /sys/module/aufs/parameters/debug. Setting 1 enables aufs debugging feature and produces many kernel debug log. To stop the debug log, set 0 to /sys/module/aufs/parameters/debug. - show me the debug log. Maybe I will ask you to apply a new debug patch and repeat these steps several time. Anyone, please cooperate this test. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
sf...@users.sourceforge.net: Anyway I will try ubuntu kernel by myself, although it will take long time... I have tried but could not reproduce the problem. - got full trusty kernel source, built it with the default configuration (as MJ posted) -- no disk space - disabled unnecessary drivers, built again -- succeeded - rebooted with the new kernel, tried, but could not reproduce the problem Now I am preparing a parition to install full ubuntu trusty by debootstrap. I am going to make btrfs as its root partition. Do I need some options for btrfs? Is it ok to mkfs.btrfs without any options simply? J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# mkdir -p dir1/dir2 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# rm -rf dir1 root@ubu14-bb-01:/var/lib/lxc/ubu14-bb-01-ubu1204-temp-g-speak-unique/foobar# ls ls: cannot open directory .: Stale file handle Hmm, I don't understand why I cannot reproduce... I may be able to try building the Ubuntu kernel, but I'm a bit slammed now, not sure when I can try it. But if your patch works it'd save us some effort. Ok, when you can, please try the attached patch. Here's the other info you requested: Ok, thanx. I understand your branches and options. You don't have compress=zlib. Oh but you have noplink. Anyway I will try ubuntu kernel by myself, although it will take long time... J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
I see very similar behavior (and always have with AUFS) and it is easily reproducible. Here is how to reproduce 100% of the time: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle To mke the error go away: $ cd . And now an ls works flawlessly. Basically, any time you have a directory with one or more files or directories inside of it, and then you recursively remove it while the working directory is the parent, subsequent attempts to access files inside of the parent will fail with the 'Stale file handle' error. For the most part this does not impact me becuase most things are not working of files/directories that are a immediate descent of the current working directory. But it would be nice if this problem did not occur. Hope this helps to at least understand the problem better and possible provide you with a workaround. On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel [1]d...@kegel.com wrote: I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: [2]http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per [3]http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. [4]http://goparallel.sourceforge.net/ -- Michael Johnson - MJ References 1. mailto:d...@kegel.com 2. http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 3. http://aufs.sourceforge.net/ 4. http://goparallel.sourceforge.net/ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
... although if it's bug #1, I'm not sure why it just showed up for me with btrfs and not ext4... On Tue, Jan 27, 2015 at 4:03 PM, Dan Kegel d...@kegel.com wrote: Thanks for the short repro script. This prevents debian packaging tools from working; seems like it'd be good to fix it in aufs rather than adding a workaround into dpkg and friends. I believe this is bug #1: http://sourceforge.net/p/aufs/bugs/1/ - Dan On Tue, Jan 27, 2015 at 3:47 PM, Michael Johnson - MJ m...@revmj.com wrote: I see very similar behavior (and always have with AUFS) and it is easily reproducible. Here is how to reproduce 100% of the time: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle To mke the error go away: $ cd . And now an ls works flawlessly. Basically, any time you have a directory with one or more files or directories inside of it, and then you recursively remove it while the working directory is the parent, subsequent attempts to access files inside of the parent will fail with the 'Stale file handle' error. For the most part this does not impact me becuase most things are not working of files/directories that are a immediate descent of the current working directory. But it would be nice if this problem did not occur. Hope this helps to at least understand the problem better and possible provide you with a workaround. On Tue, Jan 27, 2015 at 10:50 AM, Dan Kegel d...@kegel.com wrote: I've been using aufs on top of ext4 reliably for some time. Recently I tried it on top of btrfs (stock Ubuntu 14.04), and my builds are failing in rm -rf with: /bin/rm: cannot remove `/tmp/temp-lintian-lab-h2hbg82XOW/pool/o/oobleck.../unpacked': Directory not empty internal error: failed to remove unpacked directory of oobleck ... /bin/rm: fts_read failed: Stale NFS file handle Has anybody else seen this? (A superficially similar problem was mentioned five years ago: http://comments.gmane.org/gmane.linux.file-systems.aufs.user/2437 Likewise, there are superficially similar posts from docker users, but they may be from user error.) I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer supports 3.13 per http://aufs.sourceforge.net/ Not sure how easy it'd be for me to run vanilla kernels on this box. My best move is probably to drop back to ext4 for now. -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ -- Michael Johnson - MJ -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Michael Johnson - MJ: $ cd /aufsdir; $ mkdir -p dir1/dir2; $ rm -rf dir1 $ ls ls: cannot open directory .: Stale file handle At least, these steps succeeded on my test machine. $ mkdir -p dir1/dir2 $ rm -ir dir1 rm: descend into directory `dir1'? y rm: remove directory `dir1/dir2'? y aufs au_new_inode:423:rm[4413]: Warning: Un-notified UDBA or repeatedly renamed dir, b0, btrfs, dir1, hi274, i11. rm: remove directory `dir1'? y $ ls b_dst empty f_src mv501_a/ p_src| sleep* ::: (shows everything) - latest aufs3.14 - u = rw + ro /dev/ram1 /run/shm/ro ext2 ro,relatime,errors=continue,user_xattr,acl 0 0 /dev/ram0 /run/shm/rw btrfs rw,relatime,space_cache 0 0 none /run/shm/u aufs rw,relatime,si=a8fd9d47e41c7918 0 0 An interesting thing is, a warning about UDBA was produced. Which means that by removing dir2, the something about dir1 was changed unexpectedly and aufs detects it (and produced a warning). But I am not sure this is related to your problem. Anyway it will be a good help for me to investigate the problem if you post these info. - /proc/mounts (instead of the output of mount(8)) - /sys/module/aufs/* - /sys/fs/aufs/* (if you have them) - /debug/aufs/* (if you have them) - linux kernel version if your kernel is not plain, for example modified by distributor, the url where i can download its source is necessary too. - aufs version which was printed at loading the module or booting the system, instead of the date you downloaded. - configuration (define/undefine CONFIG_AUFS_xxx) - kernel configuration or /proc/config.gz (if you have it) By the way, I have posted about some unusual behaviours of btrfs. http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg02430.html Again I begin thinking btrfs is not usable still as aufs branch. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
Dan Kegel: I see that Ubuntu 14.04's using kernel 3.13.mumble, and aufs no longer Is this your Ubuntu 14.04? 2d22fc7 2014-10-09 UBUNTU: Ubuntu-3.13.0-38.65 According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses aufs3.13-20140303. As always, ubuntu uses old version enough to make it very hard for me suppport. But I am trying, as always, to support and help users. Anyway I use MJ's short steps to test this problem because it looks like same to yours. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle when using aufs on top of btrfs?
According to git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git, it uses aufs3.13-20140303. As always, ubuntu uses old version enough to make it very hard for me suppport. But I am trying, as always, to support and help users. Ah, I might be confused, sorry. If ubuntu-trusty.git is release on Apr 2014, then aufs3.13-20140303 is not bad. It should be new at that time. J. R. Okajima -- Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
Re: Stale NFS file handle
umount -f -l ro is the closest you will get. The entry will be removed from the mount table and the actual unmount will happen when it can (never) but at least you needn't worry about it. On 4/6/13, V.Krishn vkris...@gmail.com wrote: Did the following to set an aufs. cd /tmp mkdir branch ro u mount test.iso ro -o loop mount -t aufs -o br:branch:ro none u Now branch got accidently deleted and I am unable to unmount u nor ro Tried: umount -f u or umount -f ro Keep getting following error: umount2: Stale NFS file handle umount: u/: Stale NFS file handle I am using Knoppix 7.0.1. Any help would be great. -- Regards. V.Krishn -- Minimize network downtime and maximize team effectiveness. Reduce network management and security costs.Learn how to hire the most talented Cisco Certified professionals. Visit the Employer Resources Portal http://www.cisco.com/web/learning/employer_resources/index.html -- Minimize network downtime and maximize team effectiveness. Reduce network management and security costs.Learn how to hire the most talented Cisco Certified professionals. Visit the Employer Resources Portal http://www.cisco.com/web/learning/employer_resources/index.html
Re: Stale NFS file handle
On Saturday, April 06, 2013 05:53:33 PM you wrote: umount -f -l ro is the closest you will get. The entry will be removed from the mount table and the actual unmount will happen when it can (never) but at least you needn't worry about it. Thanks, this does seems to solve the ro dir (still to check if it frees the .iso file) u dir is still the same. ls show the following: d? ? ? ? ?? u I tried simulating the issue on another distro with same result (just to be sure that this is not a Knoppix issue). -- Regards. V.Krishn On 4/6/13, V.Krishn vkris...@gmail.com wrote: Did the following to set an aufs. cd /tmp mkdir branch ro u mount test.iso ro -o loop mount -t aufs -o br:branch:ro none u Now branch got accidently deleted and I am unable to unmount u nor ro Tried: umount -f u or umount -f ro Keep getting following error: umount2: Stale NFS file handle umount: u/: Stale NFS file handle I am using Knoppix 7.0.1. Any help would be great. -- Regards. V.Krishn - - Minimize network downtime and maximize team effectiveness. Reduce network management and security costs.Learn how to hire the most talented Cisco Certified professionals. Visit the Employer Resources Portal http://www.cisco.com/web/learning/employer_resources/index.html -- Minimize network downtime and maximize team effectiveness. Reduce network management and security costs.Learn how to hire the most talented Cisco Certified professionals. Visit the Employer Resources Portal http://www.cisco.com/web/learning/employer_resources/index.html
Re: Stale NFS file handle
Lucas de Vries: I tried to compile something in my aufs directory (I've tried it with multiple applications, so it's not something specific to that), and during the configure stage it just dies with: ls: cannot access .: Stale NFS file handle I tried btrfs. As I expected, the aufs XINO files don't work on btrfs due to its internal i_blocks. It is OK. When aufs detects that the first writable branch fs cannot handle XINO files, it will try /tmp as next candidate. Here is a patch attached for you to support btrfs branch. But I found some weird behaviours on btrfs, just for your information. - sparse file may not be supported. - df -i (the number of inodes) always shows 0. - repeated ln(1) retuns an error Too many links, but succeeds after a short pause. You may reproduce this by this simple script. #!/bin/sh l10=0123456789 l100=${l10}${l10}${l10}${l10}${l10}${l10}${l10}${l10}${l10}${l10} a o=1000 n=$o while [ $n -gt 0 ] do ln a $l100$n n=$((${n}-1)) done - After making btrfs full (by dd if=/dev/zero of=btrfs/file, and it returns No space left on device expectedly), df shows many space. It means that aufs mfs (and its variants) will not work correctly. But I wonder btrfs has some nice features as garbage collection since what I used /dev/zero. - racer.sh test (a fs heavy i/o test kit, you can google and find it) didn't pass even I tried without aufs. btrfs potentially deadlocks. = [ INFO: possible irq lock inversion dependency detected ] 2.6.32.7aufsD #664 - file_rename.sh/21624 just changed the state of lock: (fs_info-trans_mutex){+.+.-.}, at: [a013af43] start_transaction+0x43/0x190 [btrfs] but this lock took another, RECLAIM_FS-READ-unsafe lock in the past: (found-groups_sem){.+} and interrupts could create inverse lock ordering between them. other info that might help us debug this: 3 locks held by file_rename.sh/21624: #0: (mm-mmap_sem){++}, at: [81450911] do_page_fault+0x301/0x5c0 #1: (shrinker_rwsem){..}, at: [810d7212] shrink_slab+0x32/0x200 #2: (type-s_umount_key#37){++}, at: [8111f920] shrink_dcache_memory+0x120/0x1e0 the shortest dependencies between 2nd lock and 1st lock: - (found-groups_sem){.+} ops: 0 { HARDIRQ-ON-W at: [81087f05] __lock_acquire+0x6e5/0x1600 [81088f30] lock_acquire+0x110/0x150 [8144bdaf] down_write+0x3f/0x50 [a012c261] btrfs_read_block_groups+0x471/0x600 [btrfs] [a01397b3] open_ctree+0x1483/0x1800 [btrfs] ::: snip ::: stack backtrace: Pid: 21624, comm: file_rename.sh Not tainted 2.6.32.7aufsD #664 Call Trace: [8108637c] print_irq_inversion_bug+0x14c/0x170 [81083790] ? usage_match+0x0/0x20 [810865b0] ? check_usage_forwards+0x0/0x110 [81086646] check_usage_forwards+0x96/0x110 [81087021] mark_lock+0x2d1/0x410 [81087ca9] __lock_acquire+0x489/0x1600 [81013394] ? restore_args+0x0/0x30 [810874ed] ? trace_hardirqs_on_caller+0x14d/0x1a0 [8144cabe] ? trace_hardirqs_on_thunk+0x3a/0x3f [81088f30] lock_acquire+0x110/0x150 [a013af43] ? start_transaction+0x43/0x190 [btrfs] [8144b3c9] __mutex_lock_common+0x59/0x620 [a013af43] ? start_transaction+0x43/0x190 [btrfs] [a013af43] ? start_transaction+0x43/0x190 [btrfs] [a013af2b] ? start_transaction+0x2b/0x190 [btrfs] [8144ba6c] mutex_lock_nested+0x3c/0x50 [a0143bd0] ? btrfs_delete_inode+0x0/0x130 [btrfs] [a013af43] start_transaction+0x43/0x190 [btrfs] [a0143bd0] ? btrfs_delete_inode+0x0/0x130 [btrfs] [a013b0ae] btrfs_join_transaction+0xe/0x10 [btrfs] [a0143c7f] btrfs_delete_inode+0xaf/0x130 [btrfs] [a0143bd0] ? btrfs_delete_inode+0x0/0x130 [btrfs] [81123666] generic_delete_inode+0xc6/0x190 [81227f88] ? _atomic_dec_and_lock+0x98/0xc0 [8112378d] generic_drop_inode+0x5d/0x80 [a013c161] btrfs_drop_inode+0x21/0x40 [btrfs] [81122668] iput+0x78/0x80 [8111f2d8] dentry_iput+0x98/0x110 [8111f46e] d_kill+0x2e/0x60 [8111f76d] __shrink_dcache_sb+0x2cd/0x360 [8111f95b] shrink_dcache_memory+0x15b/0x1e0 [810d7382] shrink_slab+0x1a2/0x200 [810d9b6e] try_to_free_pages+0x28e/0x3d0 [810d6550] ? isolate_pages_global+0x0/0x280 [810d151a] __alloc_pages_nodemask+0x59a/0x9c0 [810ea96b] do_wp_page+0x59b/0x900 [8123779c] ? _raw_spin_lock+0xac/0x1b0 [810eba4c] handle_mm_fault+0x84c/0x9d0 [81450911] ? do_page_fault+0x301/0x5c0 [81450ad7] do_page_fault+0x4c7/0x5c0 [8144cafd] ?
Re: Stale NFS file handle
I tried btrfs. As I expected, the aufs XINO files don't work on btrfs due to its internal i_blocks. It is OK. When aufs detects that the first writable branch fs cannot handle XINO files, it will try /tmp as next candidate. Here is a patch attached for you to support btrfs branch. The attached file was empty, sorry. J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Re: Stale NFS file handle
Hello Lucas, Lucas de Vries: I tried to compile something in my aufs directory (I've tried it with multiple applications, so it's not something specific to that), and during the configure stage it just dies with: ls: cannot access .: Stale NFS file handle Give me these info please. (from the aufs README) -- - /proc/mounts (instead of the output of mount(8)) - /sys/module/aufs/* - /sys/fs/aufs/* (if you have them) - /debug/aufs/* (if you have them) - linux kernel version if your kernel is not plain, for example modified by distributor, the url where i can download its source is necessary too. - aufs version which was printed at loading the module or booting the system, instead of the date you downloaded. - configuration (define/undefine CONFIG_AUFS_xxx) - kernel configuration or /proc/config.gz (if you have it) - behaviour which you think to be incorrect - actual operation, reproducible one is better - mailto: aufs-users at lists.sourceforge.net -- I'm mounting three btrfs partitions with the following fstab line: data /data aufs br:/mnt/data-5=rw:/mnt/data-4=rw:/mnt/data-3=rw,create=pmfs,noatime,noauto 0 0 I have never tried btrfs. I will try in a few days once you give me the above info. I have not been able to reproduce the error with manual ls commands, it only ever occurs during compilation. Then I'd suggest you to try strace -f -o /tmp/s compile. And send me whole /tmp/s. J. R. Okajima -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Re: Stale NFS file handle (wtf?)
Nothing interesting in dmesg. Compiled using gcc 3.4, local.mk and = kernel v2.6.20.3. Not using latest daily, will recompile to that and = reload module later on. I'm guessing that will solve the problem = You have changed your environment, thank you, but you still had a problem and I cannot reproduce your problem. I think I am stucking about your problem. If anyone can reproduce Jorgen's problem, please let me know how. Jorgen, If you meet BUG at iinfo.c:62 again, please send me the stacktrace. Your new kernel configuration will show us the correct stacktrace. And send me your .config too. Junjiro Okajima - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV