> yes, that's very interesting. you've probably told us what we need to. > you got an oops in the thread holding the lock, the machine will never > recover. what filesystem is behind your afs cache?
We use a virtual ext3 filesystem mounted on a loopback device. We have done this with many afs clients and it seems to be working. Here is our fstab entry: /AFSvirtualFS /usr/vice/cache ext3 defaults,loop=/dev/loop0 0 0 Mark Henry Derrick Brashear <[email protected]> 05/12/2009 02:58 PM To Mark Henry <[email protected]> cc Felix Frank <[email protected]>, [email protected], Simon Wilkinson <[email protected]> Subject Re: [OpenAFS] /afs area is hanging On Tue, May 12, 2009 at 4:37 PM, Mark Henry <[email protected]> wrote: > > > >> I took the liberty to paste the interesting parts to >> http://pastebin.com/m53578cd5. Notice the bottom, which was the original >> bottom as well. Mark, you've been asked to look at dmesg before this, so I >> suppose this didn't happen before you tried this call-tracing? > >> Besides, it would be interesting if an upgrade to 1.4.10 makes the problem >> go away. Can you try that? > > We upgraded to 1.4.10. We got the same errors. Here is the cmdebug output: > > => cmdebug HOSTNAME > ** Cache entry @ 0x172d1880 for 2.536870937.28.1820 [afs.dev.infoprint.com] > locks: (none_waiting, write_locked(pid:-246839824 at:681)) > 3 bytes DV 0 refcnt 2 > callback 22174d40 expires 1242158668 > 0 opens 0 writers > mount point > states (0x5), stat'd, read-only which is afs_HandleLink, called from afs_lookup. Memcache or disk cache? (different implementations of that function) > > The volume # 536870937 above just happens to be root.cell but it has been > different based on which dir I do the ls in. > >> Finally, it looks like ls and sshd got locked up trying to determine your >> client machine's home cell. I can see that happening (only) if no cell has >> been set at that point. The output of fs wscell would be interesting in >> this situation, but I'm not sure wether that would lock up as well (and if >> it is at all helpful). > > We tried the fs wscell command. It worked fine if the fs command was local > and hung if the fs was being retrieved from afs. well, if afs is unhappy, one presumes afs is unhappy. > Also, here is a bit of interesting output from dmesg when the system is > hung: ah, that'd be disk cache. > AssertProcessEntry: pohm_main, pid=6518 > openafs: Can't open inode 95550 yes, that's very interesting. you've probably told us what we need to. you got an oops in the thread holding the lock, the machine will never recover. what filesystem is behind your afs cache? > ------------[ cut here ]------------ > kernel BUG at > /compile/openafs-1.4.10/src/libafs/MODLOAD-2.6.22.5-31-default-MP/osi_file.c:87! > invalid opcode: 0000 [1] SMP > last sysfs file: /class/scsi_host/host0/model > CPU 6 > Modules linked in: tun iptable_filter ip_tables x_tables amk ipv6 libafs(P) > microcode firmware_class usbhid hid ff_memless tp_dd af_packet apparmor ext2 > loop dm_mod parport_pc parport bnx2 rtc_cmos rtc_core i2c_i801 rtc_lib > ide_cd i2c_core cdrom shpchp tg3 pci_hotplug container button sg ehci_hcd > uhci_hcd usbcore sd_mod ata_piix libata edd ext3 mbcache jbd fan aacraid > scsi_mod piix ide_core thermal processor > Pid: 6519, comm: ASCIIMast Tainted: P N 2.6.22.5-31-default #1 > RIP: 0010:[<ffffffff8834dde9>] [<ffffffff8834dde9>] > :libafs:osi_UFSOpen+0x155/0x1f2 > RSP: 0000:ffff81031de299c8 EFLAGS: 00010296 > RAX: 0000000000000023 RBX: ffff81031a92b418 RCX: 0000000000000001 > RDX: ffffffff804bdfe8 RSI: 0000000000000096 RDI: ffffffff804bdfe0 > RBP: ffff81042145f000 R08: 0000000000000001 R09: ffff810001086bc0 > R10: 0000000000000046 R11: ffff81042fa74ec0 R12: ffff81042af37000 > R13: 000000000001753e R14: 000000000001753e R15: 0000000000000003 > FS: 0000000000000000(0000) GS:ffff81042ee9f1c0(0000) knlGS:0000000000000000 > CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b > CR2: 00000000f5d6050c CR3: 000000039e0b9000 CR4: 00000000000006e0 > Process ASCIIMast (pid: 6519, threadinfo ffff81031de28000, task > ffff81031ad770c0) > Stack: ffffc2000770eb90 ffffc2000770caa8 ffff81042a631000 ffff8104172d1880 > ffff81039800839c ffffffff8832f678 0000000300000000 0000000000000003 > 0000000000000000 ffffc2000770eb90 ffff8104172d1880 0000000000000000 > Call Trace: > [<ffffffff8832f678>] :libafs:afs_UFSHandleLink+0xf7/0x1bd > [<ffffffff8832aadf>] :libafs:afs_lookup+0xbb2/0x115f > [<ffffffff88352713>] :libafs:afs_linux_dentry_revalidate+0x422/0x434 > [<ffffffff883521ac>] :libafs:afs_linux_lookup+0x85/0x1ca > [<ffffffff883188c7>] :libafs:PagInCred+0x30/0xa9 > [<ffffffff8028f972>] do_lookup+0xc4/0x1ae > [<ffffffff8029179a>] __link_path_walk+0x36c/0xd8b > [<ffffffff80299115>] dput+0x26/0x115 > [<ffffffff80292066>] __link_path_walk+0xc38/0xd8b > [<ffffffff80292211>] link_path_walk+0x58/0xe0 > [<ffffffff802877e9>] do_filp_open+0x1c/0x3d > [<ffffffff80292589>] do_path_lookup+0x1ab/0x227 > [<ffffffff80292fb9>] __path_lookup_intent_open+0x56/0x97 > [<ffffffff80293148>] open_namei+0x7a/0x674 > [<ffffffff802877e9>] do_filp_open+0x1c/0x3d > [<ffffffff8028784e>] do_sys_open+0x44/0xc1 > [<ffffffff80220bb2>] ia32_sysret+0x0/0xa > > > We still can't seem to get this system to stop hanging in the /afs area. > > Mark Henry > > > _____________________________________________________________________________ > "This message and any attachments are solely for the intended recipient and > may contain confidential or privileged information. If you are not the > intended recipient, any disclosure, copying, use, or distribution of the > information included in this message and any attachments is prohibited. If > you have received this communication in error, please notify us by reply > e-mail and immediately and permanently delete this message and any > attachments. Thank you." > _____________________________________________________________________________ > -- Derrick _____________________________________________________________________________ "This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you." _____________________________________________________________________________
