On 11/07/10 12:57 PM, Nikolaus Rath wrote: > Hello, > > I have set up diskless fat clients with kernel 2.6.32. The root filesystem is > aufs. Underlying the aufs is an rw tmpfs and a ro squashfs, the later mounted > from NBD. > > The problems is that the fat clients work fine for a little while, but then > reproducibly freeze completely within a few hours after booting. [ Snip! ] > These messages come for different tasks (not just cron) and to me the call > traces look identical (but I can also attach a full set of log messages). > > > Anyone able to help? > > I can't help much, but I believe this may be the same bug that I've been dealing with. I'm using a read-only NFS-root instead of a squashfs, and my rw overlays are tmpfs, though. My systems are headless high-performance cluster nodes, but are probably not much different from your fat clients.
Same issue -- things work great for a while, and then tasks begin to hang and the system locks up.. i've got four systems that i'm currently using for this, and at least one of the four (which one changes at random) fails usually within 2-3 hours of bootup. Sometimes a failure can take as little as 30 mins, though, and sometimes it can take up to four or five days before a given system will fail.. On my systems it's not a complete lockout, though; it seems to just be a lockout on file I/O (if a processess is loaded/running and it doesn't do any file I/O -- like 'top' -- then there's no problem) -- is this the same for your issue? If you take a look at your syslog, I bet you will see at least one additional hung task, that being an aufsd process. I usually see four or five hung processes (i have a quad-core processor, you may see less if you have fewer cores?), most of which are the jobs that i'm running (cron is often in that list too, though), but there's always one aufsd process (sometimes two, but usually just one) which is always reported as the -first- hung task. Those 'hung for 120 seconds'+coredump messages repeat for each hung task twice, and then all but the aufsd process seem to get killed: I also run 'top' in a console, and press 'i' to hide tasks that are not active -- after those 'hung tasks' messages appear I see that hung aufsd process on the list with 'D' beside it, and it does not disappear no matter how long I let the system sit (as mentioned, 'top' keeps clicking away though; it's not frozen) If you could confirm whether or not you're seeing the same behaviour as me (blocked aufsd process, system just locks up for file I/O), I'd appreciate it. Ian ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
