[OpenAFS-devel] write errors, servers going 'down'

[Kirby Bakken]

Help!

I'm running RHEL4 U4 (uname -r => 2.6.9-42.0.3.ELsmp) x86_64 on one of many dual Opteron 'linux' servers.  Our servers are running 'some' level of afs...  that may be important, but for now I'm trying to figure out where to start debug....

I get these messages in 'demsg':

afs: Lost contact with file server 9.10.228.186 in cell rchland.ibm.com (all multi-homed ip addresses down for the server)
afs: Lost contact with file server 9.10.228.186 in cell rchland.ibm.com (all multi-homed ip addresses down for the server)
afs: file server 9.10.228.186 in cell rchland.ibm.com is back up (multi-homed address; other same-host interfaces may still be down)
afs: file server 9.10.228.186 in cell rchland.ibm.com is back up (multi-homed address; other same-host interfaces may still be down)

I'm also seeing 'write' errors in the dmesg log, but don't currently have an exact 'paste' of that info....

These errors only occur at 'high' load.  Multiple processes writing/reading to the same afs volume.  I'm running these options and cache settings:

LARGE="-stat 2800 -dcache 2400 -daemons 5 -volumes 128"

I had been running with 'medium' settings, and that's when I saw the write errors...  now I just see the 'Lost contact' errors, and 'failed to store file' in the program writing files. (we're compiling/linking at the time these errors occur).

I've got the cache size 'set':

CACHESIZE=600000

although when I cat out the cacheinfo file I get this:

cat /usr/vice/etc/cacheinfo
/afs:/usr/vice/cache:3628512

I'm seeing these problems both with 'kernel-smp-module-openafs-1.4.0-2.6.9_42.0.3.EL_6_rhel4' and with 'openafs-kernel-smp-1.4.2-2.6.9_42.ELsmp_1.x86_64'

We had been seeing similar problems last March on RHEL4 U3, but an 'intermediate' AFS build of 'openafs-1.4.1rc2-rhel4.0.x86_64' seemed to work 'most of the time'...  (we get hangs about once every two weeks on each of 6 of the dual Opteron servers, and can't even log-into a local console to gather info..  so we're not sure if this is afs related or what).

What do I do to figure this out, or make it go away?  Is there a 'Having problems with openafs?  Here's what to try...." set of instructions somewhere that I've missed?

Thank you very much in advance for any help.

=======================
Kirby Bakken
ESW Build Architect
Rochester, MN
email: [EMAIL PROTECTED]
ezpage:kirbyb
507-253-4549 / Tie:  553-4549
Fax:  507-253-3495

......one more straw can't possibly matter....