Hello Robin, On Wednesday 27 June 2007 14:34:11 Robin Humble wrote: > I've been trying out patchless kernels and the attached simple code > appears to trigger a failure in Lustre 1.6.0.1. I couldn't see anything > in bugzilla about it. > > typically I see 4+ open() failures out of 32 on the first run after a > Lustre filesystem is mounted. often (but not always) the number of > failures decreases to a few or 0 on subsequent runs. > > eg. typical output (where no output is success) would be: > % /opt/openmpi/1.2/bin/mpirun --hostfile hosts -np 32 ./open > /mnt/testfs/rjh open of '/mnt/testfs/rjh/blk016.dat' failed on rank 16,
[...] > > which is 32 threads across 6 nodes attempting to open and close 1 file each > (32 files total) in 1 directory over o2ib. > > if I umount and remount the filesystem then the higher rate of errors > occurs again. > cexec :11-16 umount /mnt/testfs > cexec :11-16 /usr/sbin/lustre_rmmod ; cexec :11-16 /usr/sbin/lustre_rmmod > cexec :11-16 mount -t lustre [EMAIL PROTECTED]:/testfs /mnt/testfs > > Note that the same failures happen over GigE too, but only on larger > tests. eg. -np 64 or 128. so the extra speed of IB is triggering the > bugs sooner. > > if Lustre kernel rpms (eg. 2.6.9-42.0.10.EL_lustre-1.6.0.1smp) are used > instead of the patchless kernels, then I don't see any failures. tested > out to -np 512. > > patchless 2.6.19.7 and 2.6.21.5 give failures at about the same rate. > modules for 2.6.19.7 were built using the standard Lustre 1.6.0.1 > tarball, and 2.6.21.5 modules were built using > > http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/lustre/1.6/lustre-1 >.6.0.1-ql3.tar.bz2 as that's a lot easier to work with than > https://bugzilla.lustre.org/show_bug.cgi?id=11647 for real MPI jobs you will probably also need flock support, but for this you will need -ql4 (bug #12802 and #11880). Could you please test again with a patched 2.6.20 or 2.6.21? So far we don't need patchless clients, so I don't test this extensivle. I also didn't test 2.6.21 very much. I think I will skip 2.6.21 at all and after adding sanity tests for the 2.6.22 patches will test this version more thorougly. [...] Also, did you see anything in the logs (server and clients)? > another data point is that if I rm all the files in the dir then the > test succeeds more often (up until the time the fs is umount'd and > remounted). so something about the unlink/create combo might be the > problem. eg. When I run "sanity.sh 76" it will. Test 76 is also about unlink/create actions, only that it tests the inode cache. No idea so far where I need to look into... Cheers, Bernd PS: I would test this here, but presently we have too few customer systems in repair to do these tests with ;) -- Bernd Schubert Q-Leap Networks GmbH _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
