One of the servers (on bio-ppc-head-1) was rebooted fairly recently, while the others were not.
--bob On Mon, 25 Sep 2006, Sam Lang wrote: > > Hi Bob, > > I think I found the bug and have a fix for this, but I just want to > make sure. Do you remember the last time you restarted the pvfs2 > servers? I think this error is caused at server startup on ppc64 > after at least 512 files are created. If your servers were re- > started recently (with a populated file system), then I know what's > wrong. Otherwise, if they were started once when the file system was > created and have been running ever since, I've just found a bug > elsewhere in our code, and I'll need to keep looking for yours. Let > me know. > > Thanks, > > -sam > > On Sep 25, 2006, at 1:51 PM, Robert Olson wrote: > > > This was built new for 1.5.1, and I've been using it on and off for > > a few weeks. > > > > --bob > > > > On Sep 25, 2006, at 1:37 PM, Sam Lang wrote: > > > >> > >> Hi Bob, > >> > >> Is this from a new storage space, or had things been working and > >> then you started getting the permission denied errors? If its a > >> storage space that's been around for a while, did you migrate from > >> 1.3.x or 1.4.0? > >> > >> Based on the logfiles you sent, it doesn't look like a kernel > >> module problem. We haven't seen those specific problems with > >> 1.5.1, but we didn't do much testing of the 1.5.1 release on 64 > >> bit platforms (esp. ppc). Would it be possible to get access to > >> that system so that I can do some debugging? > >> > >> Thanks, > >> > >> -sam > >> > >> On Sep 25, 2006, at 12:34 PM, Robert Olson wrote: > >> > >>> This is weird. I appear to getting permission denied on every > >>> other file to be copied from a local filesystem to the PVFS2 > >>> filesystem. This ring a bell with anyone? > >>> > >>> Thanks, > >>> --bob > >>> > >>> [EMAIL PROTECTED]:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson > >>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ > >>> sims.10': Permission denied > >>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ > >>> sims.101': Permission denied > >>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ > >>> sims.103': Permission denied > >>> cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ > >>> sims.105': Permission denied > >>> > >>> > >>> [EMAIL PROTECTED]:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/ > >>> total 88044 > >>> -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1 > >>> -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100 > >>> -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102 > >>> -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104 > >>> -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106 > >>> > >>> [EMAIL PROTECTED]:/scratch$ for i in simtab.0917/*; do echo $i; / > >>> vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done > >>> simtab.0917/sims.1 > >>> Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds > >>> simtab.0917/sims.10 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.10 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.10 > >>> simtab.0917/sims.100 > >>> Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds > >>> simtab.0917/sims.101 > >>> Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds > >>> simtab.0917/sims.102 > >>> Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds > >>> simtab.0917/sims.103 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.103 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.103 > >>> simtab.0917/sims.104 > >>> Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds > >>> simtab.0917/sims.105 > >>> Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds > >>> simtab.0917/sims.106 > >>> Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds > >>> simtab.0917/sims.107 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.107 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.107 > >>> simtab.0917/sims.108 > >>> Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds > >>> simtab.0917/sims.109 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.109 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.109 > >>> simtab.0917/sims.11 > >>> Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds > >>> simtab.0917/sims.110 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.110 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.110 > >>> simtab.0917/sims.111 > >>> Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds > >>> simtab.0917/sims.112 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.112 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.112 > >>> simtab.0917/sims.113 > >>> Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds > >>> simtab.0917/sims.114 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.114 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.114 > >>> simtab.0917/sims.115 > >>> Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds > >>> simtab.0917/sims.116 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.116 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.116 > >>> simtab.0917/sims.117 > >>> Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds > >>> simtab.0917/sims.118 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.118 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.118 > >>> simtab.0917/sims.119 > >>> Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds > >>> simtab.0917/sims.12 > >>> open: Permission denied > >>> could not open /disks/pvfs2/olson/simtab.0917/sims.12 > >>> Could not open /disks/pvfs2/olson/simtab.0917/sims.12 > >>> [etc] > >>> > >>> > >>> pvfs-statfs is happy: > >>> > >>> [EMAIL PROTECTED]:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2 > >>> > >>> aggregate statistics: > >>> --------------------------------------- > >>> > >>> fs_id: 1239905522 > >>> total number of servers (meta and I/O): 4 > >>> handles available (meta and I/O): 5365714538 > >>> handles total (meta and I/O): 5368709115 > >>> bytes available: 1218774892544 > >>> bytes total: 1495224942592 > >>> > >>> NOTE: The aggregate total and available statistics are calculated > >>> based > >>> on an algorithm that assumes data will be distributed evenly; thus > >>> the free space is equal to the smallest I/O server capacity > >>> multiplied by the number of I/O servers. If this number seems > >>> unusually small, then check the individual server statistics below > >>> to look for problematic servers. > >>> > >>> meta server statistics: > >>> --------------------------------------- > >>> > >>> server: tcp://bio-ppc-head-4:3334 > >>> RAM bytes total : 4107878400 > >>> RAM bytes free : 13307904 > >>> uptime (seconds) : 4045972 > >>> load averages : 4704 9088 5664 > >>> handles available: 2145227856 > >>> handles total : 2147483646 > >>> bytes available : 310944141312 > >>> bytes total : 373806235648 > >>> mode: serving both metadata and I/O data > >>> > >>> > >>> I/O server statistics: > >>> --------------------------------------- > >>> > >>> server: tcp://bio-ppc-head-4:3334 > >>> RAM bytes total : 4107878400 > >>> RAM bytes free : 13307904 > >>> uptime (seconds) : 4045972 > >>> load averages : 4704 9088 5664 > >>> handles available: 2145227856 > >>> handles total : 2147483646 > >>> bytes available : 310944141312 > >>> bytes total : 373806235648 > >>> mode: serving both metadata and I/O data > >>> > >>> server: tcp://bio-ppc-head-1:3334 > >>> RAM bytes total : 4107878400 > >>> RAM bytes free : 2062213120 > >>> uptime (seconds) : 832980 > >>> load averages : 30624 12352 3872 > >>> handles available: 1073741272 > >>> handles total : 1073741823 > >>> bytes available : 307329638400 > >>> bytes total : 373806235648 > >>> mode: serving only I/O data > >>> > >>> server: tcp://bio-ppc-head-2:3334 > >>> RAM bytes total : 4107878400 > >>> RAM bytes free : 12697600 > >>> uptime (seconds) : 4045977 > >>> load averages : 10592 6720 1792 > >>> handles available: 1073372712 > >>> handles total : 1073741823 > >>> bytes available : 304693723136 > >>> bytes total : 373806235648 > >>> mode: serving only I/O data > >>> > >>> server: tcp://bio-ppc-head-3:3334 > >>> RAM bytes total : 3578347520 > >>> RAM bytes free : 12324864 > >>> uptime (seconds) : 4046045 > >>> load averages : 13056 9376 3136 > >>> handles available: 1073372698 > >>> handles total : 1073741823 > >>> bytes available : 309065900032 > >>> bytes total : 373806235648 > >>> mode: serving only I/O data > >>> > >>> _______________________________________________ > >>> Pvfs2-users mailing list > >>> [email protected] > >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >>> > >> > > > > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
