Hi Bob,
I think I found the bug and have a fix for this, but I just want to
make sure. Do you remember the last time you restarted the pvfs2
servers? I think this error is caused at server startup on ppc64
after at least 512 files are created. If your servers were re-
started recently (with a populated file system), then I know what's
wrong. Otherwise, if they were started once when the file system was
created and have been running ever since, I've just found a bug
elsewhere in our code, and I'll need to keep looking for yours. Let
me know.
Thanks,
-sam
On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:
This was built new for 1.5.1, and I've been using it on and off for
a few weeks.
--bob
On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:
Hi Bob,
Is this from a new storage space, or had things been working and
then you started getting the permission denied errors? If its a
storage space that's been around for a while, did you migrate from
1.3.x or 1.4.0?
Based on the logfiles you sent, it doesn't look like a kernel
module problem. We haven't seen those specific problems with
1.5.1, but we didn't do much testing of the 1.5.1 release on 64
bit platforms (esp. ppc). Would it be possible to get access to
that system so that I can do some debugging?
Thanks,
-sam
On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:
This is weird. I appear to getting permission denied on every
other file to be copied from a local filesystem to the PVFS2
filesystem. This ring a bell with anyone?
Thanks,
--bob
[EMAIL PROTECTED]:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson
cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
sims.10': Permission denied
cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
sims.101': Permission denied
cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
sims.103': Permission denied
cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/
sims.105': Permission denied
[EMAIL PROTECTED]:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/
total 88044
-rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1
-rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100
-rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102
-rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104
-rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106
[EMAIL PROTECTED]:/scratch$ for i in simtab.0917/*; do echo $i; /
vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done
simtab.0917/sims.1
Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds
simtab.0917/sims.10
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.10
Could not open /disks/pvfs2/olson/simtab.0917/sims.10
simtab.0917/sims.100
Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds
simtab.0917/sims.101
Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds
simtab.0917/sims.102
Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds
simtab.0917/sims.103
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.103
Could not open /disks/pvfs2/olson/simtab.0917/sims.103
simtab.0917/sims.104
Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds
simtab.0917/sims.105
Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds
simtab.0917/sims.106
Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds
simtab.0917/sims.107
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.107
Could not open /disks/pvfs2/olson/simtab.0917/sims.107
simtab.0917/sims.108
Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds
simtab.0917/sims.109
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.109
Could not open /disks/pvfs2/olson/simtab.0917/sims.109
simtab.0917/sims.11
Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds
simtab.0917/sims.110
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.110
Could not open /disks/pvfs2/olson/simtab.0917/sims.110
simtab.0917/sims.111
Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds
simtab.0917/sims.112
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.112
Could not open /disks/pvfs2/olson/simtab.0917/sims.112
simtab.0917/sims.113
Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds
simtab.0917/sims.114
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.114
Could not open /disks/pvfs2/olson/simtab.0917/sims.114
simtab.0917/sims.115
Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds
simtab.0917/sims.116
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.116
Could not open /disks/pvfs2/olson/simtab.0917/sims.116
simtab.0917/sims.117
Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds
simtab.0917/sims.118
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.118
Could not open /disks/pvfs2/olson/simtab.0917/sims.118
simtab.0917/sims.119
Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds
simtab.0917/sims.12
open: Permission denied
could not open /disks/pvfs2/olson/simtab.0917/sims.12
Could not open /disks/pvfs2/olson/simtab.0917/sims.12
[etc]
pvfs-statfs is happy:
[EMAIL PROTECTED]:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2
aggregate statistics:
---------------------------------------
fs_id: 1239905522
total number of servers (meta and I/O): 4
handles available (meta and I/O): 5365714538
handles total (meta and I/O): 5368709115
bytes available: 1218774892544
bytes total: 1495224942592
NOTE: The aggregate total and available statistics are calculated
based
on an algorithm that assumes data will be distributed evenly; thus
the free space is equal to the smallest I/O server capacity
multiplied by the number of I/O servers. If this number seems
unusually small, then check the individual server statistics below
to look for problematic servers.
meta server statistics:
---------------------------------------
server: tcp://bio-ppc-head-4:3334
RAM bytes total : 4107878400
RAM bytes free : 13307904
uptime (seconds) : 4045972
load averages : 4704 9088 5664
handles available: 2145227856
handles total : 2147483646
bytes available : 310944141312
bytes total : 373806235648
mode: serving both metadata and I/O data
I/O server statistics:
---------------------------------------
server: tcp://bio-ppc-head-4:3334
RAM bytes total : 4107878400
RAM bytes free : 13307904
uptime (seconds) : 4045972
load averages : 4704 9088 5664
handles available: 2145227856
handles total : 2147483646
bytes available : 310944141312
bytes total : 373806235648
mode: serving both metadata and I/O data
server: tcp://bio-ppc-head-1:3334
RAM bytes total : 4107878400
RAM bytes free : 2062213120
uptime (seconds) : 832980
load averages : 30624 12352 3872
handles available: 1073741272
handles total : 1073741823
bytes available : 307329638400
bytes total : 373806235648
mode: serving only I/O data
server: tcp://bio-ppc-head-2:3334
RAM bytes total : 4107878400
RAM bytes free : 12697600
uptime (seconds) : 4045977
load averages : 10592 6720 1792
handles available: 1073372712
handles total : 1073741823
bytes available : 304693723136
bytes total : 373806235648
mode: serving only I/O data
server: tcp://bio-ppc-head-3:3334
RAM bytes total : 3578347520
RAM bytes free : 12324864
uptime (seconds) : 4046045
load averages : 13056 9376 3136
handles available: 1073372698
handles total : 1073741823
bytes available : 309065900032
bytes total : 373806235648
mode: serving only I/O data
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
