Here is a full strace -v output:
[r...@node1 root]# strace -v cp test.file /mnt/pvfs2/
execve("/bin/cp", ["cp", "test.file", "/mnt/pvfs2/"], [/* 22 vars */]) = 0
uname({sysname="Linux", nodename="node1", release="2.4.21-27.0.2.ELsmp",
version="#1 SMP Wed Jan 12 23:35:44 EST 2005", machine="i686"}) = 0
brk(0) = 0x9692000
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_dev=makedev(104, 3), st_ino=229475, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=32,
st_size=14525, st_atime=2009/04/07-15:54:03, st_mtime=2009/04/07-13:38:35,
st_ctime=2009/04/07-13:38:35}) = 0
old_mmap(NULL, 14525, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb75f5000
close(3) = 0
open("/lib/libacl.so.1", O_RDONLY) = 3
read(3, "\177elf\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\...@\24\0\000"..., 512)
= 512
fstat64(3, {st_dev=makedev(104, 3), st_ino=524363, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=40,
st_size=19248, st_atime=2009/04/07-15:54:03, st_mtime=2003/01/28-18:42:21,
st_ctime=2009/04/07-13:37:22}) = 0
old_mmap(NULL, 22224, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x89c000
old_mmap(0x8a1000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x4000) = 0x8a1000
close(3) = 0
mprotect(0xbfffa000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC|PROT_GROWSDOWN) =
0
open("/lib/tls/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200X\1"..., 512) =
512
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0xb75f4000
fstat64(3, {st_dev=makedev(104, 3), st_ino=14172162, st_mode=S_IFREG|0755,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=3080,
st_size=1571692, st_atime=2009/04/07-15:54:03, st_mtime=2004/10/22-04:01:20,
st_ctime=2009/04/07-13:37:20}) = 0
old_mmap(NULL, 1275340, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xd4c000
old_mmap(0xe7e000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x132000) = 0xe7e000
old_mmap(0xe81000, 9676, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xe81000
close(3) = 0
open("/lib/libattr.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\n\0"..., 512) =
512
fstat64(3, {st_dev=makedev(104, 3), st_ino=524361, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=7148,
st_atime=2009/04/07-15:54:03, st_mtime=2003/01/28-18:09:10,
st_ctime=2009/04/07-13:37:22}) = 0
old_mmap(NULL, 10124, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb1e000
old_mmap(0xb20000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x1000) = 0xb20000
close(3) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xb75f4ae0, limit:1048575,
seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1,
seg_not_present:0, useable:1}) = 0
munmap(0xb75f5000, 14525) = 0
brk(0) = 0x9692000
brk(0x96b3000) = 0x96b3000
brk(0) = 0x96b3000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_dev=makedev(104, 3), st_ino=3325956, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=62808,
st_size=32148976, st_atime=2009/04/07-15:29:59,
st_mtime=2009/04/07-13:37:19, st_ctime=2009/04/07-13:37:20}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb73f4000
close(3) = 0
geteuid32() = 0
lstat64("/mnt/pvfs2/", {st_dev=makedev(0, 10), st_ino=1048576,
st_mode=S_IFDIR|S_ISVTX|0777, st_nlink=1, st_uid=0, st_gid=0,
st_blksize=33554432, st_blocks=8, st_size=4096,
st_atime=2009/04/07-15:31:17, st_mtime=2009/04/07-15:31:17,
st_ctime=2009/04/07-15:31:17}) = 0
stat64("/mnt/pvfs2/", {st_dev=makedev(0, 10), st_ino=1048576,
st_mode=S_IFDIR|S_ISVTX|0777, st_nlink=1, st_uid=0, st_gid=0,
st_blksize=33554432, st_blocks=8, st_size=4096,
st_atime=2009/04/07-15:31:17, st_mtime=2009/04/07-15:31:17,
st_ctime=2009/04/07-15:31:17}) = 0
stat64("test.file", {st_dev=makedev(104, 3), st_ino=294926,
st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096,
st_blocks=8, st_size=5, st_atime=2009/04/07-15:29:59,
st_mtime=2009/04/07-14:45:08, st_ctime=2009/04/07-14:46:32}) = 0
stat64("/mnt/pvfs2/test.file", {st_dev=makedev(0, 10), st_ino=1048571,
st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=33554432,
st_blocks=8, st_size=5, st_atime=2009/04/07-15:31:17,
st_mtime=2009/04/07-15:31:17, st_ctime=2009/04/07-15:31:17}) = 0
open("test.file", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_dev=makedev(104, 3), st_ino=294926, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=5,
st_atime=2009/04/07-15:29:59, st_mtime=2009/04/07-14:45:08,
st_ctime=2009/04/07-14:46:32}) = 0
open("/mnt/pvfs2/test.file", O_WRONLY|O_TRUNC|O_LARGEFILE) = 4
fstat64(4, {st_dev=makedev(0, 10), st_ino=1048571, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=33554432, st_blocks=8, st_size=0,
st_atime=2009/04/07-15:31:17, st_mtime=2009/04/07-15:31:17,
st_ctime=2009/04/07-15:55:20}) = 0
fstat64(3, {st_dev=makedev(104, 3), st_ino=294926, st_mode=S_IFREG|0644,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=5,
st_atime=2009/04/07-15:29:59, st_mtime=2009/04/07-14:45:08,
st_ctime=2009/04/07-14:46:32}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
[r...@node1 root]#
On Tue, Apr 7, 2009 at 2:53 PM, Phil Carns <[email protected]> wrote:
> Hi Bart,
>
> From your strace output, my guess is that cp is running into trouble with
> the value of one of the fstat() fields, but its hard to say which one.
>
> Are you able to reproduce this reliably? Could you run the strace again
> with the -v option to see if it gives a full listing of what values were in
> the stat structs it got before crashing?
>
> -Phil
>
> Bart Taylor wrote:
>
>> Hey guys,
>>
>> I am running into a problem with a system copy command segfaulting on 2.4
>> kernels. Specifically, I am seeing this show up on RHEL3 machines running a
>> patched version of PVFS 2.6. Machines running Linux 2.6 kernels do not
>> experience this problem. I believe we may have mentioned this recently but
>> hoped it would be fixed by some updates pulled into dcache. That,
>> apparently, is not the case.
>>
>> The segfault is extremely consistent; it happens every time a cp is
>> executed with a PVFS2 file system as the target. The target file is always
>> created with a size of zero, so at least part of the command is completing.
>> 'dd' commands execute normally.
>>
>> The setup is simple: 1 server node (RHEL4 2.6 kernel) with the default
>> interactive genconfig output, and 1 client with a 2.4 kernel. Mount the
>> file system, execute a copy onto the file system.
>> Here is the conf file contents:
>>
>> <Defaults>
>> UnexpectedRequests 50
>> EventLogging none
>> LogStamp datetime
>> BMIModules bmi_tcp
>> FlowModules flowproto_multiqueue
>> PerfUpdateInterval 1000
>> ServerJobBMITimeoutSecs 30
>> ServerJobFlowTimeoutSecs 30
>> ClientJobBMITimeoutSecs 300
>> ClientJobFlowTimeoutSecs 300
>> ClientRetryLimit 5
>> ClientRetryDelayMilliSecs 2000
>> TCPBindSpecific yes
>> </Defaults>
>>
>> <Aliases>
>> Alias node1 tcp://node1:3334
>> </Aliases>
>>
>> <Filesystem>
>> Name pvfs2-fs
>> ID 1227216139
>> RootHandle 1048576
>> <MetaHandleRanges>
>> Range node1 4-2147483650
>> </MetaHandleRanges>
>> <DataHandleRanges>
>> Range node1 2147483651-4294967297
>> </DataHandleRanges>
>> <StorageHints>
>> TroveSyncMeta no
>> TroveSyncData no
>> CoalescingHighWatermark infinity
>> CoalescingLowWatermark 0
>> TroveSyncMetaTimerSecs 5
>> DBCacheSizeBytes 1073741824
>> </StorageHints>
>> </Filesystem>
>>
>> And here is the last bit of an strace on a copy command:
>>
>> [r...@node1 root]# strace cp test.file /mnt/pvfs2/
>> .....
>> brk(0) = 0x95ce000
>> open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
>> fstat64(3, {st_mode=S_IFREG|0644, st_size=32148976, ...}) = 0
>> mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb73f4000
>> close(3) = 0
>> geteuid32() = 0
>> lstat64("/mnt/pvfs2/", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...})
>> = 0
>> stat64("/mnt/pvfs2/", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) =
>> 0
>> stat64("test.file", {st_mode=S_IFREG|0644, st_size=5, ...}) = 0
>> stat64("/mnt/pvfs2/test.file", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> open("test.file", O_RDONLY|O_LARGEFILE) = 3
>> fstat64(3, {st_mode=S_IFREG|0644, st_size=5, ...}) = 0
>> open("/mnt/pvfs2/test.file", O_WRONLY|O_TRUNC|O_LARGEFILE) = 4
>> fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> fstat64(3, {st_mode=S_IFREG|0644, st_size=5, ...}) = 0
>> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
>> +++ killed by SIGSEGV +++
>>
>>
>> There is nothing in the client or server logs without turning on
>> additional logging.
>>
>> Are there any suggestions on what might be causing this? Can I provide any
>> additional information that will be helpful for debugging?
>>
>> Bart.
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Pvfs2-developers mailing list
>> [email protected]
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
>>
>
>
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
