HI Murali,
I use the RTSP over PVFS2 filesystem(pvfs2-1.5.1). Build a Media
server
system with 4 computer nodes and 2 io nodes.
But these days the PVFS2 cause 3 computer nodes kernel panic. help
me!
This is any of the log in /var/log/messages.
Help me analyse this and tell me how to fix the problem. Or I will
die ^_^
Thank you so much!!
Aug 29 14:32:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 14:32:46 redhat2 kernel: pvfs2_file_read: error writing to
handle
1046588, -- returning -110
Aug 29 17:17:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 17:17:53 redhat2 kernel: pvfs2_file_read: error writing to
handle
1047656, -- returning -110
Aug 29 18:05:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 18:05:36 redhat2 kernel: pvfs2_file_read: error writing to
handle
1046827, -- returning -110
Aug 29 18:14:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 18:14:03 redhat2 kernel: pvfs2_file_read: error writing to
handle
1046590, -- returning -110
Aug 29 19:24:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 19:24:03 redhat2 kernel: pvfs2_file_read: error writing to
handle
1048431, -- returning -110
Aug 29 20:04:40 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Aug 29 20:04:40 redhat2 kernel: pvfs2_file_read: error writing to
handle
1046679, -- returning -110
Sep 14 16:14:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:14:55 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 16:14:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:14:56 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 16:15:37 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:15:37 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 16:21:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:21:53 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 16:23:04 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:23:04 redhat2 kernel: pvfs2_file_read: error writing to
handle
1042058, -- returning -110
Sep 14 16:27:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 16:27:53 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:15:39 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:15:39 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:15:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:15:52 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:16:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:16:09 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:16:10 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:16:10 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:21:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:21:52 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:27:59 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:27:59 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:28:12 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:28:12 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 17:39:26 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:39:26 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039033, -- returning -110
Sep 14 17:43:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 17:43:06 redhat2 kernel: pvfs2_file_read: error writing to
handle
1043039, -- returning -110
Sep 14 18:08:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:08:54 redhat2 kernel: pvfs2_file_read: error writing to
handle
1040923, -- returning -110
Sep 14 18:18:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:18:53 redhat2 kernel: pvfs2_file_read: error writing to
handle
1040923, -- returning -110
Sep 14 18:27:07 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:27:07 redhat2 kernel: pvfs2_file_read: error writing to
handle
1042057, -- returning -110
Sep 14 18:33:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:33:52 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039041, -- returning -110
Sep 14 18:33:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:33:53 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 18:34:01 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:34:01 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038496, -- returning -110
Sep 14 18:50:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 18:50:06 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 19:03:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 19:03:36 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038489, -- returning -110
Sep 14 20:34:49 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 20:34:49 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 20:34:51 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 20:34:51 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 20:34:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 20:34:56 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 21:16:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 21:16:55 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 21:25:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 21:25:52 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039037, -- returning -110
Sep 14 21:33:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 21:33:09 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 21:52:30 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 21:52:30 redhat2 kernel: pvfs2_file_read: error writing to
handle
1039031, -- returning -110
Sep 14 22:17:05 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 22:17:05 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 14 22:17:45 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Sep 14 22:17:45 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038485, -- returning -110
Sep 30 14:29:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out
and retries exhausted. aborting attempt.
Sep 30 14:30:18 redhat2 last message repeated 2 times
Sep 30 14:35:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out
and retries exhausted. aborting attempt.
Sep 30 14:36:18 redhat2 last message repeated 2 times
Nov 2 22:16:08 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Nov 2 22:16:08 redhat2 kernel: pvfs2_file_read: error writing to
handle
1038504, -- returning -110
Nov 12 14:54:57 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036658, -- returning -2
Nov 12 14:55:00 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036651, -- returning -2
Nov 12 14:57:00 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036651, -- returning -2
Nov 12 14:57:04 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036658, -- returning -2
Nov 12 14:57:42 redhat2 last message repeated 2 times
Nov 12 15:00:08 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036651, -- returning -2
Nov 12 15:00:08 redhat2 last message repeated 2 times
Dec 5 12:51:01 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:51:05 redhat2 kernel: printk: 1 messages suppressed.
Dec 5 12:51:20 redhat2 kernel: printk: 3 messages suppressed.
Dec 5 12:51:35 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:52:26 redhat2 kernel: printk: 1 messages suppressed.
Dec 5 12:53:29 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:53:36 redhat2 kernel: printk: 3 messages suppressed.
Dec 5 12:53:43 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:53:52 redhat2 kernel: printk: 4 messages suppressed.
Dec 5 12:54:05 redhat2 kernel: printk: 3 messages suppressed.
Dec 5 12:54:08 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:54:13 redhat2 kernel: printk: 2 messages suppressed.
Dec 5 12:54:19 redhat2 kernel: printk: 5 messages suppressed.
Dec 5 12:54:36 redhat2 kernel: printk: 4 messages suppressed.
Dec 5 12:54:44 redhat2 kernel: printk: 5 messages suppressed.
Dec 5 12:54:46 redhat2 kernel: printk: 1 messages suppressed.
Dec 5 12:55:00 redhat2 kernel: printk: 1 messages suppressed.
Dec 5 12:55:02 redhat2 kernel: printk: 3 messages suppressed.
Dec 5 12:55:07 redhat2 kernel: printk: 1 messages suppressed.
Dec 5 12:55:38 redhat2 kernel: printk: 3 messages suppressed.
Dec 24 16:54:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Dec 24 16:54:54 redhat2 kernel: pvfs2_file_read: error writing to
handle
1042026, -- returning -110
Dec 24 17:04:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait
timed out
and retries exhausted. aborting attempt.
Dec 24 17:04:46 redhat2 kernel: pvfs2_file_read: error writing to
handle
1036052, -- returning -110
Dec 26 10:49:33 redhat2 sshd(pam_unix)[15164]: authentication
failure;
logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133 user=root
Dec 26 10:50:51 redhat2 sshd(pam_unix)[15164]: 1 more authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133
user=root
Dec 28 14:13:48 redhat2 kernel: printk: 1 messages suppressed.
Dec 28 14:14:26 redhat2 last message repeated 4 times
Dec 28 14:14:37 redhat2 kernel: printk: 2 messages suppressed.
Dec 28 14:14:46 redhat2 kernel: printk: 1 messages suppressed.
Dec 28 14:14:57 redhat2 kernel: printk: 2 messages suppressed.
Dec 28 14:42:22 redhat2 kernel: printk: 2 messages suppressed.
Dec 31 20:36:57 redhat2 kernel: Unable to handle kernel paging
request
at virtual address 67c38e12
Dec 31 20:36:57 redhat2 kernel: printing eip:
Dec 31 20:36:57 redhat2 kernel: f89e363e
Dec 31 20:36:57 redhat2 kernel: *pde = 00000000
Dec 31 20:36:57 redhat2 kernel: Oops: 0000 [#1]
Dec 31 20:36:57 redhat2 kernel: SMP
Dec 31 20:36:57 redhat2 kernel: Modules linked in: ip_vs pvfs2(U) md5
ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc dm_mirror
dm_multipath dm_mod button battery ac uhci_hcd e1000 floppy ext3 jbd
qla2300 qla2xxx scsi_transport_fc mptscsih mptbase sd_mod scsi_mod
Dec 31 20:36:57 redhat2 kernel: CPU: 2
Dec 31 20:36:57 redhat2 kernel: EIP: 0060:[<f89e363e>] Not
tainted VLI
Dec 31 20:36:57 redhat2 kernel: EFLAGS: 00010246 (2.6.9-22.ELsmp)
Dec 31 20:36:57 redhat2 kernel: EIP is at
pvfs2_devreq_writev+0x184/0x59b [pvfs2]
Dec 31 20:36:57 redhat2 kernel: eax: f5d66710 ebx: 67c38e12 ecx:
000001fd edx: f6a00000
Dec 31 20:36:57 redhat2 kernel: esi: f5d66700 edi: fffffe50 ebp:
f56cced4 esp: f56cceb8
Dec 31 20:36:57 redhat2 kernel: ds: 007b es: 007b ss: 0068
Dec 31 20:36:57 redhat2 kernel: Process pvfs2-client-co (pid: 4151,
threadinfo=f56cc000 task=f6ff00b0)
Dec 31 20:36:57 redhat2 kernel: Stack: f56cceb4 000022a8 00000004
dd4fc42c dd4fc41c 00000004 f56ccf40 80000000
Dec 31 20:36:57 redhat2 kernel: 00000000 f765fe80 00000000
c02759c6 00000040 00000000 00000000 f56ccf40
Dec 31 20:36:57 redhat2 kernel: 00000001 00000000 00000000
00000040 f63ccf48 00000a98 c0275a16 00000246
Dec 31 20:36:57 redhat2 kernel: Call Trace:
Dec 31 20:36:57 redhat2 kernel: [<c02759c6>] sock_readv_writev
+0x5e/0x81
Dec 31 20:36:57 redhat2 kernel: [<c0275a16>] sock_readv+0x2d/0x34
Dec 31 20:36:57 redhat2 kernel: [<f89e34ba>]
pvfs2_devreq_writev+0x0/0x59b [pvfs2]
Dec 31 20:36:57 redhat2 kernel: [<c015a177>] do_readv_writev
+0x19c/0x21d
Dec 31 20:36:57 redhat2 kernel: [<c015a276>] vfs_writev+0x3e/0x43
Dec 31 20:36:57 redhat2 kernel: [<c015a319>] sys_writev+0x3c/0x62
Dec 31 20:36:57 redhat2 kernel: [<c02d10cf>] syscall_call+0x7/0xb
Dec 31 20:36:57 redhat2 kernel: Code: b9 ff ff ff 5e e9 2e 04 00
00 8b
35 c4 5a 9f f8 8d 6c 24 1c 89 e8 8b 56 04 ff 56 0c 89 c7 8d 46 10
e8 0c
c6 8e c7 8b 16 8b 1c fa <8b> 03 0f 18 00 90 8d 04 fa 39 c3 74 1c
89 da
89 e8 ff 56 08 8b
Dec 31 20:36:57 redhat2 kernel: <0>Fatal exception: panic in 5
seconds
---------------------------------------------------------------------
---
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
