HI Murali, I use the RTSP over PVFS2 filesystem(pvfs2-1.5.1). Build a Media server system with 4 computer nodes and 2 io nodes. But these days the PVFS2 cause 3 computer nodes kernel panic. help me!
This is any of the log in /var/log/messages. Help me analyse this and tell me how to fix the problem. Or I will die ^_^ Thank you so much!! Aug 29 14:32:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 14:32:46 redhat2 kernel: pvfs2_file_read: error writing to handle 1046588, -- returning -110 Aug 29 17:17:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 17:17:53 redhat2 kernel: pvfs2_file_read: error writing to handle 1047656, -- returning -110 Aug 29 18:05:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 18:05:36 redhat2 kernel: pvfs2_file_read: error writing to handle 1046827, -- returning -110 Aug 29 18:14:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 18:14:03 redhat2 kernel: pvfs2_file_read: error writing to handle 1046590, -- returning -110 Aug 29 19:24:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 19:24:03 redhat2 kernel: pvfs2_file_read: error writing to handle 1048431, -- returning -110 Aug 29 20:04:40 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Aug 29 20:04:40 redhat2 kernel: pvfs2_file_read: error writing to handle 1046679, -- returning -110 Sep 14 16:14:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:14:55 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 16:14:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:14:56 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 16:15:37 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:15:37 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 16:21:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:21:53 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 16:23:04 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:23:04 redhat2 kernel: pvfs2_file_read: error writing to handle 1042058, -- returning -110 Sep 14 16:27:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 16:27:53 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:15:39 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:15:39 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:15:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:15:52 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:16:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:16:09 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:16:10 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:16:10 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:21:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:21:52 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:27:59 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:27:59 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:28:12 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:28:12 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 17:39:26 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:39:26 redhat2 kernel: pvfs2_file_read: error writing to handle 1039033, -- returning -110 Sep 14 17:43:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 17:43:06 redhat2 kernel: pvfs2_file_read: error writing to handle 1043039, -- returning -110 Sep 14 18:08:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:08:54 redhat2 kernel: pvfs2_file_read: error writing to handle 1040923, -- returning -110 Sep 14 18:18:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:18:53 redhat2 kernel: pvfs2_file_read: error writing to handle 1040923, -- returning -110 Sep 14 18:27:07 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:27:07 redhat2 kernel: pvfs2_file_read: error writing to handle 1042057, -- returning -110 Sep 14 18:33:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:33:52 redhat2 kernel: pvfs2_file_read: error writing to handle 1039041, -- returning -110 Sep 14 18:33:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:33:53 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 18:34:01 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:34:01 redhat2 kernel: pvfs2_file_read: error writing to handle 1038496, -- returning -110 Sep 14 18:50:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 18:50:06 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 19:03:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 19:03:36 redhat2 kernel: pvfs2_file_read: error writing to handle 1038489, -- returning -110 Sep 14 20:34:49 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 20:34:49 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 20:34:51 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 20:34:51 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 20:34:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 20:34:56 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 21:16:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 21:16:55 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 21:25:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 21:25:52 redhat2 kernel: pvfs2_file_read: error writing to handle 1039037, -- returning -110 Sep 14 21:33:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 21:33:09 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 21:52:30 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 21:52:30 redhat2 kernel: pvfs2_file_read: error writing to handle 1039031, -- returning -110 Sep 14 22:17:05 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 22:17:05 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 14 22:17:45 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Sep 14 22:17:45 redhat2 kernel: pvfs2_file_read: error writing to handle 1038485, -- returning -110 Sep 30 14:29:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out and retries exhausted. aborting attempt. Sep 30 14:30:18 redhat2 last message repeated 2 times Sep 30 14:35:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out and retries exhausted. aborting attempt. Sep 30 14:36:18 redhat2 last message repeated 2 times Nov 2 22:16:08 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Nov 2 22:16:08 redhat2 kernel: pvfs2_file_read: error writing to handle 1038504, -- returning -110 Nov 12 14:54:57 redhat2 kernel: pvfs2_file_read: error writing to handle 1036658, -- returning -2 Nov 12 14:55:00 redhat2 kernel: pvfs2_file_read: error writing to handle 1036651, -- returning -2 Nov 12 14:57:00 redhat2 kernel: pvfs2_file_read: error writing to handle 1036651, -- returning -2 Nov 12 14:57:04 redhat2 kernel: pvfs2_file_read: error writing to handle 1036658, -- returning -2 Nov 12 14:57:42 redhat2 last message repeated 2 times Nov 12 15:00:08 redhat2 kernel: pvfs2_file_read: error writing to handle 1036651, -- returning -2 Nov 12 15:00:08 redhat2 last message repeated 2 times Dec 5 12:51:01 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:51:05 redhat2 kernel: printk: 1 messages suppressed. Dec 5 12:51:20 redhat2 kernel: printk: 3 messages suppressed. Dec 5 12:51:35 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:52:26 redhat2 kernel: printk: 1 messages suppressed. Dec 5 12:53:29 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:53:36 redhat2 kernel: printk: 3 messages suppressed. Dec 5 12:53:43 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:53:52 redhat2 kernel: printk: 4 messages suppressed. Dec 5 12:54:05 redhat2 kernel: printk: 3 messages suppressed. Dec 5 12:54:08 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:54:13 redhat2 kernel: printk: 2 messages suppressed. Dec 5 12:54:19 redhat2 kernel: printk: 5 messages suppressed. Dec 5 12:54:36 redhat2 kernel: printk: 4 messages suppressed. Dec 5 12:54:44 redhat2 kernel: printk: 5 messages suppressed. Dec 5 12:54:46 redhat2 kernel: printk: 1 messages suppressed. Dec 5 12:55:00 redhat2 kernel: printk: 1 messages suppressed. Dec 5 12:55:02 redhat2 kernel: printk: 3 messages suppressed. Dec 5 12:55:07 redhat2 kernel: printk: 1 messages suppressed. Dec 5 12:55:38 redhat2 kernel: printk: 3 messages suppressed. Dec 24 16:54:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Dec 24 16:54:54 redhat2 kernel: pvfs2_file_read: error writing to handle 1042026, -- returning -110 Dec 24 17:04:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out and retries exhausted. aborting attempt. Dec 24 17:04:46 redhat2 kernel: pvfs2_file_read: error writing to handle 1036052, -- returning -110 Dec 26 10:49:33 redhat2 sshd(pam_unix)[15164]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133 user=root Dec 26 10:50:51 redhat2 sshd(pam_unix)[15164]: 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133 user=root Dec 28 14:13:48 redhat2 kernel: printk: 1 messages suppressed. Dec 28 14:14:26 redhat2 last message repeated 4 times Dec 28 14:14:37 redhat2 kernel: printk: 2 messages suppressed. Dec 28 14:14:46 redhat2 kernel: printk: 1 messages suppressed. Dec 28 14:14:57 redhat2 kernel: printk: 2 messages suppressed. Dec 28 14:42:22 redhat2 kernel: printk: 2 messages suppressed. Dec 31 20:36:57 redhat2 kernel: Unable to handle kernel paging request at virtual address 67c38e12 Dec 31 20:36:57 redhat2 kernel: printing eip: Dec 31 20:36:57 redhat2 kernel: f89e363e Dec 31 20:36:57 redhat2 kernel: *pde = 00000000 Dec 31 20:36:57 redhat2 kernel: Oops: 0000 [#1] Dec 31 20:36:57 redhat2 kernel: SMP Dec 31 20:36:57 redhat2 kernel: Modules linked in: ip_vs pvfs2(U) md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc dm_mirror dm_multipath dm_mod button battery ac uhci_hcd e1000 floppy ext3 jbd qla2300 qla2xxx scsi_transport_fc mptscsih mptbase sd_mod scsi_mod Dec 31 20:36:57 redhat2 kernel: CPU: 2 Dec 31 20:36:57 redhat2 kernel: EIP: 0060:[<f89e363e>] Not tainted VLI Dec 31 20:36:57 redhat2 kernel: EFLAGS: 00010246 (2.6.9-22.ELsmp) Dec 31 20:36:57 redhat2 kernel: EIP is at pvfs2_devreq_writev+0x184/0x59b [pvfs2] Dec 31 20:36:57 redhat2 kernel: eax: f5d66710 ebx: 67c38e12 ecx: 000001fd edx: f6a00000 Dec 31 20:36:57 redhat2 kernel: esi: f5d66700 edi: fffffe50 ebp: f56cced4 esp: f56cceb8 Dec 31 20:36:57 redhat2 kernel: ds: 007b es: 007b ss: 0068 Dec 31 20:36:57 redhat2 kernel: Process pvfs2-client-co (pid: 4151, threadinfo=f56cc000 task=f6ff00b0) Dec 31 20:36:57 redhat2 kernel: Stack: f56cceb4 000022a8 00000004 dd4fc42c dd4fc41c 00000004 f56ccf40 80000000 Dec 31 20:36:57 redhat2 kernel: 00000000 f765fe80 00000000 c02759c6 00000040 00000000 00000000 f56ccf40 Dec 31 20:36:57 redhat2 kernel: 00000001 00000000 00000000 00000040 f63ccf48 00000a98 c0275a16 00000246 Dec 31 20:36:57 redhat2 kernel: Call Trace: Dec 31 20:36:57 redhat2 kernel: [<c02759c6>] sock_readv_writev+0x5e/0x81 Dec 31 20:36:57 redhat2 kernel: [<c0275a16>] sock_readv+0x2d/0x34 Dec 31 20:36:57 redhat2 kernel: [<f89e34ba>] pvfs2_devreq_writev+0x0/0x59b [pvfs2] Dec 31 20:36:57 redhat2 kernel: [<c015a177>] do_readv_writev+0x19c/0x21d Dec 31 20:36:57 redhat2 kernel: [<c015a276>] vfs_writev+0x3e/0x43 Dec 31 20:36:57 redhat2 kernel: [<c015a319>] sys_writev+0x3c/0x62 Dec 31 20:36:57 redhat2 kernel: [<c02d10cf>] syscall_call+0x7/0xb Dec 31 20:36:57 redhat2 kernel: Code: b9 ff ff ff 5e e9 2e 04 00 00 8b 35 c4 5a 9f f8 8d 6c 24 1c 89 e8 8b 56 04 ff 56 0c 89 c7 8d 46 10 e8 0c c6 8e c7 8b 16 8b 1c fa <8b> 03 0f 18 00 90 8d 04 fa 39 c3 74 1c 89 da 89 e8 ff 56 08 8b Dec 31 20:36:57 redhat2 kernel: <0>Fatal exception: panic in 5 seconds
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
