Hi, Were the servers still up and running when this happened?
What's your network? Have you seen dropped packets etc? We're not used to seeing applications that are so persistent in continuing to perform I/O in the presence of failures, so there may be a bug here that is being uncovered by this continuous reading in an error case... Thanks for the report! Rob jiang yi wrote: > HI Murali, > I use the RTSP over PVFS2 filesystem(pvfs2-1.5.1). Build a Media server > system with 4 computer nodes and 2 io nodes. > But these days the PVFS2 cause 3 computer nodes kernel panic. help me! > > This is any of the log in /var/log/messages. > Help me analyse this and tell me how to fix the problem. Or I will die ^_^ > Thank you so much!! > > > Aug 29 14:32:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 14:32:46 redhat2 kernel: pvfs2_file_read: error writing to handle > 1046588, -- returning -110 > Aug 29 17:17:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 17:17:53 redhat2 kernel: pvfs2_file_read: error writing to handle > 1047656, -- returning -110 > Aug 29 18:05:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 18:05:36 redhat2 kernel: pvfs2_file_read: error writing to handle > 1046827, -- returning -110 > Aug 29 18:14:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 18:14:03 redhat2 kernel: pvfs2_file_read: error writing to handle > 1046590, -- returning -110 > Aug 29 19:24:03 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 19:24:03 redhat2 kernel: pvfs2_file_read: error writing to handle > 1048431, -- returning -110 > Aug 29 20:04:40 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Aug 29 20:04:40 redhat2 kernel: pvfs2_file_read: error writing to handle > 1046679, -- returning -110 > Sep 14 16:14:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:14:55 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 16:14:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:14:56 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 16:15:37 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:15:37 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 16:21:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:21:53 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 16:23:04 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:23:04 redhat2 kernel: pvfs2_file_read: error writing to handle > 1042058, -- returning -110 > Sep 14 16:27:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 16:27:53 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:15:39 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:15:39 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:15:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:15:52 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:16:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:16:09 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:16:10 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:16:10 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:21:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:21:52 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:27:59 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:27:59 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:28:12 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:28:12 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 17:39:26 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:39:26 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039033, -- returning -110 > Sep 14 17:43:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 17:43:06 redhat2 kernel: pvfs2_file_read: error writing to handle > 1043039, -- returning -110 > Sep 14 18:08:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:08:54 redhat2 kernel: pvfs2_file_read: error writing to handle > 1040923, -- returning -110 > Sep 14 18:18:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:18:53 redhat2 kernel: pvfs2_file_read: error writing to handle > 1040923, -- returning -110 > Sep 14 18:27:07 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:27:07 redhat2 kernel: pvfs2_file_read: error writing to handle > 1042057, -- returning -110 > Sep 14 18:33:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:33:52 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039041, -- returning -110 > Sep 14 18:33:53 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:33:53 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 18:34:01 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:34:01 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038496, -- returning -110 > Sep 14 18:50:06 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 18:50:06 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 19:03:36 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 19:03:36 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038489, -- returning -110 > Sep 14 20:34:49 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 20:34:49 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 20:34:51 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 20:34:51 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 20:34:56 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 20:34:56 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 21:16:55 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 21:16:55 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 21:25:52 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 21:25:52 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039037, -- returning -110 > Sep 14 21:33:09 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 21:33:09 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 21:52:30 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 21:52:30 redhat2 kernel: pvfs2_file_read: error writing to handle > 1039031, -- returning -110 > Sep 14 22:17:05 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 22:17:05 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 14 22:17:45 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Sep 14 22:17:45 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038485, -- returning -110 > Sep 30 14:29:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out > and retries exhausted. aborting attempt. > Sep 30 14:30:18 redhat2 last message repeated 2 times > Sep 30 14:35:14 redhat2 kernel: pvfs2: pvfs2_lookup -- wait timed out > and retries exhausted. aborting attempt. > Sep 30 14:36:18 redhat2 last message repeated 2 times > Nov 2 22:16:08 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Nov 2 22:16:08 redhat2 kernel: pvfs2_file_read: error writing to handle > 1038504, -- returning -110 > Nov 12 14:54:57 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036658, -- returning -2 > Nov 12 14:55:00 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036651, -- returning -2 > Nov 12 14:57:00 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036651, -- returning -2 > Nov 12 14:57:04 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036658, -- returning -2 > Nov 12 14:57:42 redhat2 last message repeated 2 times > Nov 12 15:00:08 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036651, -- returning -2 > Nov 12 15:00:08 redhat2 last message repeated 2 times > Dec 5 12:51:01 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:51:05 redhat2 kernel: printk: 1 messages suppressed. > Dec 5 12:51:20 redhat2 kernel: printk: 3 messages suppressed. > Dec 5 12:51:35 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:52:26 redhat2 kernel: printk: 1 messages suppressed. > Dec 5 12:53:29 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:53:36 redhat2 kernel: printk: 3 messages suppressed. > Dec 5 12:53:43 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:53:52 redhat2 kernel: printk: 4 messages suppressed. > Dec 5 12:54:05 redhat2 kernel: printk: 3 messages suppressed. > Dec 5 12:54:08 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:54:13 redhat2 kernel: printk: 2 messages suppressed. > Dec 5 12:54:19 redhat2 kernel: printk: 5 messages suppressed. > Dec 5 12:54:36 redhat2 kernel: printk: 4 messages suppressed. > Dec 5 12:54:44 redhat2 kernel: printk: 5 messages suppressed. > Dec 5 12:54:46 redhat2 kernel: printk: 1 messages suppressed. > Dec 5 12:55:00 redhat2 kernel: printk: 1 messages suppressed. > Dec 5 12:55:02 redhat2 kernel: printk: 3 messages suppressed. > Dec 5 12:55:07 redhat2 kernel: printk: 1 messages suppressed. > Dec 5 12:55:38 redhat2 kernel: printk: 3 messages suppressed. > Dec 24 16:54:54 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Dec 24 16:54:54 redhat2 kernel: pvfs2_file_read: error writing to handle > 1042026, -- returning -110 > Dec 24 17:04:46 redhat2 kernel: pvfs2: pvfs2_file_read -- wait timed out > and retries exhausted. aborting attempt. > Dec 24 17:04:46 redhat2 kernel: pvfs2_file_read: error writing to handle > 1036052, -- returning -110 > Dec 26 10:49:33 redhat2 sshd(pam_unix)[15164]: authentication failure; > logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133 user=root > Dec 26 10:50:51 redhat2 sshd(pam_unix)[15164]: 1 more authentication > failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.130.222.133 > user=root > Dec 28 14:13:48 redhat2 kernel: printk: 1 messages suppressed. > Dec 28 14:14:26 redhat2 last message repeated 4 times > Dec 28 14:14:37 redhat2 kernel: printk: 2 messages suppressed. > Dec 28 14:14:46 redhat2 kernel: printk: 1 messages suppressed. > Dec 28 14:14:57 redhat2 kernel: printk: 2 messages suppressed. > Dec 28 14:42:22 redhat2 kernel: printk: 2 messages suppressed. > Dec 31 20:36:57 redhat2 kernel: Unable to handle kernel paging request > at virtual address 67c38e12 > Dec 31 20:36:57 redhat2 kernel: printing eip: > Dec 31 20:36:57 redhat2 kernel: f89e363e > Dec 31 20:36:57 redhat2 kernel: *pde = 00000000 > Dec 31 20:36:57 redhat2 kernel: Oops: 0000 [#1] > Dec 31 20:36:57 redhat2 kernel: SMP > Dec 31 20:36:57 redhat2 kernel: Modules linked in: ip_vs pvfs2(U) md5 > ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc dm_mirror > dm_multipath dm_mod button battery ac uhci_hcd e1000 floppy ext3 jbd > qla2300 qla2xxx scsi_transport_fc mptscsih mptbase sd_mod scsi_mod > Dec 31 20:36:57 redhat2 kernel: CPU: 2 > Dec 31 20:36:57 redhat2 kernel: EIP: 0060:[<f89e363e>] Not tainted VLI > Dec 31 20:36:57 redhat2 kernel: EFLAGS: 00010246 (2.6.9-22.ELsmp) > Dec 31 20:36:57 redhat2 kernel: EIP is at > pvfs2_devreq_writev+0x184/0x59b [pvfs2] > Dec 31 20:36:57 redhat2 kernel: eax: f5d66710 ebx: 67c38e12 ecx: > 000001fd edx: f6a00000 > Dec 31 20:36:57 redhat2 kernel: esi: f5d66700 edi: fffffe50 ebp: > f56cced4 esp: f56cceb8 > Dec 31 20:36:57 redhat2 kernel: ds: 007b es: 007b ss: 0068 > Dec 31 20:36:57 redhat2 kernel: Process pvfs2-client-co (pid: 4151, > threadinfo=f56cc000 task=f6ff00b0) > Dec 31 20:36:57 redhat2 kernel: Stack: f56cceb4 000022a8 00000004 > dd4fc42c dd4fc41c 00000004 f56ccf40 80000000 > Dec 31 20:36:57 redhat2 kernel: 00000000 f765fe80 00000000 > c02759c6 00000040 00000000 00000000 f56ccf40 > Dec 31 20:36:57 redhat2 kernel: 00000001 00000000 00000000 > 00000040 f63ccf48 00000a98 c0275a16 00000246 > Dec 31 20:36:57 redhat2 kernel: Call Trace: > Dec 31 20:36:57 redhat2 kernel: [<c02759c6>] sock_readv_writev+0x5e/0x81 > Dec 31 20:36:57 redhat2 kernel: [<c0275a16>] sock_readv+0x2d/0x34 > Dec 31 20:36:57 redhat2 kernel: [<f89e34ba>] > pvfs2_devreq_writev+0x0/0x59b [pvfs2] > Dec 31 20:36:57 redhat2 kernel: [<c015a177>] do_readv_writev+0x19c/0x21d > Dec 31 20:36:57 redhat2 kernel: [<c015a276>] vfs_writev+0x3e/0x43 > Dec 31 20:36:57 redhat2 kernel: [<c015a319>] sys_writev+0x3c/0x62 > Dec 31 20:36:57 redhat2 kernel: [<c02d10cf>] syscall_call+0x7/0xb > Dec 31 20:36:57 redhat2 kernel: Code: b9 ff ff ff 5e e9 2e 04 00 00 8b > 35 c4 5a 9f f8 8d 6c 24 1c 89 e8 8b 56 04 ff 56 0c 89 c7 8d 46 10 e8 0c > c6 8e c7 8b 16 8b 1c fa <8b> 03 0f 18 00 90 8d 04 fa 39 c3 74 1c 89 da > 89 e8 ff 56 08 8b > Dec 31 20:36:57 redhat2 kernel: <0>Fatal exception: panic in 5 seconds > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
