Am Wednesday, 20. August 2008 11:52 schrieb Mike Christie: > Klemens Kittan wrote: > > Am Wednesday, 20. August 2008 09:43 schrieb Mike Christie: > >> Klemens Kittan wrote: > >>> Am Tuesday, 19. August 2008 19:15 schrieb Mike Christie: > >>>> Klemens Kittan wrote: > >>>>> Am Monday, 18. August 2008 20:10 schrieb Mike Christie: > >>>>>> Klemens Kittan wrote: > >>>>>>> Am Friday, 15. August 2008 20:03 schrieb Mike Christie: > >>>>>>>> Mike Christie wrote: > >>>>>>>>> Klemens Kittan wrote: > >>>>>>>>>> Here is the configuration of my debian kernel (2.6.25-2). > >>>>>>>>> > >>>>>>>>> Thanks. It looks like your target is responding to other IO, but > >>>>>>>>> did not respond to the ping quick enough so it timed out. Let me > >>>>>>>>> make a patch for you to test. I should hopefully have it later > >>>>>>>>> today. > >>>>>>>> > >>>>>>>> Try the attached patch over open-iscsi-2.0-869.2 tarball modules. > >>>>>>>> > >>>>>>>> To apply the patch untar and unzip the source then cd to the dir. > >>>>>>>> Then do: > >>>>>>>> > >>>>>>>> patch -p1 -i where-the-patch-is-saved/relax-ping-timer.patch > >>>>>>>> > >>>>>>>> Then do the normal make and make install. You will probably want > >>>>>>>> to reboot the box to make sure you are using the new modules. > >>>>>>> > >>>>>>> Unfortunately I got the same errors. > >>>>>> > >>>>>> Could you send the log output? > >>>>> > >>>>> Here is the /var/log/syslog. > >>>> > >>>> Shoot. For some reason that nop is just not finishing in a decent > >>>> amount of time. Could you try the attached patch. It gives the nop > >>>> even more time to complete and it spits out a bunch of debug info to > >>>> make sure open-iscsi did not leak the task. > >>> > >>> Unfortunately, the attached file is empty. > >> > >> Oh yeah, if you just log into the target and do not do any IO to the > >> disks. Do you see any messages like this: > >> > >> Aug 14 09:52:23 baltrum kernel: [81064.665749] connection2:0: ping > >> timeout of > >> 10 secs expired, last rx 4315069195, last ping 4315067926, now > >> 4315070426 Aug 14 09:52:23 baltrum kernel: [81064.669756] > >> connection2:0: detected conn > >> error (1011) > > > > I get these messages all the time (with and without IO traffic): > > you should get these. I am just worried about getting these > > > Aug 20 09:56:10 baltrum kernel: [168687.391990] connection1:0: ping > > timeout > > > of 10 secs with recv timeout of 5 secs expired last rx 4336967839, > > last ping > > > 4336967081, now 4336970339 task ffff81003797aac0 > > Aug 20 09:56:10 baltrum kernel: [168687.396001] connection1:0: > > detected conn > > > error (1011) > > when there is no IO traffic. > > > Aug 20 09:49:13 baltrum kernel: [168482.943791] send ffff8100f9c541c0 > > Aug 20 09:49:13 baltrum kernel: [168483.026754] send ffff81003797adc0 > > Aug 20 09:49:13 baltrum kernel: [168483.026817] iscsi_free_mgmt_task > > ffff8100f9c541c0 > > Aug 20 09:49:13 baltrum kernel: [168483.031189] iscsi_free_mgmt_task > > ffff81003797adc0 > > Aug 20 09:49:18 baltrum kernel: [168487.772859] send ffff8100f9c54140 > > Aug 20 09:49:18 baltrum kernel: [168488.018304] iscsi_free_mgmt_task > > ffff8100f9c54140 > > Aug 20 09:49:18 baltrum kernel: [168488.018342] send ffff81003797aac0 > > Aug 20 09:49:18 baltrum kernel: [168488.026632] iscsi_free_mgmt_task > > ffff81003797aac0 > > > > With IO traffic I get these messages: > > Could you give me a large chunk of the log? I need the stuff that > happened before this part. > > > Aug 20 09:56:10 baltrum kernel: [168687.391990] connection1:0: ping > > timeout of 10 secs with recv timeout of 5 secs expired last rx > > 4336967839, last ping 4336967081, now 4336970339 task ffff81003797aac0 > > Aug 20 09:56:10 baltrum kernel: [168687.396001] connection1:0: detected > > conn error (1011) > > Aug 20 09:56:10 baltrum iscsid: Kernel reported iSCSI connection 1:0 > > error (1011) state (3) > > Aug 20 09:56:14 baltrum iscsid: connection1:0 is operational after > > recovery (1 attempts) > > > > Thanks, > > Klemens >
Here is the whole log. Thanks, Klemens
Aug 20 06:25:10 baltrum syslogd 1.5.0#5: restart. Aug 20 06:31:01 baltrum /USR/SBIN/CRON[17699]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 06:51:33 baltrum -- MARK -- Aug 20 07:11:33 baltrum -- MARK -- Aug 20 07:17:01 baltrum /USR/SBIN/CRON[19082]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 07:31:01 baltrum /USR/SBIN/CRON[19505]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 07:51:33 baltrum -- MARK -- Aug 20 08:11:33 baltrum -- MARK -- Aug 20 08:17:01 baltrum /USR/SBIN/CRON[20887]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 08:19:32 baltrum multipathd: ift_backup: stop event checker thread Aug 20 08:19:32 baltrum multipathd: ift_vz: stop event checker thread Aug 20 08:19:32 baltrum multipathd: ift_mail: stop event checker thread Aug 20 08:19:32 baltrum multipathd: ift_sql: stop event checker thread Aug 20 08:19:32 baltrum multipathd: ift_www: stop event checker thread Aug 20 08:19:32 baltrum multipathd: ift_www: stop event checker thread Aug 20 08:19:41 baltrum kernel: [162890.258480] sd 4:0:0:0: [sdb] Synchronizing SCSI cache Aug 20 08:19:41 baltrum kernel: [163105.000312] sd 4:0:0:1: [sdd] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [162890.330664] sd 5:0:0:0: [sdc] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.000891] sd 4:0:0:2: [sdf] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.001228] sd 4:0:0:3: [sdg] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.004553] sd 4:0:0:4: [sdi] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.234605] sd 5:0:0:1: [sde] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.239171] sd 5:0:0:2: [sdh] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.242966] sd 5:0:0:3: [sdj] Synchronizing SCSI cache Aug 20 08:19:42 baltrum kernel: [163105.247935] sd 5:0:0:4: [sdk] Synchronizing SCSI cache Aug 20 08:19:42 baltrum iscsid: iscsid shutting down. Aug 20 08:31:01 baltrum /USR/SBIN/CRON[21161]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 08:51:33 baltrum -- MARK -- Aug 20 09:11:33 baltrum -- MARK -- Aug 20 09:17:01 baltrum /USR/SBIN/CRON[21166]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 09:31:01 baltrum /USR/SBIN/CRON[21169]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 09:49:01 baltrum kernel: [168474.394999] Loading iSCSI transport class v2.0-869. Aug 20 09:49:01 baltrum kernel: [168474.403942] iscsi: registered transport (tcp) Aug 20 09:49:01 baltrum iscsid: iSCSI logger with pid=21825 started! Aug 20 09:49:01 baltrum kernel: [168253.618852] scsi6 : iSCSI Initiator over TCP/IP Aug 20 09:49:01 baltrum kernel: [168474.729341] scsi7 : iSCSI Initiator over TCP/IP Aug 20 09:49:02 baltrum iscsid: transport class version 2.0-869. iscsid version 2.0-869 Aug 20 09:49:02 baltrum iscsid: iSCSI daemon with pid=21826 started! Aug 20 09:49:02 baltrum kernel: [168254.706578] scsi 7:0:0:0: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:02 baltrum kernel: [168475.812858] sd 7:0:0:0: [sdb] 2097152000 512-byte hardware sectors (1073742 MB) Aug 20 09:49:02 baltrum kernel: [168254.707617] scsi 6:0:0:0: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:03 baltrum kernel: [168475.816916] sd 7:0:0:0: [sdb] Write Protect is off Aug 20 09:49:03 baltrum kernel: [168475.820899] sd 7:0:0:0: [sdb] Mode Sense: 8f 00 00 08 Aug 20 09:49:03 baltrum kernel: [168254.848045] sd 6:0:0:0: [sdc] 2097152000 512-byte hardware sectors (1073742 MB) Aug 20 09:49:03 baltrum kernel: [168475.825003] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:03 baltrum kernel: [168475.825398] sd 7:0:0:0: [sdb] 2097152000 512-byte hardware sectors (1073742 MB) Aug 20 09:49:03 baltrum kernel: [168475.825708] sd 7:0:0:0: [sdb] Write Protect is off Aug 20 09:49:03 baltrum kernel: [168475.825711] sd 7:0:0:0: [sdb] Mode Sense: 8f 00 00 08 Aug 20 09:49:03 baltrum kernel: [168475.826155] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:03 baltrum kernel: [168475.826160] sdb: Aug 20 09:49:03 baltrum kernel: [168254.848865] sd 6:0:0:0: [sdc] Write Protect is off Aug 20 09:49:03 baltrum kernel: [168254.852051] sd 6:0:0:0: [sdc] Mode Sense: 8f 00 00 08 Aug 20 09:49:03 baltrum kernel: [168476.206649] sd 7:0:0:0: [sdb] Attached SCSI disk Aug 20 09:49:03 baltrum iscsid: connection2:0 is operational now Aug 20 09:49:03 baltrum iscsid: connection1:0 is operational now Aug 20 09:49:03 baltrum kernel: [168254.898397] sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:03 baltrum kernel: [168476.220588] scsi 7:0:0:1: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:03 baltrum kernel: [168476.221079] sd 7:0:0:1: [sdd] 1258291200 512-byte hardware sectors (644245 MB) Aug 20 09:49:03 baltrum kernel: [168476.221425] sd 7:0:0:1: [sdd] Write Protect is off Aug 20 09:49:03 baltrum kernel: [168476.221428] sd 7:0:0:1: [sdd] Mode Sense: 8f 00 00 08 Aug 20 09:49:03 baltrum kernel: [168476.221888] sd 7:0:0:1: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:03 baltrum kernel: [168476.222143] sd 7:0:0:1: [sdd] 1258291200 512-byte hardware sectors (644245 MB) Aug 20 09:49:03 baltrum kernel: [168476.222361] sd 7:0:0:1: [sdd] Write Protect is off Aug 20 09:49:03 baltrum kernel: [168476.222361] sd 7:0:0:1: [sdd] Mode Sense: 8f 00 00 08 Aug 20 09:49:03 baltrum kernel: [168476.222643] sd 7:0:0:1: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:03 baltrum kernel: [168476.222643] sdd: Aug 20 09:49:03 baltrum kernel: [168476.678434] sd 7:0:0:1: [sdd] Attached SCSI disk Aug 20 09:49:04 baltrum kernel: [168254.943165] sd 6:0:0:0: [sdc] 2097152000 512-byte hardware sectors (1073742 MB) Aug 20 09:49:04 baltrum kernel: [168476.732439] sd 6:0:0:0: [sdc] Write Protect is off Aug 20 09:49:04 baltrum kernel: [168476.735115] sd 6:0:0:0: [sdc] Mode Sense: 8f 00 00 08 Aug 20 09:49:04 baltrum kernel: [168476.738536] scsi 7:0:0:2: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:04 baltrum kernel: [168476.743126] sd 7:0:0:2: [sde] 838860800 512-byte hardware sectors (429497 MB) Aug 20 09:49:04 baltrum kernel: [168255.056001] sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:04 baltrum kernel: [168255.056009] sdc: Aug 20 09:49:04 baltrum kernel: [168476.751161] sd 7:0:0:2: [sde] Write Protect is off Aug 20 09:49:04 baltrum kernel: [168255.251065] sd 6:0:0:0: [sdc] Attached SCSI disk Aug 20 09:49:04 baltrum kernel: [168255.259284] scsi 6:0:0:1: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:04 baltrum kernel: [168255.259727] sd 6:0:0:1: [sdf] 1258291200 512-byte hardware sectors (644245 MB) Aug 20 09:49:04 baltrum kernel: [168255.259993] sd 6:0:0:1: [sdf] Write Protect is off Aug 20 09:49:04 baltrum kernel: [168255.259996] sd 6:0:0:1: [sdf] Mode Sense: 8f 00 00 08 Aug 20 09:49:04 baltrum kernel: [168255.260412] sd 6:0:0:1: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:04 baltrum kernel: [168255.260773] sd 6:0:0:1: [sdf] 1258291200 512-byte hardware sectors (644245 MB) Aug 20 09:49:04 baltrum kernel: [168255.261027] sd 6:0:0:1: [sdf] Write Protect is off Aug 20 09:49:04 baltrum kernel: [168255.261030] sd 6:0:0:1: [sdf] Mode Sense: 8f 00 00 08 Aug 20 09:49:04 baltrum kernel: [168255.261558] sd 6:0:0:1: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:04 baltrum kernel: [168255.261563] sdf: Aug 20 09:49:04 baltrum kernel: [168476.754648] sd 7:0:0:2: [sde] Mode Sense: 8f 00 00 08 Aug 20 09:49:05 baltrum kernel: [168476.756165] sd 7:0:0:2: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:05 baltrum kernel: [168255.840164] sd 6:0:0:1: [sdf] Attached SCSI disk Aug 20 09:49:05 baltrum kernel: [168255.844619] scsi 6:0:0:2: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:05 baltrum kernel: [168255.845072] sd 6:0:0:2: [sdg] 838860800 512-byte hardware sectors (429497 MB) Aug 20 09:49:05 baltrum kernel: [168255.845374] sd 6:0:0:2: [sdg] Write Protect is off Aug 20 09:49:05 baltrum kernel: [168255.845377] sd 6:0:0:2: [sdg] Mode Sense: 8f 00 00 08 Aug 20 09:49:05 baltrum kernel: [168255.845802] sd 6:0:0:2: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:05 baltrum kernel: [168255.846162] sd 6:0:0:2: [sdg] 838860800 512-byte hardware sectors (429497 MB) Aug 20 09:49:05 baltrum kernel: [168255.846420] sd 6:0:0:2: [sdg] Write Protect is off Aug 20 09:49:05 baltrum kernel: [168255.846423] sd 6:0:0:2: [sdg] Mode Sense: 8f 00 00 08 Aug 20 09:49:05 baltrum kernel: [168255.846818] sd 6:0:0:2: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:05 baltrum kernel: [168255.846823] sdg: Aug 20 09:49:05 baltrum kernel: [168256.408526] sd 6:0:0:2: [sdg] Attached SCSI disk Aug 20 09:49:05 baltrum kernel: [168476.762825] sd 7:0:0:2: [sde] 838860800 512-byte hardware sectors (429497 MB) Aug 20 09:49:05 baltrum kernel: [168476.871957] scsi 6:0:0:3: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:05 baltrum kernel: [168476.874750] sd 7:0:0:2: [sde] Write Protect is off Aug 20 09:49:05 baltrum kernel: [168476.878152] sd 7:0:0:2: [sde] Mode Sense: 8f 00 00 08 Aug 20 09:49:05 baltrum kernel: [168476.883652] sd 7:0:0:2: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:05 baltrum kernel: [168476.886119] sde: Aug 20 09:49:05 baltrum kernel: [168476.890298] sd 7:0:0:2: [sde] Attached SCSI disk Aug 20 09:49:05 baltrum kernel: [168476.898418] sd 6:0:0:3: [sdh] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:05 baltrum kernel: [168256.771337] scsi 7:0:0:3: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:05 baltrum kernel: [168476.910004] sd 6:0:0:3: [sdh] Write Protect is off Aug 20 09:49:05 baltrum kernel: [168476.910004] sd 6:0:0:3: [sdh] Mode Sense: 8f 00 00 08 Aug 20 09:49:06 baltrum kernel: [168477.010406] sd 6:0:0:3: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:06 baltrum kernel: [168256.881569] sd 7:0:0:3: [sdi] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:06 baltrum kernel: [168477.020412] sd 6:0:0:3: [sdh] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:06 baltrum kernel: [168477.020776] sd 6:0:0:3: [sdh] Write Protect is off Aug 20 09:49:06 baltrum kernel: [168477.020795] sd 6:0:0:3: [sdh] Mode Sense: 8f 00 00 08 Aug 20 09:49:06 baltrum kernel: [168256.883784] sd 7:0:0:3: [sdi] Write Protect is off Aug 20 09:49:06 baltrum kernel: [168477.163974] sd 6:0:0:3: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:06 baltrum kernel: [168477.163982] sdh: Aug 20 09:49:06 baltrum kernel: [168256.885593] sd 7:0:0:3: [sdi] Mode Sense: 8f 00 00 08 Aug 20 09:49:06 baltrum kernel: [168477.345336] sd 6:0:0:3: [sdh] Attached SCSI disk Aug 20 09:49:06 baltrum kernel: [168256.892376] sd 7:0:0:3: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:06 baltrum kernel: [168256.895561] sd 7:0:0:3: [sdi] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:06 baltrum kernel: [168256.895839] sd 7:0:0:3: [sdi] Write Protect is off Aug 20 09:49:06 baltrum kernel: [168256.895843] sd 7:0:0:3: [sdi] Mode Sense: 8f 00 00 08 Aug 20 09:49:06 baltrum kernel: [168256.896329] sd 7:0:0:3: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:06 baltrum kernel: [168256.896334] sdi: Aug 20 09:49:06 baltrum kernel: [168477.359809] scsi 6:0:0:4: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:06 baltrum kernel: [168257.253605] sd 7:0:0:3: [sdi] Attached SCSI disk Aug 20 09:49:06 baltrum kernel: [168257.345805] scsi 7:0:0:4: Direct-Access IFT A16E-G2130-4 361G PQ: 0 ANSI: 4 Aug 20 09:49:07 baltrum kernel: [168477.410156] sd 6:0:0:4: [sdj] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:07 baltrum kernel: [168257.350586] sd 7:0:0:4: [sdk] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:07 baltrum kernel: [168257.350841] sd 7:0:0:4: [sdk] Write Protect is off Aug 20 09:49:07 baltrum kernel: [168257.350845] sd 7:0:0:4: [sdk] Mode Sense: 8f 00 00 08 Aug 20 09:49:07 baltrum kernel: [168257.351242] sd 7:0:0:4: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:07 baltrum kernel: [168257.351579] sd 7:0:0:4: [sdk] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:07 baltrum kernel: [168257.351846] sd 7:0:0:4: [sdk] Write Protect is off Aug 20 09:49:07 baltrum kernel: [168257.351849] sd 7:0:0:4: [sdk] Mode Sense: 8f 00 00 08 Aug 20 09:49:07 baltrum kernel: [168257.352254] sd 7:0:0:4: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:07 baltrum kernel: [168257.352258] sdk: Aug 20 09:49:07 baltrum kernel: [168477.414016] sd 6:0:0:4: [sdj] Write Protect is off Aug 20 09:49:07 baltrum kernel: [168477.417080] sd 6:0:0:4: [sdj] Mode Sense: 8f 00 00 08 Aug 20 09:49:07 baltrum kernel: [168477.417630] sd 6:0:0:4: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:07 baltrum kernel: [168257.893763] sd 7:0:0:4: [sdk] Attached SCSI disk Aug 20 09:49:07 baltrum kernel: [168477.422119] sd 6:0:0:4: [sdj] 209715200 512-byte hardware sectors (107374 MB) Aug 20 09:49:07 baltrum kernel: [168477.434474] sd 6:0:0:4: [sdj] Write Protect is off Aug 20 09:49:07 baltrum kernel: [168477.437095] sd 6:0:0:4: [sdj] Mode Sense: 8f 00 00 08 Aug 20 09:49:07 baltrum kernel: [168477.454241] sd 6:0:0:4: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 20 09:49:07 baltrum kernel: [168477.457174] sdj: Aug 20 09:49:07 baltrum kernel: [168477.471063] sd 6:0:0:4: [sdj] Attached SCSI disk Aug 20 09:49:13 baltrum kernel: [168482.943791] send ffff8100f9c541c0 Aug 20 09:49:13 baltrum kernel: [168483.026754] send ffff81003797adc0 Aug 20 09:49:13 baltrum kernel: [168483.026817] iscsi_free_mgmt_task ffff8100f9c541c0 Aug 20 09:49:13 baltrum kernel: [168483.031189] iscsi_free_mgmt_task ffff81003797adc0 Aug 20 09:49:18 baltrum kernel: [168487.772859] send ffff8100f9c54140 Aug 20 09:49:18 baltrum kernel: [168488.018304] iscsi_free_mgmt_task ffff8100f9c54140 Aug 20 09:49:18 baltrum kernel: [168488.018342] send ffff81003797aac0 Aug 20 09:49:18 baltrum kernel: [168488.026632] iscsi_free_mgmt_task ffff81003797aac0 Aug 20 09:49:23 baltrum kernel: [168492.900298] send ffff8100f9c540c0 Aug 20 09:49:23 baltrum kernel: [168492.983627] iscsi_free_mgmt_task ffff8100f9c540c0 Aug 20 09:49:23 baltrum kernel: [168493.027258] send ffff81003797aec0 Aug 20 09:49:23 baltrum kernel: [168493.074237] iscsi_free_mgmt_task ffff81003797aec0 Aug 20 09:49:28 baltrum kernel: [168497.952427] send ffff8100d7c8af40 Aug 20 09:49:28 baltrum kernel: [168498.046838] iscsi_free_mgmt_task ffff8100d7c8af40 Aug 20 09:49:28 baltrum kernel: [168498.093262] send ffff81003797a3c0 Aug 20 09:49:28 baltrum kernel: [168498.141895] iscsi_free_mgmt_task ffff81003797a3c0 Aug 20 09:49:33 baltrum kernel: [168503.033715] send ffff8100d7c8aec0 Aug 20 09:49:33 baltrum kernel: [168503.086389] iscsi_free_mgmt_task ffff8100d7c8aec0 Aug 20 09:49:33 baltrum kernel: [168503.128135] send ffff81003797a340 Aug 20 09:49:33 baltrum kernel: [168503.180407] iscsi_free_mgmt_task ffff81003797a340 Aug 20 09:49:38 baltrum kernel: [168507.969469] send ffff8100d7c8ae40 Aug 20 09:49:38 baltrum kernel: [168508.121405] iscsi_free_mgmt_task ffff8100d7c8ae40 Aug 20 09:49:38 baltrum kernel: [168508.161152] send ffff81003797a8c0 Aug 20 09:49:38 baltrum kernel: [168508.215312] iscsi_free_mgmt_task ffff81003797a8c0 Aug 20 09:49:43 baltrum kernel: [168513.136133] send ffff8100d7c8adc0 Aug 20 09:49:43 baltrum kernel: [168513.190019] iscsi_free_mgmt_task ffff8100d7c8adc0 Aug 20 09:49:43 baltrum kernel: [168513.228843] send ffff81003797a940 Aug 20 09:49:43 baltrum kernel: [168513.282242] iscsi_free_mgmt_task ffff81003797a940 Aug 20 09:49:48 baltrum kernel: [168518.043989] send ffff8100d7c8ad40 Aug 20 09:49:48 baltrum kernel: [168518.245108] iscsi_free_mgmt_task ffff8100d7c8ad40 Aug 20 09:49:48 baltrum kernel: [168518.283844] send ffff81003797a7c0 Aug 20 09:49:48 baltrum kernel: [168518.332698] iscsi_free_mgmt_task ffff81003797a7c0 Aug 20 09:49:53 baltrum kernel: [168523.177365] send ffff8100d7c8acc0 Aug 20 09:49:53 baltrum kernel: [168523.301659] iscsi_free_mgmt_task ffff8100d7c8acc0 Aug 20 09:49:53 baltrum kernel: [168523.334712] send ffff81003797a840 Aug 20 09:49:53 baltrum kernel: [168523.388686] iscsi_free_mgmt_task ffff81003797a840 Aug 20 09:49:58 baltrum kernel: [168528.334051] send ffff8100d7c8ac40 Aug 20 09:49:58 baltrum kernel: [168528.377921] iscsi_free_mgmt_task ffff8100d7c8ac40 Aug 20 09:49:58 baltrum kernel: [168528.385927] send ffff81003797a6c0 Aug 20 09:49:58 baltrum kernel: [168528.439834] iscsi_free_mgmt_task ffff81003797a6c0 Aug 20 09:50:03 baltrum multipathd: ift_backup: event checker started Aug 20 09:50:03 baltrum multipathd: ift_vz: event checker started Aug 20 09:50:03 baltrum multipathd: ift_mail: event checker started Aug 20 09:50:03 baltrum multipathd: ift_sql: event checker started Aug 20 09:50:03 baltrum multipathd: ift_www: event checker started Aug 20 09:50:03 baltrum multipathd: path checkers start up Aug 20 09:50:08 baltrum kernel: [168538.272129] send ffff8100d7c8abc0 Aug 20 09:50:08 baltrum kernel: [168538.276122] send ffff81003797af40 Aug 20 09:50:08 baltrum kernel: [168538.349525] iscsi_free_mgmt_task ffff81003797af40 Aug 20 09:50:08 baltrum kernel: [168538.349572] iscsi_free_mgmt_task ffff8100d7c8abc0 Aug 20 09:50:10 baltrum kernel: [168320.807521] kjournald starting. Commit interval 5 seconds Aug 20 09:50:10 baltrum kernel: [168539.896925] EXT3 FS on dm-0, internal journal Aug 20 09:50:10 baltrum kernel: [168539.896925] EXT3-fs: mounted filesystem with ordered data mode. Aug 20 09:50:18 baltrum kernel: [168548.300460] send ffff8100d7c8ab40 Aug 20 09:50:18 baltrum kernel: [168548.356859] iscsi_free_mgmt_task ffff8100d7c8ab40 Aug 20 09:50:19 baltrum kernel: [168549.239126] send ffff81003797acc0 Aug 20 09:50:19 baltrum kernel: [168549.371491] iscsi_free_mgmt_task ffff81003797acc0 Aug 20 09:50:23 baltrum kernel: [168553.327692] send ffff8100d7c8aac0 Aug 20 09:50:23 baltrum kernel: [168553.412095] iscsi_free_mgmt_task ffff8100d7c8aac0 Aug 20 09:51:15 baltrum kernel: [168605.883567] send ffff8100d7c8aa40 Aug 20 09:51:21 baltrum kernel: [168612.261500] send ffff81003797a740 Aug 20 09:51:21 baltrum kernel: [168612.317311] iscsi_free_mgmt_task ffff81003797a740 Aug 20 09:51:22 baltrum kernel: [168613.053963] iscsi_free_mgmt_task ffff8100d7c8aa40 Aug 20 09:51:33 baltrum kernel: [168623.545693] send ffff8100d7c8a9c0 Aug 20 09:51:34 baltrum kernel: [168624.961758] send ffff81003797a5c0 Aug 20 09:51:34 baltrum kernel: [168625.012319] iscsi_free_mgmt_task ffff81003797a5c0 Aug 20 09:51:40 baltrum kernel: [168630.691470] iscsi_free_mgmt_task ffff8100d7c8a9c0 Aug 20 09:51:52 baltrum kernel: [168642.539117] send ffff8100f9c541c0 Aug 20 09:51:52 baltrum kernel: [168642.681662] send ffff81003797abc0 Aug 20 09:51:52 baltrum kernel: [168642.723073] iscsi_free_mgmt_task ffff81003797abc0 Aug 20 09:51:57 baltrum kernel: [168647.689330] iscsi_free_mgmt_task ffff8100f9c541c0 Aug 20 09:52:07 baltrum kernel: [168658.178166] send ffff81003797adc0 Aug 20 09:52:14 baltrum kernel: [168664.623784] send ffff8100f9c54140 Aug 20 09:52:14 baltrum kernel: [168664.672175] iscsi_free_mgmt_task ffff8100f9c54140 Aug 20 09:52:16 baltrum kernel: [168667.329751] iscsi_free_mgmt_task ffff81003797adc0 Aug 20 09:52:19 baltrum kernel: [168669.642483] send ffff8100f9c540c0 Aug 20 09:52:19 baltrum kernel: [168669.718309] iscsi_free_mgmt_task ffff8100f9c540c0 Aug 20 09:52:21 baltrum kernel: [168672.354631] send ffff81003797aac0 Aug 20 09:52:28 baltrum kernel: [168678.749898] send ffff8100d7c8af40 Aug 20 09:52:28 baltrum kernel: [168678.788231] iscsi_free_mgmt_task ffff8100d7c8af40 Aug 20 09:52:32 baltrum kernel: [168683.583784] iscsi_free_mgmt_task ffff81003797aac0 Aug 20 09:52:33 baltrum kernel: [168683.803241] send ffff8100d7c8aec0 Aug 20 09:52:33 baltrum kernel: [168683.853990] iscsi_free_mgmt_task ffff8100d7c8aec0 Aug 20 09:52:37 baltrum kernel: [168688.612688] send ffff81003797aec0 Aug 20 09:52:38 baltrum kernel: [168688.825975] send ffff8100d7c8ae40 Aug 20 09:52:38 baltrum kernel: [168688.916739] iscsi_free_mgmt_task ffff8100d7c8ae40 Aug 20 09:52:43 baltrum kernel: [168693.749117] iscsi_free_mgmt_task ffff81003797aec0 Aug 20 09:52:43 baltrum kernel: [168693.916880] send ffff8100d7c8adc0 Aug 20 09:52:43 baltrum kernel: [168693.947548] iscsi_free_mgmt_task ffff8100d7c8adc0 Aug 20 09:52:52 baltrum kernel: [168702.721532] send ffff81003797a3c0 Aug 20 09:52:56 baltrum kernel: [168706.976136] send ffff8100d7c8ad40 Aug 20 09:52:56 baltrum kernel: [168707.037661] iscsi_free_mgmt_task ffff8100d7c8ad40 Aug 20 09:53:07 baltrum kernel: [168717.891536] iscsi_free_mgmt_task ffff81003797a3c0 Aug 20 09:53:11 baltrum kernel: [168721.637083] send ffff8100d7c8acc0 Aug 20 09:53:11 baltrum kernel: [168721.734170] iscsi_free_mgmt_task ffff8100d7c8acc0 Aug 20 09:53:12 baltrum kernel: [168722.864522] send ffff81003797a340 Aug 20 09:53:16 baltrum kernel: [168726.761358] send ffff8100d7c8ac40 Aug 20 09:53:16 baltrum kernel: [168726.794344] iscsi_free_mgmt_task ffff8100d7c8ac40 Aug 20 09:53:17 baltrum kernel: [168728.052204] iscsi_free_mgmt_task ffff81003797a340 Aug 20 09:53:27 baltrum kernel: [168738.018142] send ffff8100d7c8abc0 Aug 20 09:53:34 baltrum kernel: [168744.860799] send ffff81003797a8c0 Aug 20 09:53:34 baltrum kernel: [168744.897867] iscsi_free_mgmt_task ffff81003797a8c0 Aug 20 09:53:34 baltrum kernel: [168745.171182] iscsi_free_mgmt_task ffff8100d7c8abc0 Aug 20 09:53:40 baltrum kernel: [168751.010612] send ffff81003797a940 Aug 20 09:53:40 baltrum kernel: [168535.354059] iscsi_free_mgmt_task ffff81003797a940 Aug 20 09:53:40 baltrum kernel: [168751.071397] send ffff8100d7c8ab40 Aug 20 09:53:45 baltrum kernel: [168540.233129] send ffff81003797a7c0 Aug 20 09:53:45 baltrum kernel: [168756.095837] iscsi_free_mgmt_task ffff81003797a7c0 Aug 20 09:53:47 baltrum kernel: [168758.144615] iscsi_free_mgmt_task ffff8100d7c8ab40 Aug 20 09:53:55 baltrum kernel: [168766.603052] send ffff81003797a840 Aug 20 09:53:56 baltrum kernel: [168766.640582] iscsi_free_mgmt_task ffff81003797a840 Aug 20 09:53:56 baltrum kernel: [168767.348579] send ffff8100d7c8aac0 Aug 20 09:54:01 baltrum kernel: [168771.634086] send ffff81003797a6c0 Aug 20 09:54:01 baltrum kernel: [168771.695469] iscsi_free_mgmt_task ffff81003797a6c0 Aug 20 09:54:06 baltrum kernel: [168776.619318] send ffff81003797af40 Aug 20 09:54:06 baltrum kernel: [168776.729804] iscsi_free_mgmt_task ffff81003797af40 Aug 20 09:54:09 baltrum kernel: [168780.492060] iscsi_free_mgmt_task ffff8100d7c8aac0 Aug 20 09:54:19 baltrum kernel: [168790.429060] send ffff81003797acc0 Aug 20 09:54:21 baltrum kernel: [168792.407677] send ffff8100d7c8aa40 Aug 20 09:54:21 baltrum kernel: [168792.441109] iscsi_free_mgmt_task ffff8100d7c8aa40 Aug 20 09:54:26 baltrum kernel: [168797.590901] iscsi_free_mgmt_task ffff81003797acc0 Aug 20 09:54:34 baltrum kernel: [168805.424655] send ffff81003797a740 Aug 20 09:54:35 baltrum kernel: [168806.473366] send ffff8100d7c8a9c0 Aug 20 09:54:35 baltrum kernel: [168806.515723] iscsi_free_mgmt_task ffff8100d7c8a9c0 Aug 20 09:54:40 baltrum kernel: [168811.480432] send ffff8100f9c541c0 Aug 20 09:54:40 baltrum kernel: [168811.557861] iscsi_free_mgmt_task ffff8100f9c541c0 Aug 20 09:54:41 baltrum kernel: [168812.549483] iscsi_free_mgmt_task ffff81003797a740 Aug 20 09:54:47 baltrum kernel: [168818.662026] send ffff81003797a5c0 Aug 20 09:54:53 baltrum kernel: [168823.791775] send ffff8100f9c54140 Aug 20 09:54:53 baltrum kernel: [168823.908260] iscsi_free_mgmt_task ffff8100f9c54140 Aug 20 09:54:58 baltrum kernel: [168828.913111] send ffff8100f9c540c0 Aug 20 09:54:58 baltrum kernel: [168828.946669] iscsi_free_mgmt_task ffff8100f9c540c0 Aug 20 09:55:01 baltrum kernel: [168831.738889] connection1:0: ping timeout of 10 secs with recv timeout of 5 secs expired last rx 4336950488, last ping 4336949731, now 4336952988 task ffff81003797a5c0 Aug 20 09:55:01 baltrum kernel: [168831.742893] connection1:0: detected conn error (1011) Aug 20 09:55:01 baltrum kernel: [168832.010455] iscsi_free_mgmt_task ffff81003797a5c0 Aug 20 09:55:01 baltrum kernel: [168832.010468] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum kernel: [168832.014459] end_request: I/O error, dev sdf, sector 4519328 Aug 20 09:55:01 baltrum kernel: [168832.018461] device-mapper: multipath: Failing path 8:80. Aug 20 09:55:01 baltrum kernel: [168832.022470] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum kernel: [168832.030463] end_request: I/O error, dev sdf, sector 4519072 Aug 20 09:55:01 baltrum kernel: [168832.034497] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum kernel: [168832.038461] end_request: I/O error, dev sdf, sector 4403880 Aug 20 09:55:01 baltrum kernel: [168832.042528] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum kernel: [168832.046461] end_request: I/O error, dev sdf, sector 4404904 Aug 20 09:55:01 baltrum kernel: [168832.050528] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum kernel: [168832.058462] end_request: I/O error, dev sdf, sector 4405928 Aug 20 09:55:01 baltrum kernel: [168832.062545] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:01 baltrum iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Aug 20 09:55:01 baltrum kernel: [168832.066462] end_request: I/O error, dev sdf, sector 4406952 Aug 20 09:55:01 baltrum multipathd: 8:80: mark as failed Aug 20 09:55:01 baltrum multipathd: ift_vz: remaining active paths: 1 Aug 20 09:55:02 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:55:02 baltrum multipathd: dm-0: devmap already registered Aug 20 09:55:05 baltrum multipathd: sdf: directio checker reports path is up Aug 20 09:55:05 baltrum multipathd: 8:80: reinstated Aug 20 09:55:05 baltrum multipathd: ift_vz: remaining active paths: 2 Aug 20 09:55:05 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:55:05 baltrum multipathd: dm-0: devmap already registered Aug 20 09:55:05 baltrum iscsid: connection1:0 is operational after recovery (1 attempts) Aug 20 09:55:14 baltrum kernel: [168844.722299] send ffff81003797abc0 Aug 20 09:55:15 baltrum kernel: [168845.194086] send ffff8100d7c8af40 Aug 20 09:55:15 baltrum kernel: [168845.344355] iscsi_free_mgmt_task ffff8100d7c8af40 Aug 20 09:55:20 baltrum kernel: [168850.354034] send ffff8100d7c8aec0 Aug 20 09:55:20 baltrum kernel: [168850.415232] iscsi_free_mgmt_task ffff8100d7c8aec0 Aug 20 09:55:29 baltrum kernel: [168859.866599] connection1:0: ping timeout of 10 secs with recv timeout of 5 secs expired last rx 4336957628, last ping 4336956371, now 4336960128 task ffff81003797abc0 Aug 20 09:55:29 baltrum kernel: [168859.870604] connection1:0: detected conn error (1011) Aug 20 09:55:30 baltrum kernel: [168860.135219] iscsi_free_mgmt_task ffff81003797abc0 Aug 20 09:55:30 baltrum kernel: [168860.135232] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.139223] end_request: I/O error, dev sdf, sector 5228784 Aug 20 09:55:30 baltrum kernel: [168860.143225] device-mapper: multipath: Failing path 8:80. Aug 20 09:55:30 baltrum kernel: [168860.147241] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.155224] end_request: I/O error, dev sdf, sector 5228528 Aug 20 09:55:30 baltrum kernel: [168860.159247] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.163226] end_request: I/O error, dev sdf, sector 5115584 Aug 20 09:55:30 baltrum kernel: [168860.167291] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.171225] end_request: I/O error, dev sdf, sector 5185784 Aug 20 09:55:30 baltrum kernel: [168860.175286] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.179226] end_request: I/O error, dev sdf, sector 5194544 Aug 20 09:55:30 baltrum kernel: [168860.183296] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:55:30 baltrum kernel: [168860.191226] end_request: I/O error, dev sdf, sector 5195568 Aug 20 09:55:30 baltrum multipathd: 8:80: mark as failed Aug 20 09:55:30 baltrum multipathd: ift_vz: remaining active paths: 1 Aug 20 09:55:30 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:55:30 baltrum multipathd: dm-0: devmap already registered Aug 20 09:55:30 baltrum iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Aug 20 09:55:34 baltrum iscsid: connection1:0 is operational after recovery (1 attempts) Aug 20 09:55:34 baltrum multipathd: sdf: directio checker reports path is up Aug 20 09:55:34 baltrum multipathd: 8:80: reinstated Aug 20 09:55:34 baltrum multipathd: ift_vz: remaining active paths: 2 Aug 20 09:55:34 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:55:34 baltrum multipathd: dm-0: devmap already registered Aug 20 09:55:43 baltrum kernel: [168872.607777] send ffff81003797adc0 Aug 20 09:55:52 baltrum kernel: [168669.219239] iscsi_free_mgmt_task ffff81003797adc0 Aug 20 09:55:53 baltrum kernel: [168883.425269] send ffff8100d7c8ae40 Aug 20 09:55:54 baltrum kernel: [168883.519765] iscsi_free_mgmt_task ffff8100d7c8ae40 Aug 20 09:55:57 baltrum kernel: [168674.250920] send ffff81003797aac0 Aug 20 09:56:01 baltrum kernel: [168890.668875] send ffff8100d7c8adc0 Aug 20 09:56:01 baltrum kernel: [168890.760003] iscsi_free_mgmt_task ffff8100d7c8adc0 Aug 20 09:56:10 baltrum kernel: [168687.391990] connection1:0: ping timeout of 10 secs with recv timeout of 5 secs expired last rx 4336967839, last ping 4336967081, now 4336970339 task ffff81003797aac0 Aug 20 09:56:10 baltrum kernel: [168687.396001] connection1:0: detected conn error (1011) Aug 20 09:56:10 baltrum iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Aug 20 09:56:10 baltrum kernel: [168900.456869] iscsi_free_mgmt_task ffff81003797aac0 Aug 20 09:56:10 baltrum kernel: [168900.456880] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:56:10 baltrum kernel: [168900.460876] end_request: I/O error, dev sdf, sector 6009960 Aug 20 09:56:10 baltrum kernel: [168900.464875] device-mapper: multipath: Failing path 8:80. Aug 20 09:56:10 baltrum kernel: [168900.468896] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:56:11 baltrum kernel: [168900.476880] end_request: I/O error, dev sdf, sector 5618904 Aug 20 09:56:11 baltrum kernel: [168900.480956] sd 6:0:0:1: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK Aug 20 09:56:11 baltrum kernel: [168900.488875] end_request: I/O error, dev sdf, sector 6009704 Aug 20 09:56:11 baltrum multipathd: 8:80: mark as failed Aug 20 09:56:11 baltrum multipathd: ift_vz: remaining active paths: 1 Aug 20 09:56:11 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:56:11 baltrum multipathd: dm-0: devmap already registered Aug 20 09:56:14 baltrum iscsid: connection1:0 is operational after recovery (1 attempts) Aug 20 09:56:15 baltrum multipathd: sdf: directio checker reports path is up Aug 20 09:56:15 baltrum multipathd: 8:80: reinstated Aug 20 09:56:15 baltrum multipathd: ift_vz: remaining active paths: 2 Aug 20 09:56:15 baltrum multipathd: dm-0: add map (uevent) Aug 20 09:56:15 baltrum multipathd: dm-0: devmap already registered Aug 20 09:56:26 baltrum kernel: [168915.415241] send ffff8100d7c8ad40 Aug 20 09:56:26 baltrum kernel: [168915.660178] send ffff81003797aec0 Aug 20 09:56:26 baltrum kernel: [168915.660685] iscsi_free_mgmt_task ffff81003797aec0 Aug 20 09:56:33 baltrum kernel: [168922.472994] iscsi_free_mgmt_task ffff8100d7c8ad40 Aug 20 09:56:39 baltrum kernel: [168928.294577] send ffff81003797a3c0 Aug 20 09:56:41 baltrum kernel: [168930.497632] send ffff8100d7c8acc0 Aug 20 09:56:41 baltrum kernel: [168930.573852] iscsi_free_mgmt_task ffff8100d7c8acc0 Aug 20 09:56:51 baltrum kernel: [168940.701618] send ffff8100d7c8ac40 Aug 20 09:56:51 baltrum kernel: [168940.756962] iscsi_free_mgmt_task ffff8100d7c8ac40 Aug 20 09:56:56 baltrum kernel: [168945.508192] iscsi_free_mgmt_task ffff81003797a3c0 Aug 20 09:57:05 baltrum kernel: [168954.594064] send ffff81003797a340 Aug 20 09:57:06 baltrum kernel: [168955.627727] iscsi_free_mgmt_task ffff81003797a340 Aug 20 09:57:18 baltrum kernel: [168968.129113] send ffff8100d7c8abc0 Aug 20 09:57:22 baltrum kernel: [168971.255600] iscsi_free_mgmt_task ffff8100d7c8abc0 Aug 20 09:57:22 baltrum kernel: [168971.255688] send ffff81003797a8c0 Aug 20 09:57:22 baltrum kernel: [168971.353936] iscsi_free_mgmt_task ffff81003797a8c0 Aug 20 09:57:26 baltrum kernel: [168976.167679] send ffff8100d7c8ab40 Aug 20 09:57:27 baltrum kernel: [168976.355536] send ffff81003797a940 Aug 20 09:57:27 baltrum kernel: [168976.413833] iscsi_free_mgmt_task ffff81003797a940 Aug 20 09:57:32 baltrum kernel: [168981.410107] iscsi_free_mgmt_task ffff8100d7c8ab40 Aug 20 09:57:32 baltrum kernel: [168981.410200] send ffff81003797a7c0 Aug 20 09:57:32 baltrum kernel: [168981.489594] iscsi_free_mgmt_task ffff81003797a7c0 Aug 20 09:57:51 baltrum kernel: [169000.456302] send ffff8100d7c8aac0 Aug 20 09:57:53 baltrum kernel: [169002.322846] send ffff81003797a840 Aug 20 09:57:53 baltrum kernel: [169002.411401] iscsi_free_mgmt_task ffff81003797a840 Aug 20 09:57:58 baltrum kernel: [169007.345511] send ffff81003797a6c0 Aug 20 09:57:58 baltrum kernel: [169007.468264] iscsi_free_mgmt_task ffff81003797a6c0 Aug 20 09:57:58 baltrum kernel: [169007.560092] iscsi_free_mgmt_task ffff8100d7c8aac0 Aug 20 09:58:29 baltrum kernel: [169039.343596] send ffff81003797af40 Aug 20 09:58:29 baltrum kernel: [169039.432063] iscsi_free_mgmt_task ffff81003797af40 Aug 20 09:59:16 baltrum kernel: [169087.493105] send ffff81003797acc0 Aug 20 09:59:16 baltrum kernel: [169087.497098] send ffff8100d7c8aa40 Aug 20 09:59:16 baltrum kernel: [169087.691949] iscsi_free_mgmt_task ffff8100d7c8aa40 Aug 20 09:59:21 baltrum kernel: [169092.565729] send ffff8100d7c8a9c0 Aug 20 09:59:21 baltrum kernel: [169092.735141] iscsi_free_mgmt_task ffff8100d7c8a9c0 Aug 20 09:59:23 baltrum kernel: [169094.749837] iscsi_free_mgmt_task ffff81003797acc0 Aug 20 09:59:32 baltrum kernel: [169103.940861] send ffff81003797a740 Aug 20 09:59:34 baltrum kernel: [169105.920965] send ffff8100f9c541c0 Aug 20 09:59:34 baltrum kernel: [169105.972549] iscsi_free_mgmt_task ffff8100f9c541c0 Aug 20 09:59:38 baltrum kernel: [169109.282156] iscsi_free_mgmt_task ffff81003797a740 Aug 20 10:00:08 baltrum kernel: [169139.756342] send ffff81003797a5c0 Aug 20 10:00:12 baltrum kernel: [169143.932340] send ffff8100f9c54140 Aug 20 10:00:12 baltrum kernel: [169143.991986] iscsi_free_mgmt_task ffff8100f9c54140 Aug 20 10:00:15 baltrum kernel: [169146.951134] iscsi_free_mgmt_task ffff81003797a5c0 Aug 20 10:00:25 baltrum kernel: [169156.530189] send ffff8100f9c540c0 Aug 20 10:00:27 baltrum kernel: [169158.459800] send ffff81003797abc0 Aug 20 10:00:27 baltrum kernel: [169158.562213] iscsi_free_mgmt_task ffff81003797abc0 Aug 20 10:00:30 baltrum kernel: [169161.708878] iscsi_free_mgmt_task ffff8100f9c540c0 Aug 20 10:00:43 baltrum kernel: [169174.610168] send ffff81003797adc0 Aug 20 10:00:46 baltrum kernel: [169177.645201] send ffff8100d7c8af40 Aug 20 10:00:46 baltrum kernel: [169177.697348] iscsi_free_mgmt_task ffff8100d7c8af40 Aug 20 10:00:52 baltrum kernel: [169183.817723] iscsi_free_mgmt_task ffff81003797adc0 Aug 20 10:00:58 baltrum kernel: [169190.201787] send ffff8100d7c8aec0 Aug 20 10:00:58 baltrum kernel: [169190.250285] iscsi_free_mgmt_task ffff8100d7c8aec0 Aug 20 10:00:59 baltrum kernel: [169190.406840] send ffff81003797aac0 Aug 20 10:01:03 baltrum kernel: [169195.245844] send ffff8100d7c8ae40 Aug 20 10:01:03 baltrum kernel: [169195.287008] iscsi_free_mgmt_task ffff8100d7c8ae40 Aug 20 10:01:04 baltrum kernel: [169195.521349] iscsi_free_mgmt_task ffff81003797aac0 Aug 20 10:01:18 baltrum kernel: [169210.199905] send ffff81003797aec0 Aug 20 10:01:20 baltrum kernel: [169212.146512] send ffff8100d7c8adc0 Aug 20 10:01:20 baltrum kernel: [169212.215097] iscsi_free_mgmt_task ffff8100d7c8adc0 Aug 20 10:01:25 baltrum kernel: [169217.225460] send ffff8100d7c8ad40 Aug 20 10:01:25 baltrum kernel: [169217.262020] iscsi_free_mgmt_task ffff8100d7c8ad40 Aug 20 10:01:25 baltrum kernel: [169217.363492] iscsi_free_mgmt_task ffff81003797aec0 Aug 20 10:01:32 baltrum kernel: [169224.422711] send ffff8100d7c8acc0 Aug 20 10:01:32 baltrum kernel: [169224.454411] send ffff81003797a3c0 Aug 20 10:01:32 baltrum kernel: [169224.488122] iscsi_free_mgmt_task ffff81003797a3c0 Aug 20 10:01:38 baltrum kernel: [169229.474747] iscsi_free_mgmt_task ffff8100d7c8acc0 Aug 20 10:01:43 baltrum kernel: [169234.937308] send ffff8100d7c8ac40 Aug 20 10:01:50 baltrum kernel: [169242.132483] iscsi_free_mgmt_task ffff8100d7c8ac40 Aug 20 10:02:04 baltrum kernel: [169256.005959] send ffff81003797a340 Aug 20 10:02:05 baltrum kernel: [169257.044170] iscsi_free_mgmt_task ffff81003797a340 Aug 20 10:02:17 baltrum kernel: [169269.220800] send ffff81003797a8c0 Aug 20 10:02:17 baltrum kernel: [169269.316397] iscsi_free_mgmt_task ffff81003797a8c0 Aug 20 10:02:17 baltrum kernel: [169269.346068] send ffff8100d7c8abc0 Aug 20 10:02:22 baltrum kernel: [169274.278082] send ffff81003797a940 Aug 20 10:02:22 baltrum kernel: [169274.363320] iscsi_free_mgmt_task ffff81003797a940 Aug 20 10:02:24 baltrum kernel: [169276.544359] iscsi_free_mgmt_task ffff8100d7c8abc0 Aug 20 10:02:52 baltrum kernel: [169306.180957] send ffff81003797a7c0 Aug 20 10:02:52 baltrum kernel: [169306.262090] iscsi_free_mgmt_task ffff81003797a7c0 Aug 20 10:02:53 baltrum kernel: [169307.038731] send ffff8100d7c8ab40 Aug 20 10:02:53 baltrum kernel: [169307.076489] iscsi_free_mgmt_task ffff8100d7c8ab40 Aug 20 10:03:44 baltrum kernel: [169358.687992] send ffff8100d7c8aac0 Aug 20 10:03:44 baltrum kernel: [169358.877531] send ffff81003797a840 Aug 20 10:03:44 baltrum kernel: [169358.917238] iscsi_free_mgmt_task ffff81003797a840 Aug 20 10:03:49 baltrum kernel: [169363.899484] send ffff81003797a6c0 Aug 20 10:03:49 baltrum kernel: [169363.966443] iscsi_free_mgmt_task ffff81003797a6c0 Aug 20 10:03:55 baltrum kernel: [169370.128121] send ffff81003797af40 Aug 20 10:03:55 baltrum kernel: [169370.163740] iscsi_free_mgmt_task ffff81003797af40 Aug 20 10:04:00 baltrum kernel: [169375.101647] send ffff81003797acc0 Aug 20 10:04:00 baltrum kernel: [169375.189138] iscsi_free_mgmt_task ffff81003797acc0 Aug 20 10:04:01 baltrum kernel: [169375.973788] iscsi_free_mgmt_task ffff8100d7c8aac0 Aug 20 10:04:08 baltrum kernel: [169383.002067] send ffff8100d7c8aa40 Aug 20 10:04:15 baltrum kernel: [169390.169344] iscsi_free_mgmt_task ffff8100d7c8aa40 Aug 20 10:04:16 baltrum kernel: [169390.340098] send ffff81003797a740 Aug 20 10:04:16 baltrum kernel: [169390.395316] iscsi_free_mgmt_task ffff81003797a740 Aug 20 10:04:23 baltrum kernel: [169397.745373] send ffff81003797a5c0 Aug 20 10:04:23 baltrum kernel: [169397.826135] iscsi_free_mgmt_task ffff81003797a5c0 Aug 20 10:04:28 baltrum kernel: [169402.850973] send ffff81003797abc0 Aug 20 10:04:28 baltrum kernel: [169402.887905] iscsi_free_mgmt_task ffff81003797abc0 Aug 20 10:04:29 baltrum kernel: [169403.570332] send ffff8100d7c8a9c0 Aug 20 10:04:29 baltrum kernel: [169403.615811] iscsi_free_mgmt_task ffff8100d7c8a9c0 Aug 20 10:04:36 baltrum kernel: [169410.367365] send ffff81003797adc0 Aug 20 10:04:36 baltrum kernel: [169410.473623] iscsi_free_mgmt_task ffff81003797adc0 Aug 20 10:04:36 baltrum kernel: [169410.867503] send ffff8100f9c541c0 Aug 20 10:04:36 baltrum kernel: [169410.966570] iscsi_free_mgmt_task ffff8100f9c541c0 Aug 20 10:04:39 baltrum kernel: [169413.728490] multipathd[22351]: segfault at a ip 7f15257a5a3b sp 7fff2e751f30 error 4 in libc-2.7.so[7f1525732000+14a000] Aug 20 10:04:41 baltrum kernel: [169415.385972] send ffff81003797aac0 Aug 20 10:04:41 baltrum kernel: [169415.501789] iscsi_free_mgmt_task ffff81003797aac0 Aug 20 10:04:41 baltrum kernel: [169415.952598] send ffff8100f9c54140 Aug 20 10:04:41 baltrum kernel: [169415.997188] iscsi_free_mgmt_task ffff8100f9c54140 Aug 20 10:04:46 baltrum kernel: [169420.477801] send ffff81003797aec0 Aug 20 10:04:46 baltrum kernel: [169420.526048] iscsi_free_mgmt_task ffff81003797aec0 Aug 20 10:04:46 baltrum kernel: [169420.968318] send ffff8100f9c540c0 Aug 20 10:04:46 baltrum kernel: [169421.022158] iscsi_free_mgmt_task ffff8100f9c540c0 Aug 20 10:04:49 baltrum kernel: [169423.788237] sd 6:0:0:0: [sdc] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.788872] sd 6:0:0:1: [sdf] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.792705] sd 6:0:0:2: [sdg] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.796637] sd 6:0:0:3: [sdh] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.800612] sd 6:0:0:4: [sdj] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.861349] sd 7:0:0:0: [sdb] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.861741] sd 7:0:0:1: [sdd] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.865756] sd 7:0:0:2: [sde] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.873359] sd 7:0:0:3: [sdi] Synchronizing SCSI cache Aug 20 10:04:49 baltrum kernel: [169423.877775] sd 7:0:0:4: [sdk] Synchronizing SCSI cache Aug 20 10:04:50 baltrum iscsid: iscsid shutting down. Aug 20 10:11:35 baltrum smartd[3035]: Device: /dev/sda, SMART Prefailure Attribute: 8 Seek_Time_Performance changed from 247 to 248 Aug 20 10:17:01 baltrum /USR/SBIN/CRON[23002]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 10:31:02 baltrum /USR/SBIN/CRON[23005]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 10:51:33 baltrum -- MARK -- Aug 20 11:11:33 baltrum -- MARK -- Aug 20 11:17:01 baltrum /USR/SBIN/CRON[23010]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 11:31:01 baltrum /USR/SBIN/CRON[23014]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 11:41:35 baltrum smartd[3035]: Device: /dev/sda, SMART Prefailure Attribute: 8 Seek_Time_Performance changed from 248 to 249 Aug 20 12:11:33 baltrum -- MARK -- Aug 20 12:17:01 baltrum /USR/SBIN/CRON[23028]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 12:31:01 baltrum /USR/SBIN/CRON[23031]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 12:51:33 baltrum -- MARK -- Aug 20 13:11:33 baltrum -- MARK -- Aug 20 13:17:01 baltrum /USR/SBIN/CRON[23037]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 13:29:12 baltrum upsmon[3062]: Room 2.28 00 Communication is lost. [N] Aug 20 13:31:01 baltrum /USR/SBIN/CRON[23046]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 13:51:33 baltrum -- MARK -- Aug 20 14:11:33 baltrum -- MARK -- Aug 20 14:17:01 baltrum /USR/SBIN/CRON[23189]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 20 14:31:01 baltrum /USR/SBIN/CRON[23234]: (clamav) CMD ([ -x /usr/bin/freshclam ] && /usr/bin/freshclam --quiet >/dev/null) Aug 20 14:51:33 baltrum -- MARK -- Aug 20 15:11:33 baltrum -- MARK -- Aug 20 15:17:01 baltrum /USR/SBIN/CRON[23378]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
pgpaVCBCfzZMQ.pgp
Description: PGP signature
