Hi there,

We hit a iscsid die case. use ps -ef|grep iscsid, we only saw one iscsid
process.

Unfortunately, we didn't enable debug log so there are not many logs
available. We looked at the iscsid log it states that it received some
kernel reported error. Also, in the kernel log, there is OOP complaints.

 After that, we used open-iscsi init script to restart, but the kernel
modules were not been able to removed. We issue iscsiadm -m session, it
gives:
 r...@swe_1_ser_2:~# iscsiadm --mode session
tcp: [17] []:-1,1 \ufffd\ufffd?\ufffd|0&\ufffdHYw

The iscsiadm --mode session -P 3 hangs:

r...@swe_1_ser_2:~# iscsiadm --mode session -P 3
iSCSI Transport Class version 2.0-870
iscsiadm version 2.0-870
Target: \ufffd\ufffd?\u04cbA\ufffd\ufffd\ufffd?
    Current Portal: []:-1,1
    Persistent Portal: []:-1,1
        **********
        Interface:
        **********
        Iface Name: .bss
        Iface Transport: tcp
        Iface Initiatorname: iqn.1993-08.org.debian:01:f0de76895ed6
        Iface IPaddress: [192.168.1.99]
        Iface HWaddress: default
        Iface Netdev: default
        SID: 17


iscsid log:

Sep 29 13:00:05 swe_1_ser_2 iscsid: Kernel reported iSCSI connection 16:0
error (1011) state (3)
Sep 29 13:00:05 swe_1_ser_2 iscsid: Kernel reported iSCSI connection 17:0
error (1011) state (3)
 Sep 29 13:00:10 swe_1_ser_2 iscsid: connect failed (111)
Sep 29 13:00:10 swe_1_ser_2 iscsid: connect failed (111)
Sep 29 13:00:17 swe_1_ser_2 iscsid: connect failed (111)
Sep 29 13:00:17 swe_1_ser_2 iscsid: connect failed (111)
Sep 29 13:00:23 swe_1_ser_2 iscsid: connect failed (111)
Sep 29 13:00:23 swe_1_ser_2 iscsid: connect failed (111)

kernel log:
Sep 29 13:00:05 swe_1_ser_2 kernel: [ 8459.978988]  connection16:0: ping
timeout of 15 secs expired, last rx 2035865, last ping 2038365, now 2042115
Sep 29 13:00:05 swe_1_ser_2 kernel: [ 8459.979351]  connection16:0: detected
conn error (1011)
Sep 29 13:00:05 swe_1_ser_2 kernel: [ 8460.094535]  connection17:0: ping
timeout of 15 secs expired, last rx 2035894, last ping 2038394, now 2042144
Sep 29 13:00:05 swe_1_ser_2 kernel: [ 8460.094733]  connection17:0: detected
conn error (1011)
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.129711]  session16: session
recovery timed out after 120 secs
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.129860]  session17: session
recovery timed out after 120 secs
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133734] sd 373:0:0:75: [sdd]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133747] end_request: I/O error,
dev sdd, sector 1638144
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133753] Buffer I/O error on
device sdd, logical block 204768
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133873] sd 374:0:0:78: [sde]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133878] end_request: I/O error,
dev sde, sector 24
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.133881] Buffer I/O error on
device sde, logical block 3
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.134087] sd 373:0:0:75: [sdd]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.134093] end_request: I/O error,
dev sdd, sector 1638144
Sep 29 13:02:05 swe_1_ser_2 kernel: [ 8580.134097] Buffer I/O error on
device sdd, logical block 204768
Sep 29 13:02:23 swe_1_ser_2 kernel: [ 8597.655094] sd 374:0:0:78: [sde]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:02:23 swe_1_ser_2 kernel: [ 8597.655101] end_request: I/O error,
dev sde, sector 1638144
Sep 29 13:02:23 swe_1_ser_2 kernel: [ 8597.655349] sd 374:0:0:78: [sde]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:02:23 swe_1_ser_2 kernel: [ 8597.655356] end_request: I/O error,
dev sde, sector 0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.193745] BUG: unable to handle
kernel NULL pointer dereference at virtual address 00000060
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.193951] printing eip: e08ce12a
*pde = 00000000
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.194234] Oops: 0000 [#1] SMP
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.194407] Modules linked in:
ipt_REJECT iscsi_tcp libiscsi scsi_transport_iscsi iscsi_trgt crc32c
libcrc32c nls_iso8859_1 nls_cp437 vfat fat vmmemctl cpufreq_conservative
cpufreq_ondemand cpufreq_userspace cpufreq_stats freq_table
cpufreq_powersave sbs video output sbshc dock battery iptable_filter
ip_tables x_tables vmhgfs lp loop ipv6 container serio_raw ac i2c_piix4
button intel_agp i2c_core agpgart shpchp pci_hotplug parport_pc parport
evdev psmouse pcspkr ext3 jbd mbcache sr_mod cdrom pata_acpi ata_generic sg
sd_mod floppy pcnet32 ata_piix mii mptspi mptscsih mptbase
scsi_transport_spi libata scsi_mod raid10 raid456 async_xor async_memcpy
async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_snapshot
dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
vmxnet
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.195515]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.195623] Pid: 5839, comm: iscsid
Not tainted (2.6.24-24-generic #1)
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.195719] EIP: 0060:[<e08ce12a>]
EFLAGS: 00010202 CPU: 0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196041] EIP is at
spi_device_match+0x1a/0x60 [scsi_transport_spi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196146] EAX: 00000000 EBX:
dd49e8b0 ECX: dd49e800 EDX: dd49e8b0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196237] ESI: dd49e8b0 EDI:
df847200 EBP: c0286000 ESP: d1633b8c
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196329]  DS: 007b ES: 007b FS:
00d8 GS: 0033 SS: 0068
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196429] Process iscsid (pid:
5839, ti=d1632000 task=df82c000 task.ti=d1632000)
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196531] Stack: e08d3c90 c0285c8f
e095e328 dd49e9d8 d7274030 dd49e800 dd49e8b0 df847200
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196734]        00000202 e09449cd
dd49e800 d7274030 e0944a2f dd49e800 d7274000 e0944acc
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.196883]        df847214 d72744a4
e0944b50 00000000 e0944b64 00000000 c02805c2 d72744a4
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197040] Call Trace:
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197276]  [<c0285c8f>]
attribute_container_device_trigger+0x4f/0xb0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197630]  [<e09449cd>]
__scsi_remove_device+0x3d/0x80 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197749]  [<e0944a2f>]
scsi_remove_device+0x1f/0x30 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197849]  [<e0944acc>]
__scsi_remove_target+0x8c/0xc0 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.197949]  [<e0944b50>]
__remove_child+0x0/0x20 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198060]  [<e0944b64>]
__remove_child+0x14/0x20 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198155]  [<c02805c2>]
device_for_each_child+0x22/0x40
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198242]  [<e0944b3e>]
scsi_remove_target+0x3e/0x50 [scsi_mod]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198340]  [<e09ad85c>]
__iscsi_unbind_session+0x6c/0xa0 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198481]  [<e09ad948>]
iscsi_remove_session+0xb8/0x120 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198618]  [<e09b7aee>]
iscsi_session_teardown+0x4e/0x80 [libiscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198714]  [<e09b7aa0>]
iscsi_session_teardown+0x0/0x80 [libiscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198803]  [<e09ad9b8>]
iscsi_destroy_session+0x8/0x20 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.198940]  [<e09b7b05>]
iscsi_session_teardown+0x65/0x80 [libiscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199030]  [<e09b7aa0>]
iscsi_session_teardown+0x0/0x80 [libiscsi]
ion_fn+0x0/0x30 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199259]  [<e09ac04f>]
iscsi_iter_session_fn+0x1f/0x30 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199399]  [<c02805c2>]
device_for_each_child+0x22/0x40
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199482]  [<e09ac6f0>]
iscsi_if_rx+0x0/0x860 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199575]  [<e09b7b2d>]
iscsi_host_remove+0xd/0x20 [libiscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199662]  [<e09c2324>]
iscsi_tcp_session_destroy+0x34/0x50 [iscsi_tcp]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199756]  [<e09acec4>]
iscsi_if_rx+0x7d4/0x860 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199849]  [<c018d13d>]
__slab_free+0x16d/0x2a0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.199937]  [<c0121fef>]
update_curr+0x9f/0x150
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200027]  [<e09ac714>]
iscsi_if_rx+0x24/0x860 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200122]  [<c02a3a98>]
__kfree_skb+0x8/0x80
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200210]  [<e09ac6f0>]
iscsi_if_rx+0x0/0x860 [scsi_transport_iscsi]
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200304]  [<c02c282d>]
netlink_unicast+0x1dd/0x210
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200385]  [<c021aeee>]
copy_from_user+0x2e/0x70
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200469]  [<c02c30a6>]
netlink_sendmsg+0x226/0x2f0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200553]  [<c029dc51>]
sock_sendmsg+0x111/0x130
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200636]  [<c0140c20>]
autoremove_wake_function+0x0/0x40
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200724]  [<c0140c20>]
autoremove_wake_function+0x0/0x40
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200808]  [<c0199457>]
do_lookup+0x67/0x1c0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200887]  [<c019b87b>]
__link_path_walk+0xaab/0xe10
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.200969]  [<c021aeee>]
copy_from_user+0x2e/0x70
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201046]  [<c021aeee>]
copy_from_user+0x2e/0x70
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201124]  [<c029ddc9>]
sys_sendmsg+0x159/0x270
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201204]  [<c029ec4d>]
sys_recvmsg+0x17d/0x230
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201283]  [<c01a2ffd>]
d_kill+0x3d/0x60
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201360]  [<c01a2ffd>]
d_kill+0x3d/0x60
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201436]  [<c019abca>]
getname+0xaa/0xe0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201509]  [<c021b170>]
copy_to_user+0x30/0x60
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201586]  [<c0194f59>]
cp_new_stat64+0xf9/0x110
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201674]  [<c029f334>]
sys_socketcall+0xb4/0x2b0
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201758]  [<c01043b2>]
sysenter_past_esp+0x6b/0xa9
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201881]  =======================
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.201968] Code: 08 88 50 07 b8 08
00 00 00 c3 8d b4 26 00 00 00 00 53 89 d0 89 d3 e8 b6 5b 07 00 85 c0 74 1c
8b 83 50 ff ff ff 8d 8b 50 ff ff ff <8b> 40 60 85 c0 74 09 81 78 1c c0 3c 8d
e0 74 06 31 c0 5b c3 66
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.202434] EIP: [<e08ce12a>]
spi_device_match+0x1a/0x60 [scsi_transport_spi] SS:ESP 0068:d1633b8c
Sep 29 13:02:25 swe_1_ser_2 kernel: [ 8600.203315] ---[ end trace
a04d7e28d8fbadfb ]---
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.139345] sd 374:0:0:78: [sde]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.139813] end_request: I/O error,
dev sde, sector 0
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.139895] printk: 4 messages
suppressed.
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.139971] Buffer I/O error on
device sde, logical block 0
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.140058] Buffer I/O error on
device sde, logical block 1
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.140143] Buffer I/O error on
device sde, logical block 2
Sep 29 13:06:18 swe_1_ser_2 kernel: [ 8831.140228] Buffer I/O error on
device sde, logical block 3


Any idea?

Thanks,
Kevin

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to 
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~----------~----~----~----~------~----~------~--~---

Reply via email to