Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Maybe we want to do this in 2 steps. The attached patch is a simple 2 liner that seems to fix the problem for me. If it is ok with you guys we could send that upstream and into the stable kernel so it will hit debian and other distros. I think though there are other possibly bugs and so I think the correct fix is to rework the allocation/addition and release (it seems there are similar bugs in that path) code. The rework patch I am still working on is more invasive and has dependency issues so I am not sure if it can easily go into the stable kernels you guys use. Let me know what you guys think. On 08/07/2013 01:56 PM, Mike Christie wrote: Nice debugging! Thanks. When I get off of work today I will work on a fix. It looks like we need to rework how we bring up sessions/hosts (do the 2 stage alloc then add so the interface is not presented until after we have done all the allocations and setup) and then it looks like we can hit a similar bug on the release/free side where we kfree stuff while it could be in use. On 08/07/2013 09:40 AM, Lev Vainblat wrote: Hi Mike, I'm working together with Alex on this issue. I guess the problem is in iscsi_sw_tcp_session_create(): It first calls iscsi_host_add(shost, NULL), and only later tcp_sw_host-session = session; iscsi_host_add() eventually calls scsi_sysfs_add_host(), so sysfs special file is created before tcp_sw_host-session is ready. If now iscsi_sw_tcp_host_get_param() is called from iscsiadm, it gets NULL session pointer and fails. The issue is easily reproducible if you insert some sleep in iscsi_sw_tcp_session_create() after iscsi_host_add() and calls for example iscsiadm -m session during the sleep. -Lev. -Original Message- From: Alex Lyakas Sent: Monday, July 29, 2013 9:34 PM To: Mike Christie Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Thank you, Mike. We are running kernel 3.8.13, and now we can repro this pretty reliably. Alex. -Original Message- From: Mike Christie Sent: 29 July, 2013 9:29 PM To: Alex Lyakas Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param The iscsiadm crash is sort of expected due to where the kernel is crashing initially. When I get some time at work, I will make a debug patch for you run with that should spit out some extra info. On 07/28/2013 03:33 AM, Alex Lyakas wrote: Hi Mike, Attached is a trace of another repro, maybe it will give more info. The crashing iscsiadm process (2704) was spawned via fork/exec by our zadara_vam process (2657). Thanks, Alex. -Original Message- From: Mike Christie Sent: 22 July, 2013 7:14 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Hi Mike, I'm working together with Alex on this issue. I guess the problem is in iscsi_sw_tcp_session_create(): It first calls iscsi_host_add(shost, NULL), and only later tcp_sw_host-session = session; iscsi_host_add() eventually calls scsi_sysfs_add_host(), so sysfs special file is created before tcp_sw_host-session is ready. If now iscsi_sw_tcp_host_get_param() is called from iscsiadm, it gets NULL session pointer and fails. The issue is easily reproducible if you insert some sleep in iscsi_sw_tcp_session_create() after iscsi_host_add() and calls for example iscsiadm -m session during the sleep. -Lev. -Original Message- From: Alex Lyakas Sent: Monday, July 29, 2013 9:34 PM To: Mike Christie Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Thank you, Mike. We are running kernel 3.8.13, and now we can repro this pretty reliably. Alex. -Original Message- From: Mike Christie Sent: 29 July, 2013 9:29 PM To: Alex Lyakas Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param The iscsiadm crash is sort of expected due to where the kernel is crashing initially. When I get some time at work, I will make a debug patch for you run with that should spit out some extra info. On 07/28/2013 03:33 AM, Alex Lyakas wrote: Hi Mike, Attached is a trace of another repro, maybe it will give more info. The crashing iscsiadm process (2704) was spawned via fork/exec by our zadara_vam process (2657). Thanks, Alex. -Original Message- From: Mike Christie Sent: 22 July, 2013 7:14 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Nice debugging! Thanks. When I get off of work today I will work on a fix. It looks like we need to rework how we bring up sessions/hosts (do the 2 stage alloc then add so the interface is not presented until after we have done all the allocations and setup) and then it looks like we can hit a similar bug on the release/free side where we kfree stuff while it could be in use. On 08/07/2013 09:40 AM, Lev Vainblat wrote: Hi Mike, I'm working together with Alex on this issue. I guess the problem is in iscsi_sw_tcp_session_create(): It first calls iscsi_host_add(shost, NULL), and only later tcp_sw_host-session = session; iscsi_host_add() eventually calls scsi_sysfs_add_host(), so sysfs special file is created before tcp_sw_host-session is ready. If now iscsi_sw_tcp_host_get_param() is called from iscsiadm, it gets NULL session pointer and fails. The issue is easily reproducible if you insert some sleep in iscsi_sw_tcp_session_create() after iscsi_host_add() and calls for example iscsiadm -m session during the sleep. -Lev. -Original Message- From: Alex Lyakas Sent: Monday, July 29, 2013 9:34 PM To: Mike Christie Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Thank you, Mike. We are running kernel 3.8.13, and now we can repro this pretty reliably. Alex. -Original Message- From: Mike Christie Sent: 29 July, 2013 9:29 PM To: Alex Lyakas Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param The iscsiadm crash is sort of expected due to where the kernel is crashing initially. When I get some time at work, I will make a debug patch for you run with that should spit out some extra info. On 07/28/2013 03:33 AM, Alex Lyakas wrote: Hi Mike, Attached is a trace of another repro, maybe it will give more info. The crashing iscsiadm process (2704) was spawned via fork/exec by our zadara_vam process (2657). Thanks, Alex. -Original Message- From: Mike Christie Sent: 22 July, 2013 7:14 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
The iscsiadm crash is sort of expected due to where the kernel is crashing initially. When I get some time at work, I will make a debug patch for you run with that should spit out some extra info. On 07/28/2013 03:33 AM, Alex Lyakas wrote: Hi Mike, Attached is a trace of another repro, maybe it will give more info. The crashing iscsiadm process (2704) was spawned via fork/exec by our zadara_vam process (2657). Thanks, Alex. -Original Message- From: Mike Christie Sent: 22 July, 2013 7:14 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Thank you, Mike. We are running kernel 3.8.13, and now we can repro this pretty reliably. Alex. -Original Message- From: Mike Christie Sent: 29 July, 2013 9:29 PM To: Alex Lyakas Cc: open-iscsi@googlegroups.com ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param The iscsiadm crash is sort of expected due to where the kernel is crashing initially. When I get some time at work, I will make a debug patch for you run with that should spit out some extra info. On 07/28/2013 03:33 AM, Alex Lyakas wrote: Hi Mike, Attached is a trace of another repro, maybe it will give more info. The crashing iscsiadm process (2704) was spawned via fork/exec by our zadara_vam process (2657). Thanks, Alex. -Original Message- From: Mike Christie Sent: 22 July, 2013 7:14 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko ; Liran Strugano Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. Huh. I tried upstream 2.6.39 to 3.8 and also 3.8.10 and could not hit the problem. I only hit it with that OEL kernel when using bnx2x. Also I hit the oops in a slightly different place. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
For me to replicate the problem I just login to the target using bnx2i. Just one instance of iscsiadm -m node --login using the bnx2i driver causes the problem. I have been trying to replicate when adding debug comments in the kernel but that made the problem go away. I also tried getting a crash report but with that enabled the problem went away. On 07/22/2013 09:24 AM, Alex Lyakas wrote: Hi Mike, any advice on how to proceed further with this issue? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 02 July, 2013 9:41 PM To: Mike Christie ; open-iscsi@googlegroups.com Cc: Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. Huh. I tried upstream 2.6.39 to 3.8 and also 3.8.10 and could not hit the problem. I only hit it with that OEL kernel when using bnx2x. Also I hit the oops in a slightly different place. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. Huh. I tried upstream 2.6.39 to 3.8 and also 3.8.10 and could not hit the problem. I only hit it with that OEL kernel when using bnx2x. Also I hit the oops in a slightly different place. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Hi Mike, For us it happened only once till now; and from our kernel log, I don't think anything special was going on during that time, except that we were reading the sysfs entry. Can you pls share how do you replicate the problem with the Oracle kernel? If this narrows us down a bit on how to replicate, then, yes, we can apply a debugging patch. One thing our application is doing, is to run several iscsiadm commands (via fork/exec) in parallel. Is this, in general, a safe thing to do, i.e., running multiple iscsiadm processes in parallel? Each iscsiadm process operates against a different iSCSI target. Thanks, Alex. -Original Message- From: Mike Christie Sent: 02 July, 2013 8:23 PM To: open-iscsi@googlegroups.com Cc: Alex Lyakas ; Lev Vainblat ; Yair Hershko Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param Hey, Is it easy for you to replicate this problem and if so would it be possible to run with a patch that spits out some extra debugging info? It is easy for me to replicate with the Oracle linux kernel, but when I add debugging it seems to move around or become difficult to hit. On 06/27/2013 04:54 AM, Alex Lyakas wrote: Hello Mike, thank you for responding to my bug report. Here is the information you asked for: This issue happened within a virtual machine. The network interface, that is used for iscsi within the VM is a SR-IOV Virtual Function. The VM runs a stock ixgbevf driver from 3.8.13 mainline kernel. On the physical machine, the Virtual Function is spawned out of Intel 82599EB card. The ixgbe driver for the Intel card on the physical machine is 3.11.33. The physical machine runs stock Ubuntu Precise kernel 3.2.0-29-generic #46-Ubuntu, while the VM runs mainline 3.8.13 kernel. From within the VM, we connect to targets that live both on the same physical machine and on other physical machines. I am attaching a .config file for the VM kernel. We did not build the kernel ourselves, this is a mainline build done by Ubunti here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8.13-raring/ I am also attaching the full kernel log with the information about the crash, it has more stack traces. The iscsiadm commands that we use (from another application via fork()/exec()) are: iscsiadm --mode node --portal portal --targetname name --op new iscsiadm --mode node --portal portal --targetname name --login iscsiadm --mode node --portal portal --targetname name --logout iscsiadm --mode node [--print level] iscsiadm --mode session [--print level] iscsiadm --mode host [--print level] Occasionally we also read sysfs attributes, with a script that collects all system's sysfs entries for further analysis. We use version 2.0-871 of open-iscsi, we install them via apt-get: dpkg -l: ii open-iscsi 2.0.871-0ubuntu9.12.04.1 High performance, transport independent iSCSI implementation ii open-iscsi-utils2.0.871-0ubuntu9.12.04.1 iSCSI initiatior administrative utility We open one session to each target, but we connect to multiple targets. Please let me know if any other info is needed. Thanks for your help, Alex. -Original Message- From: Mike Christie Sent: 27 June, 2013 10:13 AM To: open-iscsi@googlegroups.com Cc: Alex Lyakas Subject: Re: NULL pointer deref in iscsi_sw_tcp_host_get_param On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. Huh. I tried upstream 2.6.39 to 3.8 and also 3.8.10 and could not hit the problem. I only hit it with that OEL kernel when using bnx2x. Also I hit the oops in a slightly different place. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
On 06/26/2013 07:50 PM, Mike Christie wrote: On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. Huh. I tried upstream 2.6.39 to 3.8 and also 3.8.10 and could not hit the problem. I only hit it with that OEL kernel when using bnx2x. Also I hit the oops in a slightly different place. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
NULL pointer deref in iscsi_sw_tcp_host_get_param
Greetings all, iscsiadm has caused kernel NULL pointer dereference, that looks like follows: [ 29.833228] BUG: unable to handle kernel NULL pointer dereference at 00d0 [ 29.835268] IP: [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.836034] PGD 114f15067 PUD 114f14067 PMD 0 [ 29.836034] Oops: 0002 [#1] SMP [ 29.852450] sd 6:0:0:0: Attached scsi generic sg3 type 0 [ 29.852468] scst[2564] scst_register_device[976]: Attached to scsi6, channel 0, id 0, lun 0, type 0 [ 29.852567] sd 4:0:0:0: Attached scsi generic sg4 type 0 [ 29.852578] scst[2565] scst_register_device[976]: Attached to scsi4, channel 0, id 0, lun 0, type 0 [ 29.859101] drm serio_raw sysimgblt sysfillrect virtio_balloon nfsd(OF) nfs_acl syscopyarea auth_rpcgss nfs i2c_piix4 fscache lockd sunrpc mac_hid lp parport floppy ixgbevf [ 29.859101] CPU 1 [ 29.859101] Pid: 2569, comm: iscsiadm Tainted: GF O 3.8.13-030813-generic #201305111843 Bochs Bochs [ 29.859101] RIP: 0010:[81045c29] [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP: 0018:880114ca7d98 EFLAGS: 00010206 [ 29.859101] RAX: 0001 RBX: 00d0 RCX: 0001 [ 29.859101] RDX: 880115287000 RSI: 0003 RDI: 00d0 [ 29.859101] RBP: 880114ca7d98 R08: R09: 0c55 [ 29.859101] R10: ea000454a1c0 R11: 0246 R12: 00d0 [ 29.859101] R13: R14: 8801157f6438 R15: 880115fc8c00 [ 29.859101] FS: 7f6bbb801700() GS:88011fd0() knlGS: [ 29.859101] CS: 0010 DS: ES: CR0: 80050033 [ 29.859101] CR2: 00d0 CR3: 00011494e000 CR4: 06e0 [ 29.859101] DR0: DR1: DR2: [ 29.859101] DR3: DR6: 0ff0 DR7: 0400 [ 29.859101] Process iscsiadm (pid: 2569, threadinfo 880114ca6000, task 880114e2) [ 29.859101] Stack: [ 29.859101] 880114ca7db8 816ed63e 88011fffbd80 880115287000 [ 29.859101] 880114ca7e28 a05d194f 88011516bd20 880118bed600 [ 29.859101] 000995fdf9ab 81ef7b20 80d0 [ 29.859101] Call Trace: [ 29.859101] [816ed63e] _raw_spin_lock_bh+0x1e/0x30 [ 29.859101] [a05d194f] iscsi_sw_tcp_host_get_param+0x6f/0xe0 [iscsi_tcp] [ 29.859101] [a05ab2ac] show_host_param_ISCSI_HOST_PARAM_IPADDRESS+0x4c/0x60 [scsi_transport_iscsi] [ 29.859101] [8145e7a0] dev_attr_show+0x20/0x60 [ 29.859101] [81139aae] ? __get_free_pages+0xe/0x40 [ 29.859101] [8120f736] fill_read_buffer.isra.8+0x66/0xf0 [ 29.859101] [8120f864] sysfs_read_file+0xa4/0xc0 [ 29.859101] [8119b4d0] vfs_read+0xb0/0x180 [ 29.859101] [8119b5f2] sys_read+0x52/0xa0 [ 29.859101] [816f629d] system_call_fastpath+0x1a/0x1f [ 29.859101] Code: 00 00 48 c7 c1 31 5a 04 81 48 c7 c2 2e 5a 04 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 [ 29.859101] RIP [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP 880114ca7d98 [ 29.859101] CR2: 00d0 [ 29.926003] ---[ end trace 3db857a832dd3e91 ]--- Kernel is mainline 3.8.13. Can anybody please advise if this is some known issue and/or how to investigate this further? Thanks, Alex. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
Hello Alex, Dmitry, this looks like a real kernel issue in an non-ancient kernel. Has my bug report been noticed? Thanks, Alex. -Original Message- From: Alex Lyakas Sent: 23 June, 2013 11:06 AM To: open-iscsi@googlegroups.com Subject: NULL pointer deref in iscsi_sw_tcp_host_get_param Greetings all, iscsiadm has caused kernel NULL pointer dereference, that looks like follows: [ 29.833228] BUG: unable to handle kernel NULL pointer dereference at 00d0 [ 29.835268] IP: [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.836034] PGD 114f15067 PUD 114f14067 PMD 0 [ 29.836034] Oops: 0002 [#1] SMP [ 29.852450] sd 6:0:0:0: Attached scsi generic sg3 type 0 [ 29.852468] scst[2564] scst_register_device[976]: Attached to scsi6, channel 0, id 0, lun 0, type 0 [ 29.852567] sd 4:0:0:0: Attached scsi generic sg4 type 0 [ 29.852578] scst[2565] scst_register_device[976]: Attached to scsi4, channel 0, id 0, lun 0, type 0 [ 29.859101] drm serio_raw sysimgblt sysfillrect virtio_balloon nfsd(OF) nfs_acl syscopyarea auth_rpcgss nfs i2c_piix4 fscache lockd sunrpc mac_hid lp parport floppy ixgbevf [ 29.859101] CPU 1 [ 29.859101] Pid: 2569, comm: iscsiadm Tainted: GF O 3.8.13-030813-generic #201305111843 Bochs Bochs [ 29.859101] RIP: 0010:[81045c29] [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP: 0018:880114ca7d98 EFLAGS: 00010206 [ 29.859101] RAX: 0001 RBX: 00d0 RCX: 0001 [ 29.859101] RDX: 880115287000 RSI: 0003 RDI: 00d0 [ 29.859101] RBP: 880114ca7d98 R08: R09: 0c55 [ 29.859101] R10: ea000454a1c0 R11: 0246 R12: 00d0 [ 29.859101] R13: R14: 8801157f6438 R15: 880115fc8c00 [ 29.859101] FS: 7f6bbb801700() GS:88011fd0() knlGS: [ 29.859101] CS: 0010 DS: ES: CR0: 80050033 [ 29.859101] CR2: 00d0 CR3: 00011494e000 CR4: 06e0 [ 29.859101] DR0: DR1: DR2: [ 29.859101] DR3: DR6: 0ff0 DR7: 0400 [ 29.859101] Process iscsiadm (pid: 2569, threadinfo 880114ca6000, task 880114e2) [ 29.859101] Stack: [ 29.859101] 880114ca7db8 816ed63e 88011fffbd80 880115287000 [ 29.859101] 880114ca7e28 a05d194f 88011516bd20 880118bed600 [ 29.859101] 000995fdf9ab 81ef7b20 80d0 [ 29.859101] Call Trace: [ 29.859101] [816ed63e] _raw_spin_lock_bh+0x1e/0x30 [ 29.859101] [a05d194f] iscsi_sw_tcp_host_get_param+0x6f/0xe0 [iscsi_tcp] [ 29.859101] [a05ab2ac] show_host_param_ISCSI_HOST_PARAM_IPADDRESS+0x4c/0x60 [scsi_transport_iscsi] [ 29.859101] [8145e7a0] dev_attr_show+0x20/0x60 [ 29.859101] [81139aae] ? __get_free_pages+0xe/0x40 [ 29.859101] [8120f736] fill_read_buffer.isra.8+0x66/0xf0 [ 29.859101] [8120f864] sysfs_read_file+0xa4/0xc0 [ 29.859101] [8119b4d0] vfs_read+0xb0/0x180 [ 29.859101] [8119b5f2] sys_read+0x52/0xa0 [ 29.859101] [816f629d] system_call_fastpath+0x1a/0x1f [ 29.859101] Code: 00 00 48 c7 c1 31 5a 04 81 48 c7 c2 2e 5a 04 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 [ 29.859101] RIP [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP 880114ca7d98 [ 29.859101] CR2: 00d0 [ 29.926003] ---[ end trace 3db857a832dd3e91 ]--- Kernel is mainline 3.8.13. Can anybody please advise if this is some known issue and/or how to investigate this further? Thanks, Alex. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
NULL pointer deref in iscsi_sw_tcp_host_get_param
Greetings all, iscsiadm has caused kernel NULL pointer dereference, that looks like follows: [ 29.833228] BUG: unable to handle kernel NULL pointer dereference at 00d0 [ 29.835268] IP: [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.836034] PGD 114f15067 PUD 114f14067 PMD 0 [ 29.836034] Oops: 0002 [#1] SMP [ 29.852450] sd 6:0:0:0: Attached scsi generic sg3 type 0 [ 29.852468] scst[2564] scst_register_device[976]: Attached to scsi6, channel 0, id 0, lun 0, type 0 [ 29.852567] sd 4:0:0:0: Attached scsi generic sg4 type 0 [ 29.852578] scst[2565] scst_register_device[976]: Attached to scsi4, channel 0, id 0, lun 0, type 0 [ 29.859101] drm serio_raw sysimgblt sysfillrect virtio_balloon nfsd(OF) nfs_acl syscopyarea auth_rpcgss nfs i2c_piix4 fscache lockd sunrpc mac_hid lp parport floppy ixgbevf [ 29.859101] CPU 1 [ 29.859101] Pid: 2569, comm: iscsiadm Tainted: GF O 3.8.13-030813-generic #201305111843 Bochs Bochs [ 29.859101] RIP: 0010:[81045c29] [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP: 0018:880114ca7d98 EFLAGS: 00010206 [ 29.859101] RAX: 0001 RBX: 00d0 RCX: 0001 [ 29.859101] RDX: 880115287000 RSI: 0003 RDI: 00d0 [ 29.859101] RBP: 880114ca7d98 R08: R09: 0c55 [ 29.859101] R10: ea000454a1c0 R11: 0246 R12: 00d0 [ 29.859101] R13: R14: 8801157f6438 R15: 880115fc8c00 [ 29.859101] FS: 7f6bbb801700() GS:88011fd0() knlGS: [ 29.859101] CS: 0010 DS: ES: CR0: 80050033 [ 29.859101] CR2: 00d0 CR3: 00011494e000 CR4: 06e0 [ 29.859101] DR0: DR1: DR2: [ 29.859101] DR3: DR6: 0ff0 DR7: 0400 [ 29.859101] Process iscsiadm (pid: 2569, threadinfo 880114ca6000, task 880114e2) [ 29.859101] Stack: [ 29.859101] 880114ca7db8 816ed63e 88011fffbd80 880115287000 [ 29.859101] 880114ca7e28 a05d194f 88011516bd20 880118bed600 [ 29.859101] 000995fdf9ab 81ef7b20 80d0 [ 29.859101] Call Trace: [ 29.859101] [816ed63e] _raw_spin_lock_bh+0x1e/0x30 [ 29.859101] [a05d194f] iscsi_sw_tcp_host_get_param+0x6f/0xe0 [iscsi_tcp] [ 29.859101] [a05ab2ac] show_host_param_ISCSI_HOST_PARAM_IPADDRESS+0x4c/0x60 [scsi_transport_iscsi] [ 29.859101] [8145e7a0] dev_attr_show+0x20/0x60 [ 29.859101] [81139aae] ? __get_free_pages+0xe/0x40 [ 29.859101] [8120f736] fill_read_buffer.isra.8+0x66/0xf0 [ 29.859101] [8120f864] sysfs_read_file+0xa4/0xc0 [ 29.859101] [8119b4d0] vfs_read+0xb0/0x180 [ 29.859101] [8119b5f2] sys_read+0x52/0xa0 [ 29.859101] [816f629d] system_call_fastpath+0x1a/0x1f [ 29.859101] Code: 00 00 48 c7 c1 31 5a 04 81 48 c7 c2 2e 5a 04 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 [ 29.859101] RIP [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP 880114ca7d98 [ 29.859101] CR2: 00d0 [ 29.926003] ---[ end trace 3db857a832dd3e91 ]--- Kernel is mainline 3.8.13. Can anybody please advise if this is some known issue and/or how to investigate this further? Thanks, Alex. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
On 06/23/2013 03:06 AM, Alex Lyakas wrote: Greetings all, iscsiadm has caused kernel NULL pointer dereference, that looks like follows: [ 29.833228] BUG: unable to handle kernel NULL pointer dereference at 00d0 [ 29.835268] IP: [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.836034] PGD 114f15067 PUD 114f14067 PMD 0 [ 29.836034] Oops: 0002 [#1] SMP [ 29.852450] sd 6:0:0:0: Attached scsi generic sg3 type 0 [ 29.852468] scst[2564] scst_register_device[976]: Attached to scsi6, channel 0, id 0, lun 0, type 0 [ 29.852567] sd 4:0:0:0: Attached scsi generic sg4 type 0 [ 29.852578] scst[2565] scst_register_device[976]: Attached to scsi4, channel 0, id 0, lun 0, type 0 [ 29.859101] drm serio_raw sysimgblt sysfillrect virtio_balloon nfsd(OF) nfs_acl syscopyarea auth_rpcgss nfs i2c_piix4 fscache lockd sunrpc mac_hid lp parport floppy ixgbevf [ 29.859101] CPU 1 [ 29.859101] Pid: 2569, comm: iscsiadm Tainted: GF O 3.8.13-030813-generic #201305111843 Bochs Bochs [ 29.859101] RIP: 0010:[81045c29] [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP: 0018:880114ca7d98 EFLAGS: 00010206 [ 29.859101] RAX: 0001 RBX: 00d0 RCX: 0001 [ 29.859101] RDX: 880115287000 RSI: 0003 RDI: 00d0 [ 29.859101] RBP: 880114ca7d98 R08: R09: 0c55 [ 29.859101] R10: ea000454a1c0 R11: 0246 R12: 00d0 [ 29.859101] R13: R14: 8801157f6438 R15: 880115fc8c00 [ 29.859101] FS: 7f6bbb801700() GS:88011fd0() knlGS: [ 29.859101] CS: 0010 DS: ES: CR0: 80050033 [ 29.859101] CR2: 00d0 CR3: 00011494e000 CR4: 06e0 [ 29.859101] DR0: DR1: DR2: [ 29.859101] DR3: DR6: 0ff0 DR7: 0400 [ 29.859101] Process iscsiadm (pid: 2569, threadinfo 880114ca6000, task 880114e2) [ 29.859101] Stack: [ 29.859101] 880114ca7db8 816ed63e 88011fffbd80 880115287000 [ 29.859101] 880114ca7e28 a05d194f 88011516bd20 880118bed600 [ 29.859101] 000995fdf9ab 81ef7b20 80d0 [ 29.859101] Call Trace: [ 29.859101] [816ed63e] _raw_spin_lock_bh+0x1e/0x30 [ 29.859101] [a05d194f] iscsi_sw_tcp_host_get_param+0x6f/0xe0 [iscsi_tcp] [ 29.859101] [a05ab2ac] show_host_param_ISCSI_HOST_PARAM_IPADDRESS+0x4c/0x60 [scsi_transport_iscsi] [ 29.859101] [8145e7a0] dev_attr_show+0x20/0x60 [ 29.859101] [81139aae] ? __get_free_pages+0xe/0x40 [ 29.859101] [8120f736] fill_read_buffer.isra.8+0x66/0xf0 [ 29.859101] [8120f864] sysfs_read_file+0xa4/0xc0 [ 29.859101] [8119b4d0] vfs_read+0xb0/0x180 [ 29.859101] [8119b5f2] sys_read+0x52/0xa0 [ 29.859101] [816f629d] system_call_fastpath+0x1a/0x1f [ 29.859101] Code: 00 00 48 c7 c1 31 5a 04 81 48 c7 c2 2e 5a 04 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 [ 29.859101] RIP [81045c29] __ticket_spin_lock+0x9/0x30 [ 29.859101] RSP 880114ca7d98 [ 29.859101] CR2: 00d0 [ 29.926003] ---[ end trace 3db857a832dd3e91 ]--- Kernel is mainline 3.8.13. We have not seen it before. I am not seeing it here. What else are you running in your environment and has any of that changes recently. Is this in a vm or real machine? Could you also attached your kernel config file? Did you build the kernel yourself? What are you doing when this happens? Are you just logging in or running some other command? Do you see this with other kernel versions, because the iscsi code mentioned in the trace has not changed. The actual oops seems to be in the spin lock code though. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
Re: NULL pointer deref in iscsi_sw_tcp_host_get_param
On 06/26/2013 05:27 PM, Mike Christie wrote: We have not seen it before. I am not seeing it here. Oh wait, I can hit it when using bnx2i and the OEL kernel 2.6.39-400.17.1.el6uek.x86_64 kernel. I do not hit it with iscsi_tcp though. Have not tried other upstream kernels with offload yet. What version of the open-iscsi tools are you using and are you using software iscsi or a offload driver? How many sessions do you have running? -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/groups/opt_out.
