On Mon, Nov 16, 2009 at 09:39:00PM -0500, Hoot, Joseph wrote: > > On Nov 16, 2009, at 8:19 PM, Hoot, Joseph wrote: > > > thanks. That helps. So I know that with the EqualLogic targets, there is > > a "Group IP" which, I believe, responds with an iscsi login_redirect. > > > > 1) Could the "Login authentication failed" message be the response because > > of a login redirect messages from the EQL redirect? > > > > and then my next question is more for curiosity sake: > > > > 2) Are there plans in the future to have more than one connection per > > session? and I guess in addition to that, would that mean multiple > > connections to a single volume over the same nic? > > > > > > > Also Mike, I'm seeing one or two of these every 30-40 minutes if I slam our > EqualLogic with roughly 7-15k IOPS (reads and writes) non stop on 3 volumes. > In this type of scenario, would you expect to see timeouts like this once in > awhile? If so, do you think increasing my NOOP timeouts would assist so we > don't get these? maybe set it to 15 seconds instead of 10? >
Equallogic does active loadbalancing (redirects) during operation.. dunno about the errors though. -- Pasi > > > > > On Nov 16, 2009, at 7:18 PM, Mike Christie wrote: > > > >> Hoot, Joseph wrote: > >>> Hi all, > >>> > >>> I'm trying to understand what I'm seeing in my /var/log/messages. Here's > >>> what I have: > >>> > >>> Nov 13 10:49:47 oim6102506 kernel: connection5:0: ping timeout of 10 > >>> secs expired, last rx 191838122, last ping 191839372, now 191841872 > >>> Nov 13 10:49:47 oim6102506 kernel: connection5:0: detected conn error > >>> (1011) > >>> Nov 13 10:49:47 oim6102506 iscsid: Kernel reported iSCSI connection 5:0 > >>> error (1011) state (3) > >>> Nov 13 10:49:50 oim6102506 iscsid: Login authentication failed with > >>> target > >>> iqn.2001-05.com.equallogic:0-8a0906-e7d1dea02-786272c42554aef2-ovm-2-lun03 > >>> Nov 13 10:49:52 oim6102506 iscsid: connection5:0 is operational after > >>> recovery (1 attempts) > >>> > >>> the first line, what is "connection5:0"? is that referenced from > >>> iscsiadm somewhere? I only ask because I'm seeing iscsid messages and > >>> kernel messages. I also have dm-multipath running, which usually shows > >>> up as dm-multipath or something like that. I understand that iscsid is > >>> the process that is logging in and out. But is the "kernel:" message > >>> just an iscsi modules that is loaded into the kernel, which is why it is > >>> being logged as "kernel:"? > >>> > >> > >> It is the session id and connection id. > >> > >> connection$SESSION_ID:$CONNECTION_ID > >> > >> If you run iscsiadm -m session -P 1 or -P 3 > >> > >> You will see > >> > >> #iscsiadm -m session -P 1 > >> Target: iqn.1992-08.com.netapp:sn.33615311 > >> Current Portal: 10.15.85.19:3260,3 > >> Persistent Portal: 10.15.85.19:3260,3 > >> Iface Transport: tcp > >> Iface IPaddress: 10.11.14.37 > >> Iface HWaddress: default > >> Iface Netdev: default > >> SID: 7 > >> iSCSI Connection State: LOGGED IN > >> Internal iscsid Session State: NO CHANGE > >> > >> > >> Session number is the SID value. > >> > >> If you run > >> iscsiadm -m session > >> tcp [2] 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311 > >> > >> the session number/SID is the value in brackets. > >> > >> > >> If you run iscsiadm in session mode (iscsiadm -m session) then you can > >> use the -R argument and pass in a SID to do an opertaion like > >> > >> iscsiadm -m session -R 2 --rescan > >> > >> would rescan that session. > >> > >> Connection number is currently always zero. > >> > >> > >> For the second question, iscsid handles login and logout, and error > >> handling, and the kernel basically passes iscsi packets around. > >> > >> > >> Nov 13 10:49:47 oim6102506 kernel: connection5:0: ping timeout of 10 > >> secs expired, last rx 191838122, last ping 191839372, now 191841872 > >> > >> > >> so here the iscsi kernel code sends a iscsi ping/nop every noop_interval > >> seconds, and if we do not get a response withing noop_timeout seconds it > >> will fire off a connection error. > >> > >> > >> > >> Nov 13 10:49:47 oim6102506 kernel: connection5:0: detected conn error > >> (1011) > >> > >> > >> Here is the kernel code notifying userspace of the problem. > >> > >> > >> Nov 13 10:49:47 oim6102506 iscsid: Kernel reported iSCSI connection 5:0 > >> error (1011) state (3) > >> > >> > >> And there iscsid is accepting the error (probably no need for the error > >> to be logged twice). > >> > >> > >> Nov 13 10:49:50 oim6102506 iscsid: Login authentication failed with target > >> > >> > >> And then here iscsid handled the error by killing the tcp/ip connection, > >> reconnection the tcp/ip connection, and then re-logging into the iscsi > >> target. But for some reason we could not log back in right away. > >> > >> > >> iqn.2001-05.com.equallogic:0-8a0906-e7d1dea02-786272c42554aef2-ovm-2-lun03 > >> Nov 13 10:49:52 oim6102506 iscsid: connection5:0 is operational after > >> recovery (1 attempts) > >> > >> But it looks like we tried again and we got back in. > >> > >> > >> > >>> Maybe one of these modules? > >>> iscsi_tcp 19785 46 > >>> libiscsi_tcp 21829 1 iscsi_tcp > >>> libiscsi2 41285 3 ib_iser,iscsi_tcp,libiscsi_tcp > >>> scsi_transport_iscsi2 37197 5 ib_iser,iscsi_tcp,libiscsi2 > >>> scsi_transport_iscsi 6085 1 scsi_transport_iscsi2 > >>> > >>> I'm just trying to make sure that all of my timeout values line up > >>> properly so that, for example, OCFS2 filesystem doesn't timeout prior to > >>> iSCSI. I'd rather verify that iSCSI reconnections are attempted every 10 > >>> seconds and then dm-multipath has 30 seconds to timeout and then OCFS2 > >>> has 60 seconds before it times out (or something along those lines). So > >>> if I understand the logging a bit more, it'll help me. > >>> > >>> Thanks > >>> Joe > >>> > >>> > >>> =========================== > >>> Joseph R. Hoot > >>> Lead System Programmer/Analyst > >>> (w) 716-878-4832 > >>> (c) 716-759-HOOT > >>> joe.h...@itec.suny.edu > >>> GPG KEY: 7145F633 > >>> =========================== > >>> > >>> > >>> --~--~---------~--~----~------------~-------~--~----~ > >>> You received this message because you are subscribed to the Google Groups > >>> "open-iscsi" group. > >>> To post to this group, send email to open-iscsi@googlegroups.com > >>> To unsubscribe from this group, send email to > >>> open-iscsi+unsubscr...@googlegroups.com > >>> For more options, visit this group at > >>> http://groups.google.com/group/open-iscsi > >>> -~----------~----~----~----~------~----~------~--~--- > >>> > >> > >> -- > >> > >> You received this message because you are subscribed to the Google Groups > >> "open-iscsi" group. > >> To post to this group, send email to open-is...@googlegroups.com. > >> To unsubscribe from this group, send email to > >> open-iscsi+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/open-iscsi?hl=. > >> > >> > > > > =========================== > > Joseph R. Hoot > > Lead System Programmer/Analyst > > (w) 716-878-4832 > > (c) 716-759-HOOT > > joe.h...@itec.suny.edu > > GPG KEY: 7145F633 > > =========================== > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > "open-iscsi" group. > > To post to this group, send email to open-is...@googlegroups.com. > > To unsubscribe from this group, send email to > > open-iscsi+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/open-iscsi?hl=. > > > > > > =========================== > Joseph R. Hoot > Lead System Programmer/Analyst > (w) 716-878-4832 > (c) 716-759-HOOT > joe.h...@itec.suny.edu > GPG KEY: 7145F633 > =========================== > > -- > > You received this message because you are subscribed to the Google Groups > "open-iscsi" group. > To post to this group, send email to open-is...@googlegroups.com. > To unsubscribe from this group, send email to > open-iscsi+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/open-iscsi?hl=. > > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=.
