bigcatxjs wrote: > UPDATE: RHEL 5.3 Host is showing errors. No Disk I/O to SAN volume > (last I/O Thursday 12th March); >
Is there anything in the log before this? Something about a ping or nop timing out? > Mar 13 10:38:49 MYHOST53 kernel: connection1:0: iscsi: detected conn > error (1011) > Mar 13 10:38:49 MYHOST53 iscsid: Kernel reported iSCSI connection 1:0 > error (1011) state (3) > Mar 13 10:38:52 MYHOST53 iscsid: received iferror -38 > Mar 13 10:38:52 MYHOST53 last message repeated 2 times > Mar 13 10:38:52 MYHOST53 iscsid: connection1:0 is operational after > recovery (1 attempts) > Mar 13 11:00:06 MYHOST53 kernel: connection1:0: iscsi: detected conn > error (1011) > Mar 13 11:00:06 MYHOST53 iscsid: Kernel reported iSCSI connection 1:0 > error (1011) state (3) > Mar 13 11:00:09 MYHOST53 iscsid: received iferror -38 > Mar 13 11:00:09 MYHOST53 last message repeated 2 times > Mar 13 11:00:09 MYHOST53 iscsid: connection1:0 is operational after > recovery (1 attempts) > > Thanks, Rich. > > END. > > On Mar 13, 10:01 am, bigcatxjs <ad...@richardjamestrading.co.uk> > wrote: >> Thanks Mike, >> >>> For this RHEL 5.2 setup, does it make a difference if you do not use >>> ifaces and setup the box like in 5.3 below? >> I have used bonded ifaces so that the I/O requests can be split across >> multiple NICS (both Server-side and on the Datacore San Melody SM node >> NICS). This split is acheived by ensuring that the volumes used by >> Oracle containing DATA and INDEX datafiles route through one named >> Iface and that volumes used by Oracle for SYSTEM, BACKUP, and REDO >> data / logs etc route through the other. We have seen a performance >> uplift by maintaining this split despite the time-out issues. We have >> a W2K3 x86_64 STD Oracle host that runs on one iface - this is much >> slower than the RHEL 5.2 x86_64 host even though the hardware is >> identical. We did have RHEL 5.1 x86_64 Oracle hosts running on one >> iface - again, this was noticibly slower than the bonded ifaces >> approach. This have been upgraded to RHEL 5.2 with the multiple >> ifaces. >> >>> There was a bug in 5.2 where the initiator would think it detected a >>> timeout when it did not. It is fixed in 5.3. >> Good. Then I should expect to see less errors. >> >>> The messages can also occur when there really is a problem with the >>> network or if the target is bogged down. >> We have spread the primary volumes across both SM nodes. The nodes >> are WK23K x86 (no x64 option for the DataCore Software) DELL 2850's. >> There are two switches (one for SM1, one for SM2) that are linked >> using teamed Fibre (2GB sec capacity). Thus I/O should route evenly >> across both switches. The SM mirroring takes advantage of the Fibre. >> With the RHEL 5.2 host, you will note that both ifaces are goiing to >> SM2 node, but utilising different NICS on the SM2 node. These volumes >> are then mirrored to SM1 (except the BACKUP volume, which is a linear >> volume). We know that the switches aren't congested, but we don't >> accurately know if SM1 or SM2 are congested. We only have a logical >> spread of volumes presented across multiple NICS to at least try and >> minimise congestion. >> >>> At these times is there lots of disk IO? Is there anything in the target >>> logs? >> It is fair to say that all these volumes take a heavy hit, in terms of >> I/O. Each host (excluding the RHEL 5.3. test host) run two Oracle >> databases, of which some have intra-database replication (Oracle >> Streams) enabled. The issue on the RHEL 5.2 host occures every 10 >> secs or so during Office Hours when it is being utilised. >> >>> So the RHEL5.3 box is having troubles too? There is nothing in the log >>> below. >> The error with the RHEL 5.3 host was as follows; >> >>> Mar 11 18:12:03 MYHOST53 iscsid: received iferror -38 >>> Mar 11 18:12:03 MYHOST53 last message repeated 2 times >>> Mar 11 18:12:03 MYHOST53 iscsid: connection1:0 is operational now >> This looked similar to previous RHEL 5.2 errors. >> >>> Can you replicate this pretty easily? If you just login the session, >>> then let it sit (do not run the db or any disk IO), will you see the >>> ping timeout errors? >> I can test this with the RHEL 5.3 host. Unfortunately, it will be >> difficult to down the RHEL 5.2 host's database services until we have >> a scheduled outage window. >> >> Today, there have been no further errors on RHEL 5.3 host :>). >> >>> It might be helpful to run ethereal/wireshark while you run your test >>> then send the /var/log/messages and trace so I can check and see if the >>> ping is really timing out or not. For the test you only need one session >>> logged in (this will reduce log and trace info), and once you see the >>> first ping timeout error you can stop tracing/logging and send it. >> Yes; there is also an Oracle tool (Orion) that we could also use. >> >> I think that I will monitor the RHEL 5.3 host for any further errors. >> If the incidence of errors is reduced, then this gives justification >> to upgrading the RHEL 5.2 host to 5.3. Such an outage would provide >> me with an opportunity to perform the tests above as well. >> >> Many thanks, >> Richard. >> >> END. >> >> With the RHEL 5.2 host >> >> On Mar 12, 5:53 pm, Mike Christie <micha...@cs.wisc.edu> wrote: >> >> >> >>> bigcatxjs wrote: >>> For this RHEL 5.2 setup, does it make a difference if you do not use >>> ifaces and setup the box like in 5.3 below? >>>> iscsiadm: >>>> iSCSI Transport Class version 2.0-724 >>>> iscsiadm version 2.0-868 >>>> Target: iqn.2000-08.com.datacore:sm2-3 >>>> Current Portal: 172.16.200.9:3260,1 >>>> Persistent Portal: 172.16.200.9:3260,1 >>>> ********** >>>> Interface: >>>> ********** >>>> Iface Name: iface0 >>>> Iface Transport: tcp >>>> Iface Initiatorname: iqn.1994-05.com.redhat:7fe2f44ea9de >>>> Iface IPaddress: 172.16.200.39 >>>> Iface HWaddress: 00:14:22:0d:0a:fa >>>> Iface Netdev: default >>>> SID: 1 >>>> iSCSI Connection State: LOGGED IN >>>> iSCSI Session State: Unknown >>>> Internal iscsid Session State: NO CHANGE >>>> ************************ >>>> Negotiated iSCSI params: >>>> ************************ >>>> HeaderDigest: None >>>> DataDigest: None >>>> MaxRecvDataSegmentLength: 131072 >>>> MaxXmitDataSegmentLength: 262144 >>>> FirstBurstLength: 0 >>>> MaxBurstLength: 1048576 >>>> ImmediateData: No >>>> InitialR2T: Yes >>>> MaxOutstandingR2T: 1 >>>> ************************ >>>> Attached SCSI devices: >>>> ************************ >>>> Host Number: 1 State: running >>>> scsi1 Channel 00 Id 0 Lun: 0 >>>> Attached scsi disk sdb State: running >>>> scsi1 Channel 00 Id 0 Lun: 1 >>>> Attached scsi disk sde State: running >>>> scsi1 Channel 00 Id 0 Lun: 2 >>>> Attached scsi disk sdf State: running >>>> Target: iqn.2000-08.com.datacore:sm2-4 >>>> Current Portal: 172.16.200.10:3260,1 >>>> Persistent Portal: 172.16.200.10:3260,1 >>>> ********** >>>> Interface: >>>> ********** >>>> Iface Name: iface2 >>>> Iface Transport: tcp >>>> Iface Initiatorname: iqn.1994-05.com.redhat:7fe2f44ea9de >>>> Iface IPaddress: 172.16.200.56 >>>> Iface HWaddress: 00:14:22:b1:d6:a6 >>>> Iface Netdev: default >>>> SID: 2 >>>> iSCSI Connection State: LOGGED IN >>>> iSCSI Session State: Unknown >>>> Internal iscsid Session State: NO CHANGE >>>> ************************ >>>> Negotiated iSCSI params: >>>> ************************ >>>> HeaderDigest: None >>>> DataDigest: None >>>> MaxRecvDataSegmentLength: 131072 >>>> MaxXmitDataSegmentLength: 262144 >>>> FirstBurstLength: 0 >>>> MaxBurstLength: 1048576 >>>> ImmediateData: No >>>> InitialR2T: Yes >>>> MaxOutstandingR2T: 1 >>>> ************************ >>>> Attached SCSI devices: >>>> ************************ >>>> Host Number: 2 State: running >>>> scsi2 Channel 00 Id 0 Lun: 0 >>>> Attached scsi disk sdc State: running >>>> scsi2 Channel 00 Id 0 Lun: 1 >>>> Attached scsi disk sdd State: running >>>> Log Errors;co >>>> Mar 12 09:30:48 MYHOST52 last message repeated 2 times >>>> Mar 12 09:30:48 MYHOST52 iscsid: connection2:0 is operational after >>>> recovery (1 attempts) >>>> Mar 12 09:32:52 MYHOST52 kernel: ping timeout of 5 secs expired, last >>>> rx 19592296349, last ping 19592301349, now 19592306349 >>> There was a bug in 5.2 where the initiator would think it detected a >>> timeout when it did not. It is fixed in 5.3. >>> The messages can also occur when there really is a problem with the >>> network or if the target is bogged down. >>> At these times is there lots of disk IO? Is there anything in the target >>> logs? >>> I am also not sure how well some targets handle bonding plus ifaces. Is >>> iface* using a bonded interface? >>> Can you replicate this pretty easily? If you just login the session, >>> then let it sit (do not run the db or any disk IO), will you see the >>> ping timeout errors? >>> It might be helpful to run ethereal/wireshark while you run your test >>> then send the /var/log/messages and trace so I can check and see if the >>> ping is really timing out or not. For the test you only need one session >>> logged in (this will reduce log and trace info), and once you see the >>> first ping timeout error you can stop tracing/logging and send it. >>>> From RHEL 5.3 x86 Host; >>> So the RHEL5.3 box is having troubles too? There is nothing in the log >>> below. >>>> iscsiadm; >>>> iSCSI Transport Class version 2.0-724 >>>> iscsiadm version 2.0-868 >>>> Target: iqn.2000-08.com.datacore:sm2-3 >>>> Current Portal: 172.16.200.9:3260,1 >>>> Persistent Portal: 172.16.200.9:3260,1 >>>> ********** >>>> Interface: >>>> ********** >>>> Iface Name: default >>>> Iface Transport: tcp >>>> Iface Initiatorname: iqn.2005-03.com.redhat:01.406e5fd710e2 >>>> Iface IPaddress: 172.16.200.69 >>>> - Hide quoted text - >> - Show quoted text -... >> >> read more ยป > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~----------~----~----~----~------~----~------~--~---
