Re: Problems connecting to iSCSI target from bonded interface
On 12/07/2011 08:34 PM, Albert Chin wrote: On Wed, Dec 07, 2011 at 08:23:35PM -0600, Albert Chin wrote: On Wed, Dec 07, 2011 at 05:44:14PM -0600, Mike Christie wrote: Also for your kernel you will need to set the net.ipv4.conf.default.rp_filter to 0 or 2. What does this do? Found documentation for this but why is rp_filter=1 bad? That's what it's currently set to. Ditto for net.ipv4.conf.all.rp_filter. Are you recommending it because of the dropped packets we're seeing on bond1: Does it work with rp_filter=1 now again? In recent kernels with this commit: (git commit 27fed4175acf81ddd91d9a4ee2fd298981f60295 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=27fed4175acf81ddd91d9a4ee2fd298981f60295) they changed the behavior so that you needed to set it to 0 or 2, or packets would not be delivered to the initiator when using iface binding which uses SO_BINDTODEVICE. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Problems connecting to iSCSI target from bonded interface
On Thu, Dec 08, 2011 at 02:37:48PM -0600, Mike Christie wrote: On 12/07/2011 08:34 PM, Albert Chin wrote: On Wed, Dec 07, 2011 at 08:23:35PM -0600, Albert Chin wrote: On Wed, Dec 07, 2011 at 05:44:14PM -0600, Mike Christie wrote: Also for your kernel you will need to set the net.ipv4.conf.default.rp_filter to 0 or 2. What does this do? Found documentation for this but why is rp_filter=1 bad? That's what it's currently set to. Ditto for net.ipv4.conf.all.rp_filter. Are you recommending it because of the dropped packets we're seeing on bond1: Does it work with rp_filter=1 now again? In recent kernels with this commit: Yes. As soon as I modified it to bind to bond1, it worke.d -- albert chin (ch...@thewrittenword.com) -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Problems connecting to iSCSI target from bonded interface
We're running Ubuntu 11.10 with the 3.0.0-12-server kernel. We have the open-iscsi software successfully connecting to our OpenSolaris COMSTAR target using one of the GigE interfaces on the server (Intel SR2625URLX). The problem is that when we try to connect using a 802.3ad bonded interface comprised of two GigE NICs, we're seeing some odd network behaviour. First, the successful config: trunks:/# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:1e:67:1c:fd:d4 inet addr:10.191.57.22 Bcast:10.191.57.255 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:fe1c:fdd4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1268 errors:0 dropped:0 overruns:0 frame:0 TX packets:744 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:711237 (711.2 KB) TX bytes:205042 (205.0 KB) Memory:b1d2-b1d4 # ifconfig eth1 eth1 Link encap:Ethernet HWaddr 00:1e:67:1c:fd:d5 inet addr:10.191.58.9 Bcast:10.191.58.255 Mask:255.255.255.0 inet6 addr: fe80::21e:67ff:fe1c:fdd5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:42 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2520 (2.5 KB) TX bytes:468 (468.0 B) Memory:b1d0-b1d2 # netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.191.57.254 0.0.0.0 UG0 0 0 eth0 10.191.57.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.191.58.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty # iscsiadm -m iface -I iface0 --op=new New interface iface0 added # iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress \ -v 00:1e:67:1c:fd:d5 iface0 updated. # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty iface0 tcp,00:1e:67:1c:fd:d5,empty,empty,empty # iscsiadm -m discovery -t st -p 10.191.58.1:3260 -I iface0 -P 1 ... Target: iqn.1986-03.com.sun:02:cea07f79-828f-ebcd-9d06-b8b0c3e8e712 Portal: 10.191.58.1:3260,2 Iface Name: iface0 # iscsiadm -m node \ -T iqn.1986-03.com.sun:02:cea07f79-828f-ebcd-9d06-b8b0c3e8e712 \ -p 10.191.58.1:3260 -I iface0 -l Logging in to [iface: iface0, target: iqn.1986-03.com.sun:02:cea07f79-828f-ebcd-9d06-b8b0c3e8e712, portal: 10.191.58.1,3260] Login to [iface: iface0, target: iqn.1986-03.com.sun:02:cea07f79-828f-ebcd-9d06-b8b0c3e8e712, portal: 10.191.58.1,3260]: successful # netstat -an | grep 10.191.58 tcp0 0 10.191.58.9:52393 10.191.58.1:3260 ESTABLISHED Now, if I try the above but with the bonded interface: # ifconfig bond1 bond1 Link encap:Ethernet HWaddr 00:1b:21:d3:f6:08 inet addr:10.191.58.9 Bcast:10.191.58.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fed3:f608/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets:33 errors:0 dropped:31 overruns:0 frame:0 TX packets:799 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3964 (3.9 KB) TX bytes:98800 (98.8 KB) # cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 200 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 17 Partner Key: 26 Partner Mac Address: 00:04:96:18:54:d5 Slave Interface: eth5 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1b:21:d3:f6:09 Aggregator ID: 1 Slave queue ID: 0 Slave Interface: eth4 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1b:21:d3:f6:08 Aggregator ID: 1 Slave queue ID: 0 # netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.191.57.254 0.0.0.0 UG0 0 0 eth0 10.191.57.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.191.58.0 0.0.0.0 255.255.255.0 U 0 0 0 bond1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty # iscsiadm -m iface -I iface0 --op=new New interface iface0 added # iscsiadm -m iface -I iface0 --op=update
Re: Problems connecting to iSCSI target from bonded interface
On 12/06/2011 11:49 PM, Albert Chin wrote: # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty # iscsiadm -m iface -I iface0 --op=new New interface iface0 added # iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress \ -v 00:1b:21:d3:f6:08 iface0 updated. # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty iface0 tcp,00:1b:21:d3:f6:08,empty,empty,empty Instead of hwaddress use the iface.net_ifacename setting. So delete the old iface iscsiadm -m iface -I iface -o delete then do iscsiadm -m iface -I iface0 --op=new iscsiadm -m iface -I iface0 --op=update -n iface.net_ifacename -v bond1 Then do discovery again then login. The iscsi tools will get confused and possibly pick the wrong interface to bind to, because bond1 has mac 00:1b:21:d3:f6:08 but then so did eth6 and we do not know which one you really meant. Also for your kernel you will need to set the net.ipv4.conf.default.rp_filter to 0 or 2. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Problems connecting to iSCSI target from bonded interface
On Wed, Dec 07, 2011 at 05:44:14PM -0600, Mike Christie wrote: On 12/06/2011 11:49 PM, Albert Chin wrote: # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty # iscsiadm -m iface -I iface0 --op=new New interface iface0 added # iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress \ -v 00:1b:21:d3:f6:08 iface0 updated. # iscsiadm -m iface default tcp,empty,empty,empty,empty iser iser,empty,empty,empty,empty iface0 tcp,00:1b:21:d3:f6:08,empty,empty,empty Instead of hwaddress use the iface.net_ifacename setting. So delete the old iface iscsiadm -m iface -I iface -o delete then do iscsiadm -m iface -I iface0 --op=new iscsiadm -m iface -I iface0 --op=update -n iface.net_ifacename -v bond1 Then do discovery again then login. The iscsi tools will get confused and possibly pick the wrong interface to bind to, because bond1 has mac 00:1b:21:d3:f6:08 but then so did eth6 and we do not know which one you really meant. Yeah. Ran iscsid in debug mode and saw this. Thanks. Also for your kernel you will need to set the net.ipv4.conf.default.rp_filter to 0 or 2. What does this do? -- albert chin (ch...@thewrittenword.com) -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Problems connecting to iSCSI target from bonded interface
On Wed, Dec 07, 2011 at 08:23:35PM -0600, Albert Chin wrote: On Wed, Dec 07, 2011 at 05:44:14PM -0600, Mike Christie wrote: Also for your kernel you will need to set the net.ipv4.conf.default.rp_filter to 0 or 2. What does this do? Found documentation for this but why is rp_filter=1 bad? That's what it's currently set to. Ditto for net.ipv4.conf.all.rp_filter. Are you recommending it because of the dropped packets we're seeing on bond1: # ifconfig bond1 bond1 Link encap:Ethernet HWaddr 00:1b:21:d3:f6:08 inet addr:10.191.58.9 Bcast:10.191.58.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fed3:f608/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets:9839923 errors:0 dropped:4434 overruns:0 frame:0 TX packets:15240717 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2224524704 (2.2 GB) TX bytes:81839661996 (81.8 GB) We see the same issue with dropped packets on the first bonded interface, bond0: # ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:1b:21:d3:f6:0a UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:15989558 errors:0 dropped:4436 overruns:0 frame:0 TX packets:8623601 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15769948311 (15.7 GB) TX bytes:2909403062 (2.9 GB) -- albert chin (ch...@thewrittenword.com) -- You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
