Thanks for the response Jim. I actually came across these bugs while doing my
due diligence before posting to the forums. I tried the workarounds but am
still seeing the issue. Once the network hang occurs and I try an arp -a it
just hangs until I reset the interface. Not sure if there is something I'm
missing when setting the arp intervals and static arp entries.
-Marc
Before issue:
r...@opensolaris:~# ndd /dev/ip ip_ire_arp_interval
86400000
r...@opensolaris:~# ndd /dev/arp arp_cleanup_interval
3600000
r...@opensolaris:~# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
e1000g0 10.0.1.1 255.255.255.255 o 00:1f:f3:42:73:a0
e1000g0 10.0.1.4 255.255.255.255 o 00:1e:52:76:14:f2
e1000g0 10.0.1.200 255.255.255.255 SPLA 00:15:58:2e:af:8f
e1000g0 BASE-ADDRESS.MCAST.NET 240.0.0.0 SM 01:00:5e:00:00:00
r...@opensolaris:~# arp -s 10.0.1.1 00:1f:f3:42:73:a0
r...@opensolaris:~# arp -s 10.0.1.4 00:1e:52:76:14:f2
r...@opensolaris:~# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
e1000g0 10.0.1.1 255.255.255.255 S 00:1f:f3:42:73:a0
e1000g0 10.0.1.4 255.255.255.255 S 00:1e:52:76:14:f2
e1000g0 10.0.1.200 255.255.255.255 SPLA 00:15:58:2e:af:8f
e1000g0 BASE-ADDRESS.MCAST.NET 240.0.0.0 SM 01:00:5e:00:00:00
r...@opensolaris:~#
After Issue:
r...@opensolaris:~# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- -------- ---------------
^C
r...@opensolaris:~# ndd /dev/arp arp_cleanup_interval
3600000
r...@opensolaris:~# ndd /dev/ip ip_ire_arp_interval
86400000
r...@opensolaris:~#
> Date: Mon, 12 Jan 2009 15:32:53 -0500
> From: [email protected]
> Subject: Re: [storage-discuss] ISCI Network Hang - Lun becomes unavailable
> To: [email protected]
> CC: [email protected]
>
> M wrote:
>
> Under heavy load testing, instances of this problem have been seen and
> resolved on OpenSolaris.
>
> http://bugs.opensolaris.org/view_bug.do?bug_id=6638604
> http://bugs.opensolaris.org/view_bug.do?bug_id=6680855
>
> Like 6637806, it is possible that ARP cache policies on Mac OS X, in
> combination with iSCSI timeout polices do no scale on a network that
> is under load. Review the technical details and work-around associated
> with 6637806, and see of static ARP resolves your issues. If yes,
> then you will need to ask the Mac OS X forum about their ARP caching
> policies, and how to change them.
>
> Regards,
>
> Jim Dunham
>
>
>
> > I am sharing out ZFS ISCSI luns to my Mac. When copying large files,
> > the network will hang in the middle of the transfer and the LUN will
> > become unavailable until I plumb the NIC. This issue appears to only
> > occur only when I am reading files (ie..syncing an ipod) and not
> > writing (I'm not 100% sure though). When I snoop the interface I
> > notice a bunch of ARP lookups. Any ideas? Thanks in advanced.
> >
> > Solaris Build: OpenSolaris 2008.11 snv_101b_rc2 X86
> > Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
> > Use is subject to license terms.
> > Assembled 19 November 2008
> >
> >
> > Below is the output from snoop. 10.0.1.200 is the Solaris server and
> > 10.0.1.4 is the Mac client.
> >
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072298406 Len=2352 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072300758 Len=4390 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072305148 Len=2920 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072308068 Len=1700 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072309768 Len=2920 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072312688 Len=5320 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072318008 Len=2920 Win=64240
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072195178
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072198098
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072203178
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072206098
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072320928 Len=5320 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072326248 Len=2920 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072329168 Len=5080 Win=64240
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Push Ack=405482834
> > Seq=1072334248 Len=2920 Win=64240
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072209958
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072212878
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Ack=1072215278
> > Seq=405482834 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 362)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 362)
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51699 Push Ack=133176904
> > Seq=1610109515 Len=48 Win=65535 Options=<nop,nop,tstamp 628354654
> > 67148>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Ack=1610109563
> > Seq=133176952 Len=0 Win=65160 Options=<nop,nop,tstamp 67569 628354643>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Push Ack=1610109563
> > Seq=133176904 Len=48 Win=65160 Options=<nop,nop,tstamp 67582
> > 628354643>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Ack=405483698
> > Seq=1073633082 Len=1460 Win=64240
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 363)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 363)
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 364)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 364)
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51699 Push Ack=133176904
> > Seq=1610109515 Len=48 Win=65535 Options=<nop,nop,tstamp 628354670
> > 67148>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Ack=1610109563
> > Seq=133176952 Len=0 Win=65160 Options=<nop,nop,tstamp 67729 628354643>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Push Ack=1610109563
> > Seq=133176904 Len=48 Win=65160 Options=<nop,nop,tstamp 67753
> > 628354643>
> > 10.0.1.200 -> 10.0.1.1 DNS C 250.255.255.239.in-addr.arpa.
> > Internet PTR ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 365)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 365)
> > 10.0.1.200 -> 10.0.1.1 DNS C _nfsv4idmapdomain.nyc.rr.com.
> > Internet TXT ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 366)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 366)
> > 10.0.1.200 -> 10.0.1.4 TCP D=51683 S=3260 Ack=405483698
> > Seq=1073633082 Len=1460 Win=64240
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 367)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 367)
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51699 Push Ack=133176904
> > Seq=1610109515 Len=48 Win=65535 Options=<nop,nop,tstamp 628354702
> > 67148>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Ack=1610109563
> > Seq=133176952 Len=0 Win=65160 Options=<nop,nop,tstamp 68050 628354643>
> > 10.0.1.200 -> 10.0.1.4 TCP D=51699 S=3260 Push Ack=1610109563
> > Seq=133176904 Len=48 Win=65160 Options=<nop,nop,tstamp 68094
> > 628354643>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 368)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 368)
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 369)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 369)
> > 10.0.1.200 -> 10.0.1.1 DNS C 250.255.255.239.in-addr.arpa.
> > Internet PTR ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 370)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 370)
> > 10.0.1.200 -> 10.0.1.1 DNS C _nfsv4idmapdomain. Internet TXT ?
> > 10.0.1.4 -> 10.0.1.255 UDP D=631 S=631 LEN=140
> > 10.0.1.200 -> 10.0.1.1 DNS C 255.1.0.10.in-addr.arpa. Internet
> > PTR ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 371)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 371)
> > 10.0.1.4 -> 10.0.1.255 UDP D=631 S=631 LEN=175
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 372)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 372)
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 373)
> > 10.0.1.200 -> 10.0.1.4 ICMP Echo reply (ID: 40450 Sequence
> > number: 373)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51699 Push Ack=133176904
> > Seq=1610109515 Len=48 Win=65535 Options=<nop,nop,tstamp 628354766
> > 67148>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 374)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,nop,wscale 1,nop,nop,tstamp
> > 628354778 0,sackOK,eol>
> > 10.0.1.200 -> 10.0.1.1 DNS C 250.255.255.239.in-addr.arpa.
> > Internet PTR ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 375)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.200 -> 10.0.1.1 DNS C _nfsv4idmapdomain. Internet TXT ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,nop,wscale 1,nop,nop,tstamp
> > 628354787 0,sackOK,eol>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 376)
> > 10.0.1.200 -> 10.0.1.1 DNS C 255.1.0.10.in-addr.arpa. Internet
> > PTR ?
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,nop,wscale 1,nop,nop,tstamp
> > 628354797 0,sackOK,eol>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 377)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,sackOK,eol>
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 378)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,sackOK,eol>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 379)
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,sackOK,eol>
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.200 -> 10.0.1.1 DNS C 250.255.255.239.in-addr.arpa.
> > Internet PTR ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 380)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 381)
> > 10.0.1.200 -> 10.0.1.1 DNS C 255.1.0.10.in-addr.arpa. Internet
> > PTR ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51683 Fin Ack=1073633082
> > Seq=405483698 Len=0 Win=65535
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > 10.0.1.4 -> 10.0.1.200 TCP D=3260 S=51734 Syn Seq=3476025541
> > Len=0 Win=65535 Options=<mss 1460,sackOK,eol>
> > 10.0.1.4 -> 10.0.1.200 ICMP Echo request (ID: 40450 Sequence
> > number: 382)
> > 10.0.1.200 -> (broadcast) ARP C Who is 10.0.1.4, 10.0.1.4 ?
> > --
> > This message posted from opensolaris.org
> > _______________________________________________
> > storage-discuss mailing list
> > [email protected]
> > http://mail.opensolaris.org/mailman/listinfo/storage-discuss
>
_________________________________________________________________
Windows Live™ Hotmail®: Chat. Store. Share. Do more with mail.
http://windowslive.com/howitworks?ocid=TXT_TAGLM_WL_t1_hm_justgotbetter_howitworks_012009
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
