Jurgen,
J?rgen Keil wrote:
> Assuming I have a system with gldv3 ethernet nic, where the external link is
> down,
> and the build 66 xen bits: Shouldn't it be possible to communicate between a
> Solaris domU and the host Solaris dom0?
Yes, that path should be preserved when the physical link is down.
> It seems that (with build66 xen) the local network connection between domU
> and dom0
> only works, when the external link is up (and IPMP is disabled).
Thanks for the report, yes it's a bug. The link state of the primary
interface (in this case the interface plumbed by dom0) passed to DLS
should be reported as being up if a VNIC is defined on the same NIC. I
filed 6599495.
Thanks,
Nicolas.
>
> Details:
>
> Tecra S1 laptop, iprb and ipw nics, Solaris dom0, and Solaris domU.
> xen b66 bits, iprb glv3 nic driver copied from snv_71.
>
> System is not connected to a network (at least not on iprb, ipw is unplumbed)
>
> # dladm show-dev
> ipw0 link: up speed: 0Mb duplex: unknown
> iprb0 link: down speed: 0Mb duplex: half
>
> # ifconfig -a
> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
> index 1
> inet 127.0.0.1 netmask ff000000
> iprb0: flags=201000803<UP,BROADCAST,MULTICAST,IPv4,CoS> mtu 1500 index 2
> inet 192.168.2.21 netmask ffffff00 broadcast 192.168.2.255
> groupname mobile
> ether 0:a0:d1:d5:bc:b4
> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
> index 1
> inet6 ::1/128
> iprb0: flags=202000800<MULTICAST,IPv6,CoS> mtu 1500 index 2
> inet6 fe80::2a0:d1ff:fed5:bcb4/10
> groupname mobile
> ether 0:a0:d1:d5:bc:b4
> ipw0: flags=202100840<RUNNING,MULTICAST,ROUTER,IPv6,CoS> mtu 1500 index 3
> inet6 fe80::204:23ff:fe4f:69bf/10
> ether 0:4:23:4f:69:bf
> ipw0:1: flags=202180841<UP,RUNNING,MULTICAST,ADDRCONF,ROUTER,IPv6,CoS> mtu
> 1500 index 3
> inet6 fec0::2:204:23ff:fe4f:69bf/64
>
> # arp -an
> Net to Media Table: IPv4
> Device IP Address Mask Flags Phys Addr
> ------ -------------------- --------------- -------- ---------------
> iprb0 192.168.2.21 255.255.255.255 SPLdA 00:a0:d1:d5:bc:b4
> iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00
>
> Note: Flags 'd' for iprb0's 192.168.2.21 address:
>
> d Unverified; this is a local IP address that is
> currently undergoing Duplicate Address Detection.
> ARP will not respond to requests for this address
> until Duplicate Address Detection completes.
>
>
> Problem: no network connectivity from domU to dom0. Test is an attempt to
> telnet from
> domU to ip address 192.168.2.21 (ip address of iprb0 in dom0).
>
> Problem #1:
> ===========
>
> arp requests from domU to dom0 are dropped by dom0. dtrace show this:
>
> ...
> 0 -> putnext
> 0 -> ar_rput
> 0 | ar_rput:rput_normal
> 0 | ar_rput:arp-physical-in-start
> 0 | ar_rput:arp-physical-in-end
> 0 -> ar_ce_resolve_all
> 0 -> ar_ce_hash
> 0 <- ar_ce_hash returns d57bc35c
> 0 <- ar_ce_resolve_all returns 1
> 0 -> ar_ce_lookup_entry
> 0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800,
> dst_paddr 192.168.2.21 plen 4
> 0 -> ar_ce_hash
> 0 <- ar_ce_hash returns d57bc450
> 0 <- ar_ce_lookup_entry returns e9586654
> 0 -> bcmp
> 0 <- bcmp returns 1
> 0 | ar_rput:rput_request_unverified
> 0 -> freemsg
> 0 -> dblk_lastfree_desb
> 0 -> xnb_rx_complete
> 0 -> xnb_rx_schedule_unmop
> 0 <- xnb_rx_schedule_unmop returns e0687000
> 0 -> xnb_rx_perform_pending_unmop
> 0 -> HYPERVISOR_grant_table_op
> 0 <- HYPERVISOR_grant_table_op returns 0
> 0 <- xnb_rx_perform_pending_unmop returns 0
> ...
>
>
> usr/src/uts/common/inet/arp/arp.c:
>
> 3280 /*
> 3281 * If the target is unverified by DAD, then one of two things
> is true:
> 3282 * either it's someone else claiming this address (on a probe
> or an
> 3283 * announcement) or it's just a regular request. The former
> is
> 3284 * failure, but a regular request is not.
> 3285 */
> 3286 if (dst_ace->ace_flags & ACE_F_UNVERIFIED) {
> 3287 /*
> 3288 * Check for a reflection. Some misbehaving bridges
> will
> 3289 * reflect our own transmitted packets back to us.
> 3290 */
> 3291 if (hlen == dst_ace->ace_hw_addr_length &&
> 3292 bcmp(src_haddr, dst_ace->ace_hw_addr, hlen) == 0)
> {
> 3293 DTRACE_PROBE3(rput_probe_reflected, arl_t *,
> arl,
> 3294 arh_t *, arh, ace_t *, dst_ace);
> 3295 freeb(mp);
> 3296 freemsg(mp1);
> 3297 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END,
> 3298 "arp_rput_end: q %p (%S)", q,
> "reflection");
> 3299 return;
> 3300 }
> 3301 if (is_probe || op == ARP_RESPONSE) {
> 3302 ar_client_notify(arl, mp1, AR_CN_FAILED);
> 3303 ar_ce_delete(dst_ace);
> 3304 } else if (err == AR_CHANGED) {
> 3305 ar_client_notify(arl, mp1, AR_CN_ANNOUNCE);
> 3306 } else {
> 3307 DTRACE_PROBE3(rput_request_unverified, arl_t
> *, arl,
> 3308 arh_t *, arh, ace_t *, dst_ace);
> 3309 freemsg(mp1);
> 3310 }
> 3311 freeb(mp);
> 3312 TRACE_2(TR_FAC_ARP, TR_ARP_RPUT_END,
> 3313 "arp_rput_end: q %p (%S)", q, "unverified");
> 3314 return;
> 3315 }
>
> Problem #2:
> ===========
>
> ARP duplicate address detection can be disabled, but there still isn't
> "network" connectivity between domU and dom0:
>
> # ndd -set /dev/arp arp_fastprobe_count 0
> # ifconfig iprb0 inet 0
> # ifconfig iprb0 inet max
> # arp -an
> Net to Media Table: IPv4
> Device IP Address Mask Flags Phys Addr
> ------ -------------------- --------------- -------- ---------------
> iprb0 192.168.2.21 255.255.255.255 SPLA 00:a0:d1:d5:bc:b4
> iprb0 224.0.0.0 240.0.0.0 SM 01:00:5e:00:00:00
>
>
> dtrace shows that we're now trying to xmit something, but the response
> messages seem to be dropped immediately in ar_xmit():
>
> 0 -> putnext
> 0 -> ar_rput
> 0 | ar_rput:rput_normal
> 0 | ar_rput:arp-physical-in-start
> 0 | ar_rput:arp-physical-in-end
> 0 -> ar_ce_resolve_all
> 0 -> ar_ce_hash
> 0 <- ar_ce_hash returns d57bc35c
> 0 -> ar_ce_resolve
> 0 -> bcmp
> 0 <- bcmp returns 0
> 0 <- ar_ce_resolve returns 0
> 0 <- ar_ce_resolve_all returns 2
> 0 -> ar_ce_lookup_entry
> 0 | ar_ce_lookup_entry:entry arl d7c05774, proto 800,
> dst_paddr 192.168.2.21 plen 4
> 0 -> ar_ce_hash
> 0 <- ar_ce_hash returns d57bc450
> 0 <- ar_ce_lookup_entry returns e95861d4
> 0 -> ddi_get_lbolt
> 0 <- ddi_get_lbolt returns 767f2
> 0 -> ar_xmit
> 0 | ar_xmit:entry arl d7c05774 ap d78136c0
> ap->ap_link_down 1
> 0 <- ar_xmit returns 3
> 0 -> freemsg
> 0 -> dblk_lastfree
> 0 -> kmem_cache_free
> 0 <- kmem_cache_free returns a4a1
> 0 <- dblk_lastfree returns a4a1
> 0 -> dblk_lastfree_desb
> 0 -> xnb_rx_complete
> 0 -> xnb_rx_schedule_unmop
> 0 <- xnb_rx_schedule_unmop returns e0687000
>
> It seems arp does not send replies because the iprb0 link is down:
>
> 4271 /* ar_xmit is called to transmit an ARP Request or Response. */
> 4272 static void
> 4273 ar_xmit(arl_t *arl, uint32_t operation, uint32_t proto, uint32_t plen,
> 4274 const uchar_t *haddr1, const uchar_t *paddr1, const uchar_t
> *haddr2,
> 4275 const uchar_t *paddr2, const uchar_t *dstaddr, arp_stack_t *as)
> 4276 {
> 4277 arh_t *arh;
> 4278 uint8_t *cp;
> 4279 uint_t hlen;
> 4280 mblk_t *mp;
> 4281 arlphy_t *ap = arl->arl_phy;
> 4282
> 4283 if (ap == NULL) {
> 4284 DTRACE_PROBE1(xmit_no_arl_phy, arl_t *, arl);
> 4285 return;
> 4286 }
> 4287
> 4288 /* IFF_NOARP flag is set or link down: do not send arp
> messages */
> 4289 if ((arl->arl_flags & ARL_F_NOARP) || ap->ap_link_down)
> 4290 return;
>
>
> # mdb -k
> Loading modules: [ unix genunix specfs dtrace xpv_uppc scsi_vhci zfs random
> ip hook neti sctp arp usba uhci s1394 nca fctl lofs audiosup md crypto fcip
> fcp logindmux ptm ufs sppp nfs ipc ]
>> d78136c0::print arlphy_t
> {
> ap_arp_hw_type = 0x1
> ap_arp_addr = 0xd7d2a528
> ap_hw_addr = 0xd8c96d40
> ap_hw_addrlen = 0x6
> ap_xmit_mp = 0xd8c9a020
> ap_xmit_addroff = 0x14
> ap_xmit_sapoff = 0x1a
> ap_saplen = 0xfffffffe
> ap_defend_start = 0
> ap_defend_count = 0
> ap_notifies = 0x1
> ap_link_down = 0x1
> }
>> d7c05774::print arl_t
> {
> arl_next = 0
> arl_rq = 0xd8c98e30
> arl_wq = 0xd8c98eb0
> arl_ppa = 0
> arl_name = [ "iprb0" ]
> arl_unbind_mp = 0xd4289a00
> arl_detach_mp = 0
> arl_provider_style = 0x500
> arl_queue = 0
> arl_queue_tail = 0
> arl_flags = 0
> arl_dlpi_pending = 0xffff
> arl_dlpi_deferred = 0
> arl_state = 0x2
> arl_closing = 0
> arl_index = 0x1
> arl_phy = 0xd78136c0
> }
>
> Problem #3:
> ===========
>
> After connecting iprb0 to a switch; link is now up. interface setup still
> has a
> groupname. Still no network connectivity between dom0 and domU.
>
> # dladm show-dev
> ipw0 Link: up speed: 0Mb Duplex: unknown
> iprb0 Link: up speed: 100Mb Duplex: full
> vnic0 Link: unknown speed: 0Mb Duplex: unknown
> # ifconfig iprb0 group "abc"
> # ifconfig iprb0
> iprb0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500
> index 2
> inet 192.168.2.21 netmask ff000000 broadcast 192.255.255.255
> groupname abc
> ether 0:a0:d1:d5:bc:b4
>
> After deleting the groupname, it's finally possible to telnet from domU back
> into dom0...
>
>
> This message posted from opensolaris.org
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
--
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
droux at sun.com - http://blogs.sun.com/droux
