Re: [zones-discuss] reviews for 6933462 onu should support zones
> '...' substitutions won't be in POSIX anymore. That may be, but we would have to absolutely hate our customers to remove support for this construct from our shells, so the positions of POSIX on the matter seem a bit academic. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] s10 brand Phase I webrev
> > I don't see how that addresses the primary point, which is that Solaris > > brands seem to suffer from code duplication. Are you asserting that the > > amount of code duplication between the sn1 and solaris10 brands is unique > > to that situation and is not something that will occur again when we cons > > up the next solarisX brand? > > Yes. If we were to ship another brand that was > fundamentally similar to the solaris10 brand (e.g. the solaris9 > brand on Solaris Next), then I think it would make sense for > that project to try to make more of the code common with the > then currently shipping solaris10 brand. However, I don't think > that is necessary for a brand we don't ship (but I can also > understand if you don't agree with me). Indeed, I don't agree, for three reasons: 1. You are establishing a precedent that the way we develop new Solaris brands is via cut-and-paste in a manner that's expedient for the implementor. When the next brand comes along, this existing example will likely be used as justification for why they should not be "burdened" with the task of properly factoring the software. 2. Shipping or not, the sn1 codebase is still part of our product and needs to be maintained. By forking the source, we are doubling the cost of that maintenance and making it easy for things to get out of sync (and indeed, the tax of maintaining these parallel brands has already started to be levied before the solaris10 brand has even integrated). We've all seen this many times but yet we keep doing it. 3. It is precisely because of our limited resources that we must adhere to best practices and commonize code rather than fork it. All that said, ultimately this is an issue for you and the rest of the zones team to resolve, so I'll bow out of this, but I strongly disagree with the approach that's been taken. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] s10 brand Phase I webrev
> > I was wondering about this too. Indeed, there seems be a sizeable amount > > of duplicated code now. Why is this the right design? > > Because the sn1 brand is an internal brand for testing > and is not delivered to customers. Once the solaris10 > brand is integrated, the sn1 brand will no longer serve > its original role and could even be removed from the source > tree if we wanted to. I don't see how that addresses the primary point, which is that Solaris brands seem to suffer from code duplication. Are you asserting that the amount of code duplication between the sn1 and solaris10 brands is unique to that situation and is not something that will occur again when we cons up the next solarisX brand? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] s10 brand Phase I webrev
> also, i would have though you'd commited to doing this work when you > decided to fork the sn1 brand code instead of making it common. I was wondering about this too. Indeed, there seems be a sizeable amount of duplicated code now. Why is this the right design? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Can't set up AutoInstall in a non-global zone with exclusive IP
> On a related topic, is there a reason you can't "snoop -d /dev/igb1" in > the non-global zone, where igb1 is the NIC dedicated to the zone? That should work fine (well, "snoop -d igb1" should). -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
> > Do a "man snoop" and search for the word "zone". > > My argument wasn't that there were zero bugs in the OS. That keyword > seems to me to be pretty clearly a defect in snoop. (And apparently a > recent one; less than a year old.) ... and indeed, there's a CR open to allow snoop to accept zone names. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] s10 brand Phase I webrev
> We've completed the development for the Phase I > work on the solaris10 brand. I've posted a > full webrev at: > > http://cr.opensolaris.org/~gjelinek/webrev.646/ > > Let me know if there are any comments. I see that ip-type=exclusive is regarded as "experimental" in s10_support.c; is that planned for Phase II? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S10 brand spec.
> > As I've mentioned in the past to Dan, it's worth noting that Crossbow is > > not the only project that has significant impact here -- e.g., Clearview > > UV and Clearview IPMP (among other projects) also made significant changes > > to both kernel and userland that are presumed to be in lockstep. I > > suspect you will be safe with shared-stack, but exclusive stack will pose > > some significant challenges, and in some cases you may find the path of > > least resistance is to use the S11 userland binaries inside the S10 zone. > > Using the S11 binaries might be what we wind up doing > for this. We haven't started to investigate the exclusive > stack yet, so we don't know all of the issues. We do > already have a mechanism for running binaries from the > global zone and we do use that in some other cases already. > One concern might be around 3rd party S10 apps that want to > do networking stuff directly. I'm not sure how common > that would be or what, if anything, would not work. That should be pretty uncommon as the affected interfaces are unstable, undocumented, and undergoing fairly regular change. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S10 brand spec.
> Part 2: solaris10 Brand > > > The solaris10 brand is conceptually similar to the existing solaris8 > and solaris9 brands and builds directly on the BrandZ infrastructure > that was created to support the lx brand. Familiarity with BrandZ > and the solaris8 and solaris9 brands is assumed. > > At this time the design and development of the brand is only > supporting the shared stack [6] networking model in which the zone's > network is managed by the global zone. The exclusive stack model > is anticipated to require more complex solutions or emulation due > to the introduction of Crossbow [7] into S.next. The exclusive > stack issues will be resolved before commitment review. Jerry, As I've mentioned in the past to Dan, it's worth noting that Crossbow is not the only project that has significant impact here -- e.g., Clearview UV and Clearview IPMP (among other projects) also made significant changes to both kernel and userland that are presumed to be in lockstep. I suspect you will be safe with shared-stack, but exclusive stack will pose some significant challenges, and in some cases you may find the path of least resistance is to use the S11 userland binaries inside the S10 zone. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] solaris10 brand project proposal
> I admit, I'm curious what needs to be emulated? I would think that > Solaris 10 zones would work just fine under OpenSolaris as native > branded zones. >From the networking side at least, there's a fair bit of work here. In particular, there are a number of features that have been removed (e.g. implicit VLANs) or have changed dramatically (IPMP, GLDv3) and will require some cleverness (possibly significant cleverness) to preserve the S10 experience when run atop a Nevada kernel. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] native p2v code review
> > Thanks for looking at this. I'll take a look > > at each of your recommendations. One factor > > is that we plan on backporting this to S10, so > > I want to keep things working on that release > > with a minimum of changes. > > Erm... the warnings returned by $ ksh93 -n # apply to ksh88 > (e.g. /usr/bin/ksh in Solaris <= 11/B106 and /usr/xpg4/bin/sh), too. > This features does not enforce any feature outside the POSIX shell > standard (but checks - if they are used - ksh88&&ksh93-specific > features, too). > Or short: These issues should be fixed for Solaris 10, too (as a > side-effect you get more performance for the script (even on Solaris > 10), too). I didn't see any response to my question about whether the OpenSolaris (or perhaps just ON) community had reached agreement on the advice offered by shlint. I could name several dozen personal best practices that I wish every C program in ON followed; that doesn't give me the right to just update a private version of cstyle and start enforcing those things each time someone posts a code review. Please start with getting consensus on shlint. If you're unable to get consensus with what you have, then you need to modify your proposal until you do. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] ip_restrict_interzone_loopback again
> You mean if zone1 and zone2 were plumbed on e1000g1:1, and e1000g1:2, > traffic will never be observable no matter what. I can live with > this. All IP packets can be observed as of Nevada build 103; see http://blogs.sun.com/seb/ for some examples with zones. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] native p2v code review
> "p2v.ksh" triggers lots of warnings when scanned via $ shlint p2v.ksh # Has consensus been reached in the OpenSolaris community on the advice offered by shlint? If not, it would seem that needs to be done first. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S10U6 zone features
> Vanity naming is not a zone feature, the reason I mentioned it is that > it wold be a nice feature when migrating zones between machines with > different NIC:s. Indeed, this was one of the uses we had in mind for the feature (and of course, you can use it today with OpenSolaris :-) -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] The quick & dirty guide to zones on iSCSI LUNs
> The original question here was about what sort of source selection > behavior to expect, and what happens when the "wrong" routes are > present when a TCP connection is attempted. As far as the user is > concerned, the answers are (1) you must discard a TCP socket if it > fails to connect and you're going to try again and (2) when you try > again, you get another roll of the dice as far as forwarding and > source selection is concerned. If something new is available, you may > well end up using it. > > The fact that we cache and that our caching has limitations doesn't > really enter into that discussion. I never said it did. I just wanted to make it clear that our implementation does not actually do source address selection on each packet, even if the observed behavior is indistuishable from one that does. Yes, a small point. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] The quick & dirty guide to zones on iSCSI LUNs
> You're assuming the packets sent on this UDP socket all have the same > destination and that routing does not change between transmissions. I'm not assuming that, just pointing out that in general there will not be a different source address in each packet, and that for a given destination we latch the source address for long periods of time. That is, one could read your original statement as a promise that we will repeat source address selection for each packet, which is not the case. (And yes, I realize this is an implementation detail. Source address selection algorithms for an unconnected and unbound UDP socket seems entirely like "implementation-defined behavior" to me.) -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] The quick & dirty guide to zones on iSCSI LUNs
> If you don't explicitly bind a preferred address to use (most
> applications do not), then the kernel will choose an address for you.
> With UDP, this happens on a packet-by-packet basis.
Really? I'd expect the first packet to a given destination to construct
an IRE cache entry, and then for future packets to go through the fastpath
in udp_send_data(), which will then set the source address consistently
from ire_src_addr:
if (src == INADDR_ANY && !connp->conn_unspec_src) {
if (CLASSD(dst) && !(ire->ire_flags & RTF_SETSRC))
ipha->ipha_src = ipif->ipif_src_addr;
else
--> ipha->ipha_src = ire->ire_src_addr;
}
if (ipif != NULL)
ipif_refrele(ipif);
udp_xmit(connp->conn_wq, mp, ire, connp, connp->conn_zoneid);
Of course, the IRE will have IRE_MARK_TEMPORARY set, so the IRE may not
last *all* that long, but source address selection shouldn't occur on a
packet-by-packet basis.
--
meem
___
zones-discuss mailing list
zones-discuss@opensolaris.org
Re: [zones-discuss] dhcp/zone
> > > Contrary to what everyone else says, with a bit of customization dhcpd > > > can run zones just fine. Solaris 10 update 3 and later (I forget > > > which Nevada build) have all the required bits in place. > > > > > > > > http://mail.opensolaris.org/pipermail/install-discuss/2007-March/004266.html > > > > Yep, it can be made to work, though I don't think it's officially > > supported. When I last looked at this issue, the changes needed to the > > source to make it work without any special tweaks was straightforward. > > If you offer up hints, I make take this on in the future. I've got a > couple other things I want to work on before this one, though. It's been a while, but the main things I recall are: * The DHCP server used to open /dev/ip rather than using a UDP socket to do some control operations, which forced it to require more privileges than it really needed. This was fixed via 6357132 in Nevada build 36. * The DHCP server is generally designed to work with physical IP interfaces. Some work would be needed to allow it to work with logicals (if that's even desirable); 5030697 touches on this. * The DHCP server doesn't support different IP subnets on the same IP interface, which constrains the way things could work with zones. This is covered by 5010613. * The DHCP server populates the ARP table so that it can unicast DHCP responses to the client before the client has agreed to use the IP address (this is basically due to the same DHCP RFC braindamage I described in PSARC/2007/571). If it can't install an ARP entry, it will broadcast the response instead, which will work fine. So it may be sufficient to simply lower the log level of those failures. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] dhcp/zone
> Contrary to what everyone else says, with a bit of customization dhcpd > can run zones just fine. Solaris 10 update 3 and later (I forget > which Nevada build) have all the required bits in place. > > http://mail.opensolaris.org/pipermail/install-discuss/2007-March/004266.html Yep, it can be made to work, though I don't think it's officially supported. When I last looked at this issue, the changes needed to the source to make it work without any special tweaks was straightforward. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Subnet in a local zone
> I just installed a local zone with a different IP subnet and different > netmask than its global zone. There is no default route shown with > netstat -r. In a local zone which is in the same subnet / netmask as > its globsl zone, there is a default route. > > So, the question is, can this be done or must a LZ be in the same > subnet as the GZ? I think the Zones FAQ indirectly answers this: Q: How do I configure a default route in a container? A: All routes, including default routes, must be configured by the global zone administrator. Although a default route cannot be assigned directly to a Container, a default route can be assigned to each subnet. All zones on one subnet will then use the same default route. [July 2006] -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
re: [zones-discuss] Re: dladm create-vnic not available in build 63?
> Actually, my bad, the reason is that the vlan-type is non-vlan as shown > in the dladm output. Actually, the reason is that VNIC support is not yet in OpenSolaris; it's under development as part of Project Crossbow. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Re: [install-discuss] DHCP Server in zone, WAS: Install software from SXCE DVD?
> My statement "At the beginning..." meant "When the first part(s) of > Crossbow are in Solaris 10." At this point, IP Instances will probably > be that "first part" and that is what I was talking about. OK. From an engineering perspective, I consider the projects complementary but independent. But maybe the I-Teams for those projects will disagree with me ;-) -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Re: [install-discuss] DHCP Server in zone, WAS: Install software from SXCE DVD?
> > It might be nice to have a full list of the (otherwise Sun supported) > > NICs which don't work, if there isn't one already. I wasn't aware > > of this particular limitation. Do you know of such a list? > > At the beginning, Crossbow will only support GLDv3 NICs. Is there a > list of all NICs supported by Sun with Solaris 10? I don't follow. Crossbow is not currently in any release of S10. There are plans for Crossbow (once available) to make use of the softmac GLDv3 driver being provided by Clearview to allow any Solaris network driver to be used -- GLDv3 or otherwise. For a more definitive statement, please ask crossbow-discuss. If you're talking about IP Instances, the restriction is not tied specifically to GLDv3, but rather on the DLPI styles offered by the network driver, as Jim previously described. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Re: [install-discuss] DHCP Server in zone, WAS: Install software from SXCE DVD?
> - Currently must be tied to a physical NIC -- in other words > you must dedicate a real NIC (not a logical interface) > to each IP instance you want to run. Or you can dedicate a VLAN on that NIC. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Re: [install-discuss] DHCP Server in zone, WAS: Install software from SXCE DVD?
> >The IP Instances part of project crossbow deliver the feature to have a zone > >have its own view of the stack. It is available as a BFU on top of NV, but > >not yet integrated into NV. > > IP instances have integrated. (build 56 or something? Build 57 (and it's mostly separate from Crossbow). -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
re: [zones-discuss] zoneadm attach enhancement
> 6414178 RFE: prepare in advance for zone migration I'm a bit confused here. It seems like the implemented functionality serves the same purpose as "preparing in advance" for zone migration, but does not actually "prepare in advance". Should the synopsis be updated? Or am I missing something? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
re: [zones-discuss] Re: [crossbow-discuss] Design review of IP Instances part of Crossbow
> > Should these functions be dealing with "link names" rather than > > "interfaces" to avoid confusion with the interfaces that ifconfig(1M) > > deals with? > > Good point. > We've renamed them to have "datalink" in their names. > Jerry also pointed out that we should be more careful about terminology > like "ifname" and "interface name" in the code, so we have done that as > well. (Basically the general concept which corresponds to physical=bge0 > is something we call "network interface (name)". Those could be for > shared-, or exclusive-IP zones. But when it comes to adding a datalink > name for an exclusive IP zone we call the zone_add_datalink() syscall.) Glad to hear this. Consistent and correct use of these terms will be increasingly important given upcoming technologies like datalink vanity naming and VNICs (among others). -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] DHCP-/BOOTP-server in a local zone ?
> You need to interpose the zone_create function, add the privilege to > the privilege mask and call the original zone_create. Egad. The zone_create() function is not part of any documented or stable API; interpose on it at your peril. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Que: zones and ipmp
> > What happens when ce0 goes down ? Understood for running zone the i/p > > will failover to ce2 BUT if zone needs rebooting, do I now have to > > change physical to ce2 (ce0 is still down) ? If there any workaround ? > > I haven't tried this myself, but since the system knows that ce0 and > ce2 are in the IPMP group, everything will be done for your zones so > that the IP address is plumbed on an available interface in that IPMP > group. Don't try to configure your zone(s) with networks on more than > one interface in the group! > > It should be complete transparent to your zone(s) what the state of the > underlying IPMP group is as long as one interface is still > available. You might see the physical interface change (e.g. ce0:3 one > time and ce2:7 another), but your application should be insensitive to > that. The above is correct. BTW, after the Clearview IPMP Rearchitecture is done, addresses no longer move, and this sort of madness goes away. Early access bits to play with that do this are not too far off. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Zone installation problem.
> In general, we have the 4 standard values (/usr, /platform, > /lib & /sbin) for a sparse zone and no settings for a whole-root > zone. The one gray area is probably /opt. We were just > talking about this the other day and if we should impose > some restrictions on inherit-pkg-dir to make things a little > more sane. It might be helpful for us to hear from people > what inherit-pkg-dir settings they use, beyond the standard > sparse settings, and why they chose those. We are thinking > about the possibility of having a setting for a sparse vs. > whole-root zone instead of having to set inherit-pkg-dir, > but we also recognize that there might be some issues around /opt. FWIW, I think this is a wise move; inherit-pkg-dir is a lot of rope. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [Fwd: Reminder: Design review of IP Instances part of Crossbow]]
> This is an issue in S10 at two levels: > 1. The /etc/hostname. and related files are ignored when a zone > is booted. FWIW, as NWAM is currently specified, they will soon be ignored for global zone boots too (with appropriate transition mechanisms). -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
[zones-discuss] Re: [networking-discuss] Re: [crossbow-discuss] Design review of IP Instances part of Crossbow
> > > With regard to the third bullet, please see my concerns above about the > > > introduction of "list -l". I think this should be part of a general > > > zone status/health facility or perhaps something that dladm(1M) can > > > print about the link names and how their assignment zone-wise. > > > > Displaying the zone with dladm show-link seems a nice approach to me. > > Which project is planning on making GLDv3 zone aware? > IP Instances isn't. Dunno. My comment was about the overall user experience, not about which project does what. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
[zones-discuss] re: [networking-discuss] Re: [crossbow-discuss] Design review of IP Instances part of Crossbow
> With regard to the third bullet, please see my concerns above about the > introduction of "list -l". I think this should be part of a general > zone status/health facility or perhaps something that dladm(1M) can > print about the link names and how their assignment zone-wise. Displaying the zone with dladm show-link seems a nice approach to me. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] feedback requested: syslogging zone state transitions...
> I think that some minimal syslogging is warranted (I've tried to keep it > simple here) because we do know that customers have a degree of comfort > with syslog, and because it fits readily into various existing 3rd party > monitoring packages. But it falls apart in the long run since the output format cannot really be versioned or cleanly extended and the output is locale-specific. It also doesn't solve the synchronization problem you mention below. > As for GPEC, that's what our existing C api is based upon. Take a look at > zonecfg_notify_*() in libzonecfg. It's a real horror show but it does > solve the "get the state and then subscribe to future changes and don't > miss anything in between" problem. Hmm, I had to put together some APIs for IPMP that used sysevents (GPEC didn't exist at the time), and I actually thought the result was pretty elegant. Yes, it's more code than just dumping some messages to syslog, but it is easy to version and extend, and it is simple to write robust consumers for those events. Further, it is straightforward for those consumers to correlate those events with the state APIs. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] feedback requested: syslogging zone state transitions...
> Such as? No strong preference, but it should be something that can be versioned, extensible, parsed unambiguously and easily, and unaffected by locale. Maybe general purpose event channels? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
re: [zones-discuss] feedback requested: syslogging zone state transitions...
> So here are my questions: > > - Do you think this is useful? > > - Do you think the log level (Info) is right? daemon.info is > *not* logged by default, whereas notice is. (So basically: do > you want these messages in /var/adm/messages by default, or not?) > > - Do you think the facility of 'daemon' is OK? With Solaris > syslog you can't AFAIK route messages based on the value of > 'program' (which in this case is 'zoneadmd'). > > - Any comment about whether the info provided is sufficient? > For example, when a zone reboots it goes through numerous > state transitions, but I chose to express this as one > big transition-- does that work for everyone? Encouraging programmatic use of syslog seems a step in the wrong direction to me. Surely we can provide a better mechanism to notify them of state changes? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] zones and network interfaces
> > The fix for bug 6395576 adds the following failure mode: > > > > If two interfaces are in an IPMP group and there are no global zone > > addresses attached to the interfaces (link-based failure detection), > > then upon a link failure and failback, a zone IP address can migrate > > back to the physical interface (e.g. bge2) rather than back to the > > virtual interface where it started (e.g. bge2:1). When the zone is > > halted in this situation, the IP address stays active on bge2 until > > you unplumb it. This also prevents the zone from completely shutting > > down. > > Mike, that seems odd. I'll try to followup with David about this > at our next meeting to understand why this is the case. It makes sense to me -- there's nothing on the physical interface, so IPMP migrates the address back there rather than to a logical. The workaround would be to set a 0.0.0.0 -failover address on the physical interface. Address migration is a major wart in the IPMP design, and something that Clearview is in the process of disposing of. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] routeing in zones
> I will most likely be able to test it once it is there - it would be easier > for me to test against S10 than it would be against Nevada. However, > assuming these changes fall within usr/src/cmd-inet/usr.lib/in.mpathd and > not somewhere under usr/src/uts, it would probably be pretty > straight-forward for me to build and test it against S10. Indeed; the fix involves just 5 lines of changes to in.mpathd (though the function being changed was recently overhauled by my fix for 6397456 -- but I think it should still be straightforward). > As such, if it makes it into Nevada (or you send me diffs) in the next > couple of days, I should be able to test it this week. It would be great if we could verify this fixes your issues before going back to Nevada. > I also have a case open against this problem, and will be looking for a test > binary and patch through official support channels. Is there a CR open yet? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] routeing in zones
Mike, Sorry for the delay. I think you have exposed at least 3 bugs in the current implementation: 1. It's possible for in.mpathd to get confused and think that a replacement ipif is a test address. In your case, the replacement ipif was the 0.0.0.0 NOFAILOVER address that appeared on bge49000 after bge0's link was brought down, and removed from bge49000 when bge0's link was brought up. It is straightforward to enhance in.mpathd to ignore these. 2. In the case where a test address is considered a duplicate, in.mpathd currently provides very confusing output. In particular, it first states that probe-based failure detection has been disabled (but does not indicate on which interface), and then states that it has been enabled (because a test address is available). I should point out that (2) is exposed because of (1); once (1) is fixed, this bug will again be masked, except for the obscure case of someone setting IPv6 link-local addresses. However, even in the IPv6 case, I'm not sure that in.mpathd has to report a warning (I'm looking into it further). However, assuming we do continue to check for duplicate test addresses, the logic needs to be updated to (a) indicate the interface on which probe-based failure detection has been disabled, and (b) not subsequently contradict itself and indicate that probe-based failure detection has been enabled. 3. Even when a test address is flagged as a duplicate, in.mpathd still enables probe-based failure detection -- despite its claims to the contrary. This then leads to the case you hit, where an interface never repairs because probe-based failure detection is left enabled but cannot possibly succeed. Again, if (1) is fixed, this issue becomes masked again -- though it should be fixed. I have a fix for (1) available that I can putback to Nevada immediately. Please let me know if you'd like to test it. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] routeing in zones
> I find that when I do cable pulls (pull bge0, replace bge0, pull bge1, > replace bge1) that a zone IP address has been transitioned to a test > address. Yuck! A test address? It actually ends up with IFF_NOFAILOVER set? -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Non-global zone sending TCP SYN-ACK packet over wrong interface.
> Are you saying that if you have multiple interfaces on the same subnet, > they must be in an IPMP group, or the config is broken? Yes -- though the issue is really with having multiple interfaces on the same link, rather than the same subnet. > Or if you have multiple IP addresses (on the same subnet) on the same > interface, it is broken? The latter does not seem so, but want to make > sure Right, that is fine -- in fact, it is an extremely common configuration. > > With IPMP, outbound interface selection is determined in a round-robin > > fashion the first time an application attempts to send to a particular > > destination that does not yet have an IRE cache entry. > > See illgrp_scheduler() for details. > > For servers expecting connections to be established from a client, will > the IRE entry be set on the initial incoming datagram? Or the first > outgoing? Tobias' setup suggests that some responses go out one > interface, and others a different one. It will be set up when the server first needs to contact the client. In the case of a TCP connection, this will be done in the kernel as part of the three-way handshake in tcp_conn_request() -- so, for TCP, it is effectively a set on the initial incoming datagram (TCP SYN). > Is this due to the fact that there are, based my expectation of the > first part, multiple interfaces on the same subnet but not in an IPMP > group, therefore not associated correctly. Sorry, what is "this" in "is this due to the fact"? > I still don't understand the IRE behavior differences when IPMP is > enable vs. when not. Even after having tried to make sense of it > multiple times since IPMP was delivered. What differences are you referring to? The primary difference is that we round-robin across the available interfaces; this is necessary for outbound load-spreading. A consequence of this is that one cannot predict what interface in the group packets will be sent on. However, once an IRE cache entry to a particular destination has been created, the outgoing interface will remain constant until the IRE cache entry is removed. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Non-global zone sending TCP SYN-ACK packet over wrong interface.
> Yes, IPMP is IP Multipathing. Even if you have multiple addresses on > the same subnet on different interfaces, as you do, it does not > automatically enable IPMP. Though the resulting configuration is unsupported and broken. That is, if you have multiple IP interfaces on the same link, they *must* be configured into an IPMP group. If you don't do this, you will see duplicate broadcast and multicast traffic, among other problems. > And I still get confused by that selection Back with interface > groups (pre IPMP), outbound traffic on would leave on the same > interface a connection came in on, but I'm not sure that is still the > case. With IPMP, outbound interface selection is determined in a round-robin fashion the first time an application attempts to send to a particular destination that does not yet have an IRE cache entry. See illgrp_scheduler() for details. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
[zones-discuss] re: TCP/IP behavior in IPMP & Zone ?
> I need your expertise to confirm the following issue: > > Issue Description: The customer appl A sends a lookup request which is > destined to be handled by appl B. Both applications are using Cluster > Logical IP addressed destined for different subnets as configured using > multiple defaultrouters bound to separate IPMP groups. Their > expectation is that TCP communications between applications will cross > the physical network such that external firewall ACL rules can be > implemented/honored. > > During internal acceptance testing the customer failed both physical > links servicing the IPMP group of appl B. Packets from appl A still > arrived and were accepted by Appl B. This has highlighted that traffic > was crossing (round-robin) on the TCP stack/loopback and not making its > way onto the physical wire which negates enforcement of the external > firewall rules which expect a received packet /port to appear > consistently from the same host source interface. > > 1. So, Is the observation true and TCP/IP stack works as designed ? > > 2. Is there a way to tune or force it (TCP/IP stack)to cross the physical > network such that external firewall ACL rules can be implemented/honored ? Answers to both of these require more information. You ask about zones in the subject line, but talk about "Cluster Logical IP addresses" above. Is Sun Cluster being used? If so, you're probably asking the wrong aliases. > 3. This will bring up other questions when talking about S10 and > zones. If a packet > is source on zoneA and destined for zoneB will it go on the wire? No. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
