Re: [zones-discuss] networking
Hi, The clprivnet0 interface is something provided by Solaris Cluster. The clprivnet software is a kind of highly available trunking driver that communicates across all of the private networks that connect the machines of the cluster. This set of networks is often called the private interconnect. When there are Zone Clusters, the software automatically sets a subnet and IP addresses for each Zone Cluster. I believe that the cluster software does so as well for the Global Cluster. The administrator specifies a set of subnets and IP addresses for this purpose when configuring the cluster. If you have further questions related to clprivnet, I would suggest sending those questions to sunclus...@sun.com Regards, Ellard On 02/16/10 14:19, Enda O'Connor wrote: Hi Are you sure cluster is disabled, what does /usr/cluster/bin/status show? Enda On 16/02/2010 21:59, Dombrowski, Neil wrote: -Original Message- From: sowmini.varad...@sun.com [mailto:sowmini.varad...@sun.com] Sent: Tuesday, February 16, 2010 1:16 PM To: Dombrowski, Neil Cc: zones-discuss@opensolaris.org Subject: Re: [zones-discuss] networking On (02/16/10 19:03), Dombrowski, Neil wrote: I'm new to zones, and this appears to be a conundrum for me: I have a global zone that shows multiple default routes (on different interfaces). It also shows a third separate interface (clprivnet0) with an IP that's not in anyone's documentation(actually there are two physical servers set up the same way). My guess is that these two servers were to be clustered at one point, but this was aborted before I came onboard. Regardless, the global zone's routing table looks busy, is it because it's showing the routes for the zones? If so, is it possible to have the global zone routing differently than the local zones? hard to answer, without more data on what the subnets for the various zones are, and what the desired routing is. The global zone's netstat may show routes that are only accessible from a non-global zone, so the fact that the routing table is busy does not say anything without more information about the subnet configuration. --Sowmini For an example, let's say zone1 has a default route using gateway 172.16.1.1 and zone2 has a default router using gateway 192.168.0.1. If I am logged into the global zone, and it needs to send a packet to 10.10.10.10, will it use one of the non-global-zone's default route? Looking at /etc/defaultrouter for the global zone, it shows the gateway IPs for the two non-global zones, and also 10.10.10.1 . when I try to traceroute to 10.10.10.10 it never shows a single hop (as if it's not going to any gateway). So, why am I not getting to 10.10.10.10? And if I removed the other default routes in the global zone, will I be damaging the routing for the local zones? If I add a static route in the global zone will that be propagated to the non-local zones(I wouldn't want that)? If there's a good doc out there that explains this, I'd appreciate a pointer to it, or whatever advice you have for me. Thanks, Neil ___ zones-discuss mailing list zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
-Original Message- From: steffen.weibe...@sun.com [mailto:steffen.weibe...@sun.com] Sent: Wednesday, February 17, 2010 12:02 PM To: zones-discuss@opensolaris.org Cc: Dombrowski, Neil Subject: Re: [zones-discuss] networking On 02/16/10 17:17, Christine Tran wrote: On Tue, Feb 16, 2010 at 4:59 PM, Dombrowski, Neil neil.dombrow...@hp.com wrote: For an example, let's say zone1 has a default route using gateway 172.16.1.1 and zone2 has a default router using gateway 192.168.0.1. If I am logged into the global zone, and it needs to send a packet to 10.10.10.10, will it use one of the non-global-zone's default route? It will round-robin between the two gateways IF it has interfaces local to that network. That is, you need something like this: assume 24-bit mask, e1000g0 172.16.1.10 and e1000g1 192.168.0.10 (the 10 is just an example.) If you only have one interface local to one gateway, it will use that gateway. What I'm guessing is that you have your zones plumbed on a virtual interface, but nothing plumbed on the actual interface, from the global zone's perspective. In your ifconfig -a output, when you've removed all the entries for zones, do you actually have an interface that can reach a router? CT ___ zones-discuss mailing list zones-discuss@opensolaris.org To elaborate a little, if your global zone has an IP address on net0, and the other zones have IP address(es) on net1, net2, and net3, the only default route(s) the global will use are those related to net0. If a zone also has an IP address on net0, and it is on a different subnet than that/those used by the global zone, the global will still only use those related to it, not those added for the non-global zone. I had tested this a while back and had a discussion with an engineer around that. The result was that while I generally suggest the non-global zones use different IP subnet(s) and different interfaces than the global zone, the minimum requirement is that the zones use different IP subnet(s), and default routing will be fine. I believe in your case each zone will only use it's default route. You can verify this easily with the 'route get 10.10.10.10' command. It will list which interface is being used. Whether it is wise to have 172.16.1.0/24 and 192.168.0.0/24 on the same interface is a separate question. There is not enough information to make a guess at how your system is actually configured, and whether all the zones are sharing a single interface. Its also not clear which build or update of OpenSolaris or Solaris is being used. My recent testing was with Solaris 10 10/09 and a recent Nevada build (IIRC). The above should apply to any update with at least with 'defrouter' zone configuration option (8/07 I believe). Steffen Thanks to everyone's responses; I'm learning quite a bit here! My next question (which I think may have been partially answered already); it's obvious now that the global zone inherits the ngzones (non-global zones) routing information; is that a two-way street? If zone1 has a default route using 10.10.10.1 as it's gateway, and in the global zone I use a different router on the same network (10.10.10.5) as my default gateway, will that affect/interrupt zone1's routing table? I'll be experimenting a bit with this on my opensolaris box; hopefully it will match what solaris will do on our sparc servers. Neil ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
Dombrowski, Neil wrote: My next question (which I think may have been partially answered already); it's obvious now that the global zone inherits the ngzones (non-global zones) routing information; is that a two-way street? If zone1 has a default route using 10.10.10.1 as it's gateway, and in the global zone I use a different router on the same network (10.10.10.5) as my default gateway, will that affect/interrupt zone1's routing table? I'll be experimenting a bit with this on my opensolaris box; hopefully it will match what solaris will do on our sparc servers. A shared-stack zone cannot modify the kernel's forwarding table. It inherits -- read-only -- the forwarding table that is established by the global zone. Actually, there's no real inheritance going on here. There's just one forwarding table. The non-global zone is permitted to view it, and all of its packets are delivered according to it, but only the global zone can modify it. The only special thing going on with Solaris Zones is that when the non-global zone uses the table, it ignores any entries that it's not permitted to use -- where permitted is defined as for the output physical interface identified by the route, there exists at least one IP address [logical interface] configured and marked 'up' for that zone. If you establish two default routes in the global zone, then the system will treat them as equivalent. Packets from the global zone may be sent to either router without distinction. That might not be what you want. In general, if you want isolation, then you want the exclusive IP stack zone model. The shared stack model was designed for a BSD-Jails-like environment, where you're consolidating numerous servers that were previously configured side-by-side on a single network. Shared doesn't work as well when the zones are mutually hostile. -- James Carlson 42.703N 71.076W carls...@workingcode.com ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
On 02/17/10 14:38, James Carlson wrote: Dombrowski, Neil wrote: My next question (which I think may have been partially answered already); it's obvious now that the global zone inherits the ngzones (non-global zones) routing information; is that a two-way street? If zone1 has a default route using 10.10.10.1 as it's gateway, and in the global zone I use a different router on the same network (10.10.10.5) as my default gateway, will that affect/interrupt zone1's routing table? I'll be experimenting a bit with this on my opensolaris box; hopefully it will match what solaris will do on our sparc servers. A shared-stack zone cannot modify the kernel's forwarding table. It inherits -- read-only -- the forwarding table that is established by the global zone. Actually, there's no real inheritance going on here. There's just one forwarding table. The non-global zone is permitted to view it, and all of its packets are delivered according to it, but only the global zone can modify it. The only special thing going on with Solaris Zones is that when the non-global zone uses the table, it ignores any entries that it's not permitted to use -- where permitted is defined as for the output physical interface identified by the route, there exists at least one IP address [logical interface] configured and marked 'up' for that zone. If you establish two default routes in the global zone, then the system will treat them as equivalent. Packets from the global zone may be sent to either router without distinction. That might not be what you want. And I wrote up http://blogs.sun.com/stw/entry/what_happened_to_my_packets after coming up with this for a customer who was loosing connections because there were multiple default routes to different subnets, and connections would intermittently not work as they were using the 'wrong' default route. In general, if you want isolation, then you want the exclusive IP stack zone model. The shared stack model was designed for a BSD-Jails-like environment, where you're consolidating numerous servers that were previously configured side-by-side on a single network. Shared doesn't work as well when the zones are mutually hostile. Yup. If you run out of NIC on Solaris 10 (which does not have VNICs), you can use VLANs, if that works in your environment. http://blogs.sun.com/stw/entry/using_ip_instances_with_vlans Steffen ___ zones-discuss mailing list zones-discuss@opensolaris.org
[zones-discuss] networking
I'm new to zones, and this appears to be a conundrum for me: I have a global zone that shows multiple default routes (on different interfaces). It also shows a third separate interface (clprivnet0) with an IP that's not in anyone's documentation(actually there are two physical servers set up the same way). My guess is that these two servers were to be clustered at one point, but this was aborted before I came onboard. Regardless, the global zone's routing table looks busy, is it because it's showing the routes for the zones? If so, is it possible to have the global zone routing differently than the local zones? Thanks, Neil ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
On (02/16/10 19:03), Dombrowski, Neil wrote: I'm new to zones, and this appears to be a conundrum for me: I have a global zone that shows multiple default routes (on different interfaces). It also shows a third separate interface (clprivnet0) with an IP that's not in anyone's documentation(actually there are two physical servers set up the same way). My guess is that these two servers were to be clustered at one point, but this was aborted before I came onboard. Regardless, the global zone's routing table looks busy, is it because it's showing the routes for the zones? If so, is it possible to have the global zone routing differently than the local zones? hard to answer, without more data on what the subnets for the various zones are, and what the desired routing is. The global zone's netstat may show routes that are only accessible from a non-global zone, so the fact that the routing table is busy does not say anything without more information about the subnet configuration. --Sowmini ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
-Original Message- From: sowmini.varad...@sun.com [mailto:sowmini.varad...@sun.com] Sent: Tuesday, February 16, 2010 1:16 PM To: Dombrowski, Neil Cc: zones-discuss@opensolaris.org Subject: Re: [zones-discuss] networking On (02/16/10 19:03), Dombrowski, Neil wrote: I'm new to zones, and this appears to be a conundrum for me: I have a global zone that shows multiple default routes (on different interfaces). It also shows a third separate interface (clprivnet0) with an IP that's not in anyone's documentation(actually there are two physical servers set up the same way). My guess is that these two servers were to be clustered at one point, but this was aborted before I came onboard. Regardless, the global zone's routing table looks busy, is it because it's showing the routes for the zones? If so, is it possible to have the global zone routing differently than the local zones? hard to answer, without more data on what the subnets for the various zones are, and what the desired routing is. The global zone's netstat may show routes that are only accessible from a non-global zone, so the fact that the routing table is busy does not say anything without more information about the subnet configuration. --Sowmini For an example, let's say zone1 has a default route using gateway 172.16.1.1 and zone2 has a default router using gateway 192.168.0.1. If I am logged into the global zone, and it needs to send a packet to 10.10.10.10, will it use one of the non-global-zone's default route? Looking at /etc/defaultrouter for the global zone, it shows the gateway IPs for the two non-global zones, and also 10.10.10.1 . when I try to traceroute to 10.10.10.10 it never shows a single hop (as if it's not going to any gateway). So, why am I not getting to 10.10.10.10? And if I removed the other default routes in the global zone, will I be damaging the routing for the local zones? If I add a static route in the global zone will that be propagated to the non-local zones(I wouldn't want that)? If there's a good doc out there that explains this, I'd appreciate a pointer to it, or whatever advice you have for me. Thanks, Neil ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] networking
There is a /usr/cluster/bin/scstat command, which on each server shows only itself (the two servers don't seem to have any info here on the other server). I also see a bunch of /usr/cluster/lib/something processes running on each box. I'm making inquiries as to whether the cluster has ever worked, but so far it sounds like it was tabled for higher priority projects, and never gotten back to. Neil -Original Message- From: zones-discuss-boun...@opensolaris.org [mailto:zones-discuss-boun...@opensolaris.org] On Behalf Of Enda O'Connor Sent: Tuesday, February 16, 2010 4:20 PM To: zones-discuss@opensolaris.org Subject: Re: [zones-discuss] networking Hi Are you sure cluster is disabled, what does /usr/cluster/bin/status show? Enda On 16/02/2010 21:59, Dombrowski, Neil wrote: -Original Message- From: sowmini.varad...@sun.com [mailto:sowmini.varad...@sun.com] Sent: Tuesday, February 16, 2010 1:16 PM To: Dombrowski, Neil Cc: zones-discuss@opensolaris.org Subject: Re: [zones-discuss] networking On (02/16/10 19:03), Dombrowski, Neil wrote: I'm new to zones, and this appears to be a conundrum for me: I have a global zone that shows multiple default routes (on different interfaces). It also shows a third separate interface (clprivnet0) with an IP that's not in anyone's documentation(actually there are two physical servers set up the same way). My guess is that these two servers were to be clustered at one point, but this was aborted before I came onboard. Regardless, the global zone's routing table looks busy, is it because it's showing the routes for the zones? If so, is it possible to have the global zone routing differently than the local zones? hard to answer, without more data on what the subnets for the various zones are, and what the desired routing is. The global zone's netstat may show routes that are only accessible from a non-global zone, so the fact that the routing table is busy does not say anything without more information about the subnet configuration. --Sowmini For an example, let's say zone1 has a default route using gateway 172.16.1.1 and zone2 has a default router using gateway 192.168.0.1. If I am logged into the global zone, and it needs to send a packet to 10.10.10.10, will it use one of the non-global-zone's default route? Looking at /etc/defaultrouter for the global zone, it shows the gateway IPs for the two non-global zones, and also 10.10.10.1 . when I try to traceroute to 10.10.10.10 it never shows a single hop (as if it's not going to any gateway). So, why am I not getting to 10.10.10.10? And if I removed the other default routes in the global zone, will I be damaging the routing for the local zones? If I add a static route in the global zone will that be propagated to the non-local zones(I wouldn't want that)? If there's a good doc out there that explains this, I'd appreciate a pointer to it, or whatever advice you have for me. Thanks, Neil ___ zones-discuss mailing list zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
Darren Reed wrote: Guys, In most parts of the source code, the zoneid is unsigned, except for where we use ALL_ZONES. Then in some places, we assign or expect -1 to be the zoneid, for example in what psh prints and expects to see. It would seem that we want the zoneid to be unsigned except for when its value is -1. This seems... like it could lead to confusion or other bad things. It looks to me like it's neither signed nor unsigned, but rather an opaque type. It supports only equality tests, not general arithmetic ones, so signedness doesn't seem to come into it. What kind of confusion are you expecting? -- James Carlson 42.703N 71.076W carls...@workingcode.com ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
Darren Reed wrote: James Carlson wrote: What kind of confusion are you expecting? If it is an opaque type, then how does it get printed? You have to use one of the look-up functions to convert it to a string for printing. Zones are named, not numbered, even in the kernel. This was a key architectural decision made by (I think) Andy Tucker back when Kevlar was being designed. I wanted zoneids to be like UIDs, GIDs, and other UNIX IDs -- used as small integers everywhere, and converted to names when necessary by use of name services. Andy and the other kernel folks disagreed, and felt that strings were better, and integers would be allocated on the fly (non-permanently) and never used as zone identifiers, except in performance-sensitive (and entirely internal) contexts. As an unsigned integer for all values, except -1, or as a signed integer? I still think it's properly neither. Users can't reasonably do anything with those ephemeral numbers, so printing them (or using them in user interfaces) would be a mistake. -- James Carlson 42.703N 71.076W carls...@workingcode.com ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
Darren Reed wrote: On 18/09/09 10:44 AM, James Carlson wrote: Darren Reed wrote: As an unsigned integer for all values, except -1, or as a signed integer? I still think it's properly neither. Users can't reasonably do anything with those ephemeral numbers, so printing them (or using them in user interfaces) would be a mistake. Do a man snoop and search for the word zone. My argument wasn't that there were zero bugs in the OS. That keyword seems to me to be pretty clearly a defect in snoop. (And apparently a recent one; less than a year old.) It was that the zone IDs are inherently ephemeral, and were designed to be that way. They're not intended as any sort of administrative interface. The kernel assigns them arbitrarily as the zones are booting, and there's no guarantee at all that any particular number represents any zone over time. At best, they're useful in undocumented or volatile debug interfaces (such as in mdb). It's the zone names that are fixed and are intended as administrative interfaces. And that since you can't do any reasonable arithmetic comparisons between them (what does it mean for one zone ID to be less than another?), it doesn't really matter what a zoneid_t is inside. -- James Carlson 42.703N 71.076W carls...@workingcode.com ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
John Leser wrote: Darren Reed wrote: Do a man snoop and search for the word zone. Oh, that was a bit of a let-down... Anyway, this seems to pose an interesting challenge to programs like snoop that want to encode zone ID information in output files. The zone ID numbers are essentially meaningless in a disk file - who knows what zone names they corresponded to at the time the capture was done. To be at all correct, IPNET headers have to encode the string literal for the zone name. Ugly. Not necessarily ... it's ok if IPNET headers have zone IDs in them, as long as all the tools that use them translate to and from string format when interacting with the user or transporting them externally (as in over a network or via a file). (I think you're right, though, that ideally the interface between kernel and user space should deal with strings, because there's no guarantee that a given zone ID that you see in an IPNET header will necessarily correspond to the named zone you get moments later with getzonenamebyid() -- because the ID might have been discarded by the kernel between those two events.) Yes, it does mean that some file formats are going to be a little less pretty than others. Also, zone IDs are more ephemeral than UIDs or GIDs, because if I log out and back in, I keep the same UID. If I halt and boot the same zone, it gets a new zone ID each time. Exactly. It's a different design. As I said before, when I was working on the old Kevlar team, I wanted a UID-like experience for these, because I _assumed_ that people would want to administer zone names centrally across many machines, and because it seemed to me that integers would be easier to encode and work with. The rest of the team, though, assumed differently: that if there was any coordination between machines, it would be on the basis of the assigned zone name, and that strings were just as easy to use. -- James Carlson 42.703N 71.076W carls...@workingcode.com ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] is the zoneid signed or unsigned?
Do a man snoop and search for the word zone. My argument wasn't that there were zero bugs in the OS. That keyword seems to me to be pretty clearly a defect in snoop. (And apparently a recent one; less than a year old.) ... and indeed, there's a CR open to allow snoop to accept zone names. -- meem ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [networking-discuss] FW: using load balancers with zones]
On 11/10/08 13:56, Steve Lawrence wrote: ... I am sure there must be a way to tell ipf to force packets from the be- net to the fe-net to go out on the wire, presumably using dup-to but I was unable to make it work, so I am using source-nat at the moment. I look forward to hearing that someone else has already figured this out and here is how they did it, but for now I need to work on installing software on mail-fe-1 because it was supposed to be working last Tuesday ;) I think the best answer to your question is to download the latest snapshot of a crossbow build and use that. What you need is local (on the zones box) switching that recognises packets don't need to leave the host to find the destination. Darren ___ zones-discuss mailing list zones-discuss@opensolaris.org
