It isn't popular much anymore, but once upon a time, network topology for clustering was a big topic. Since then, switches have gotten pretty fast and worrying about these things has gone out of fashion a bit other than something on the level of the current rack-aware locality in Hadoop.
With 4 NIC's, you could replay some history, however, by building what amounts to a 4-dimensional hyper-torus. I *think* you could pretty well by having four parallel two-level switch networks and assign boxes to second level switches according to a systematic pattern so that you would have local access to much of the cluster. As a simple example, suppose that you have 16 machines M1 through M16, each with two interfaces. You would have two top level switches T1 and T2 and each of those would have four second level switches S1.1 ... S1.4 connected to T1 and S2.1 ... S2.4 connected to T2. The machine connectivity on each interface would look like this: Machine eth0 eth1 1 S1.1 S2.1 2 S1.1 S2.2 3 S1.1 S2.3 4 S1.1 S2.4 5 S1.2 S2.1 6 S1.2 S2.2 7 S1.2 S2.3 8 S1.2 S2.4 9 S1.3 S2.1 10 S1.3 S2.2 11 S1.3 S2.3 12 S1.3 S2.4 13 S1.4 S2.1 14 S1.4 S2.2 15 S1.4 S2.3 16 S1.4 S2.4 With this arrangement machine M1 is on a local switch with M2, M3, M4, M5, M9, and M13 which is twice as many machines as would be local if only one interface were used. With four interfaces and four machines on a local switch, the entire cluster is local and T1 and T2 should see no traffic. In a larger cluster, you get the same 16x benefit in locality. The cost is that your ops guys will kill you if you suggest something this elaborate. The wiring between racks alone will make this a nightmare. On 2/12/08 3:01 PM, "Jason Venner" <[EMAIL PROTECTED]> wrote: > In terms of more exotic situations we were discussing having 4 NIC's 1 > for the local subset, 1 for a pair of local subsets, 1 for another pair > of local subsets, 1 for the backbone.
