>>> Wondering if anyone out there is doing both I/O to storage as well as >>> MPI over the same IB fabric. >>> >> >> I would say that is the norm. we certainly connect local storage (Lustre) >> to nodes via the same fabric as MPI. gigabit is completely >> inadequate for modern nodes, so the only alternatives would be 10G >> or a secondary IB fabric, both quite expensive propositions, no? >> >> I suppose if your cluster does nothing but IO-light serial/EP jobs, >> you might think differently. >> > Really? I'm surprised by that statement. Perhaps I'm just way behind on the > curve though. It is typical here to have local node storage, local > lustre/pvfs storage, local NFS storage, and global GPFS storage running over > the GigE network.
sure, we use Gb as well, but only as a crutch, since it's so slow. or does each node have, say, a 4x bonded Gb for this traffic? or are we disagreeing on whether Gb is "slow"? 80-ish MB/s seems pretty slow to me, considering that's less than any single disk on the market... >> how much inter-chassis MPI do you do? how much IO do you do? >> IB has a small MTU, so I don't really see why mixed traffic would be a big >> problem. of course, IB also doesn't do all that wonderfully >> with hotspots. but isn't this mostly an empirical question you can >> answer by direct measurement? >> > How would I measure by direct measurement? I meant collecting the byte counters from nics and/or switches while real workloads are running. that tells you the actual data rates, and should show how close you are to creating hotspots. > My question really was twofold: 1) is anyone doing this successfully and 2) > does anyone have an idea of how loudly my users will scream when their MPI > jobs suddenly degrade. You've answered #1 and seem to believe that for #2, > no one will notice. we've always done it, though our main experience is with clusters that have full-bisection fabrics. our two more recent clusters have half-bisection fabrics, but I suspect that most users are not looking closely enough at performance to notice and/or complain. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
