My initial questions about bandwidth in a cluster: I want to understand the bandwidth when using TofuD in A64FX or Infiniband HDR. With HDR everything is clear - 25 GB / s for 4 links or 75 GB / s for 12 links. But PCIe-v3 x16 in A64FX will keep bandwidth.
Everything is more cunning with TofuD. Each link has 2 lanes, 28.05 Gbps x 2 gives about 7 GB / s (more really 6.8 GB / s). 6 TNI or 6 links give a total of 40.8 GB / s for injection - that's more than 25 GB / s in 4x HDR. This is for 2.2 GHz, I understand that at 1.8 GHz all numbers will decrease accordingly. If I calculate above correctly, then: Can I get close to 40.8 GB / s in a simple MPI PUT to another node - or will the limit be 6.8 GB / s? Then HDR will give more bandwidth (in Ookami w/Infiniband: 19.4 GB / s as maximum for OSU MPI). Ookami works w/Infiniband, not TofuD because of such bandwidth for a not very large cluster - or because of financial reasons (cost of TofuD routers?) ? Mikhail Kuzminsky _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
