On Fri, 21 Apr 2006, Bernard King-Smith wrote:
> Grant Grundler wrote: > > Grant> My guess is it's an easier problem to fix SDP than reducing TCP/IP > > Grant> cache/CPU foot print. I realize only a subset of apps can (or will > > Grant> try to) use SDP because of setup/config issues. I still believe > SDP > > Grant> is useful to a majority of apps without having to recompile them. > > > > I agree that reducing any protocol footprint is a very challenging job, > > however, going to a larger MTU drops the overhead much faster. If IB > > supported a 60K MTU then the TCP/IP overhead would be 1/30 that of what > we > > measure today. Traversng the TCP/IP stack once for a 60K packet is much > > lower than 30 times using 2000 byte packets for the same amount of data > > transmitted. > > Grant> I agree that's effective for workloads which send large messages. > Grant> And that's typical for storage workloads. > Grant> But the world is not just an NFS server. ;) > > However, NFS is not the only large data transfer workload I come > across. If IB wants to achieve high volumes there needs to be some > kind of commercial workload that works well on it besides the large > applications that can afford to port to SDP or uDAPL. HPC is a nitch > that is hard to sustain a viable business in ( though everyone can > point to a couple of companies, most are not long term, 15 years or > more ). Running NFS on IPoIB isn't the only option. As an alternative, NFS can be run directly on an RDMA network using the RPC-RDMA protocol. > However, in clustering the areas that benefit from large block transfer > efficiency include: > > File Serving ( NFS, GPFS, XFS etc.) > Application backup > Parallel databases > Database upload/update flows > Web server graphics > Web server MP3s > Web server streaming video > Local workstation backup > Collaboration software > Local mail replication > > My concern is that if IB does not support these operations as well as > Ethernet, then it is a hard sell into commercial accounts/workloads for IB. > > Bernie King-Smith > IBM Corporation > Server Group > Cluster System Performance > [EMAIL PROTECTED] (845)433-8483 > Tie. 293-8483 or wombat2 on NOTES > > "We are not responsible for the world we are born into, only for the world > we leave when we die. > So we have to accept what has gone before us and work to change the only > thing we can, > -- The Future." William Shatner > > > > Grant Grundler > <[EMAIL PROTECTED]> > > To > 04/21/2006 04:10 Bernard > PM King-Smith/Poughkeepsie/[EMAIL > PROTECTED] > cc > Grant Grundler <[EMAIL PROTECTED]>, > > [email protected], Roland > Dreier <[EMAIL PROTECTED]> > Subject > Re: Speeding up IPoIB. > > > > > > > > > > > On Thu, Apr 20, 2006 at 09:03:29PM -0400, Bernard King-Smith wrote: > > Grant> My guess is it's an easier problem to fix SDP than reducing TCP/IP > > Grant> cache/CPU foot print. I realize only a subset of apps can (or will > > Grant> try to) use SDP because of setup/config issues. I still believe > SDP > > Grant> is useful to a majority of apps without having to recompile them. > > > > I agree that reducing any protocol footprint is a very challenging job, > > however, going to a larger MTU drops the overhead much faster. If IB > > supported a 60K MTU then the TCP/IP overhead would be 1/30 that of what > we > > measure today. Traversng the TCP/IP stack once for a 60K packet is much > > lower than 30 times using 2000 byte packets for the same amount of data > > transmitted. > > I agree that's effective for workloads which send large messages. > And that's typical for storage workloads. > But the world is not just an NFS server. ;) > > > Grant> I'm not competent to disagree in detail. > > Grant> Fabian Tillier and Caitlin Bestler can (and have) addressed this. > > > > I would be very interested in any pointers to their work. > > They have posted to this forum recently on this topic. > The archives are here in case you want to look them up: > http://www.openib.org/contact.html > > > This goes back to systems where the system is busy doing nothing, > generally > > when waiting for memory or a cache line miss, or I/O to disks. This is > > where hyperthreading has shown some speedups for benchmarks where > > previously they were totally CPU limited, and with hyperthreading there > is > > a gain. > > While there are workloads that benefit, I don't buy the hyperthreading > argument in general. Co-workers have demonstrate several "normal" > workloads that don't benefit and are faster with hyperthreading > disabled. > > > The unused cycles are "wait" cycles when something can run if it > > can get in quickly. You can't get a TCP stack in the wait, but small > parts > > of the stackor driver could fit in the other thread. Yes I do > benchmarking > > and was skeptical at first. > > ok. > > thanks, > grant > > > _______________________________________________ > openib-general mailing list > [email protected] > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
