Rayson, Is VFIO stabilized and in the kernel yet? I saw Jeff's demo at SC11 and he said that VFIO was not yet ready for primetime. If he gave an estimated timeframe, I do not remember.
Also, keep in mind, that it allows direct access to Ethernet frames. If you want any sort of reliability (timeout and retransmission) and I am pretty sure PVFS2 expects it, the BMI VFIO driver will need to implement it. I do not think that Jeff implemented any for his demo Open-MPI BTL. Like you, I think it is an interesting idea. It is an alternative approach to bypassing the TCP/IP compared to Open-MX which still sends through the kernel albeit directly to the Ethernet driver. Scott On Jan 13, 2012, at 4:22 PM, Rayson Ho wrote: > Hi Walter, > > Thanks for looking into it. > > My knowledge on PVFS is limited (I've read about it, and I enjoyed > your RCE cast). I am going to try PVFS for a new project, but I know > nothing about VFIO & PVFS internals! > > BTW, I looked into the code, and the BMI for IB or TCP is less than > 5000 lines of code (I did "cat *.c | wc" - I know the actual number > of lines of code is way less). Does it make sense to start a Google > Summer of Code project for it?? > > Rayson > > ================================= > Open Grid Scheduler / Grid Engine > http://gridscheduler.sourceforge.net/ > > Scalable Grid Engine Support Program > http://www.scalablelogic.com/ > > > On Fri, Jan 13, 2012 at 10:45 AM, Walter Ligon <[email protected]> wrote: >> I'm not aware of anyone doing that, but it sure sounds like something we >> would want to do! I'm putting it on the list. If someone wants to do it, >> by all means get in touch. >> >> Thanks, >> >> Walt >> >> >> On 1/12/12 2:43 PM, Rayson Ho wrote: >>> >>> We have some discussions on another HPC clustering mailing list on >>> InfiniBand vs 10GbE, and with VFIO even traditional Ethernet NICs can >>> perform OS-bypass. I was wondering if any PVFS2 developers are working >>> on adding VFIO support? >>> >>> http://blogs.cisco.com/performance/open-mpi-over-linux-vfio/ >>> http://lwn.net/Articles/474088/ >>> >>> Thanks! >>> >>> Rayson >>> >>> ================================= >>> Open Grid Scheduler / Grid Engine >>> http://gridscheduler.sourceforge.net/ >>> >>> Scalable Grid Engine Support Program >>> http://www.scalablelogic.com/ >>> _______________________________________________ >>> Pvfs2-developers mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers >> >> _______________________________________________ >> Pvfs2-developers mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > > > > -- > Rayson > > ================================================== > Open Grid Scheduler - The Official Open Source Grid Engine > http://gridscheduler.sourceforge.net/ > > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
