Hi Xiaoyi, This is great !
For clusters that have moved to Hadoop 2.x versions, do you have any work in progress to rebase Hadoop-RDMA to 2.x ? Also, I am wondering if the development that you are doing will be available in Mellanox UDA ? Or is it an independent project ? - milind --- Milind Bhandarkar Chief Scientist Pivotal +1-650-523-3858 (W) +1-408-666-8483 (C) On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi [email protected] < [email protected]> wrote: > The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA > 0.9.8 > (technology preview) package with the following features and bug-fixes: > > * Hadoop-RDMA 0.9.8 Features > > - Based on Apache Hadoop 1.2.1 > - High performance design with native InfiniBand and RoCE support > at the verbs-level for HDFS, MapReduce, and RPC components > - Compliant with Apache Hadoop 1.2.1 APIs and applications > - Easily configurable for native InfiniBand, RoCE and the traditional > sockets-based support (Ethernet and InfiniBand with IPoIB) > - HDFS over native InfiniBand and RoCE > - RDMA-based write > - RDMA-based replication > - Tuned packet size > - MapReduce over native InfiniBand and RoCE > - RDMA-based shuffle > - Pre-fetching and caching of map outputs > - In-memory merge > - Overlapping of merge and reduce > - RPC over native InfiniBand and RoCE > - JVM-bypassed buffer management > - On-demand InfiniBand connection setup > - RDMA or send/recv based adaptive communication > - Tested with > - Mellanox InfiniBand adapters (DDR, QDR, and FDR) > - RoCE > - Various multi-core platforms > - Different file systems with disks and SSDs > > * Bug Fixes (since 0.9.1) > > - MapReduce over InfiniBand > - Reducing the waiting time for the pre-fetch cache read > - Enabling Reduce tasks to randomize the shuffle pattern > - HDFS over InfiniBand > - Fix an issue in processing multiple blocks concurrently in a > single task > > For downloading Hadoop-RDMA 0.9.8 package and the associated user guide, > please visit the following URL: > > http://hadoop-rdma.cse.ohio-state.edu > > Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8 > version can be viewed by visiting the `Performance' tab of the above > website. > > All questions, feedbacks and bug reports are welcome. Please post it > to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at > cse.ohio-state.edu). > > Thanks, > > The Hadoop-RDMA Team >
