Hi Matthew, please have a look at: http://www.spinics.net/lists/linux-rdma/msg16710.html http://wiki.ceph.com/01Planning/02Blueprints/Emperor/msgr%3A_implement_infiniband_support_via_rsockets
Maybe you should switch this discussion from ceph-user to the ceph-devel ML. Kind Regards, -Dieter On Fri, Aug 09, 2013 at 09:11:07AM +0200, Matthew Anderson wrote: > So I've had a chance to re-visit this since Bécholey Alexandre was kind > > enough to let me know how to compile Ceph with the RDMACM library > > (thankyou again!). > > > > At this stage it compiles and runs but there appears to be a problem with > > calling rshutdown in Pipe as it seems to just wait forever for the pipe to > > close which causes commands like 'ceph osd tree' to hang indefinitely > > after they work successfully. Debug MS is here - > > [1]http://pastebin.com/WzMJNKZY > > > > I also tried RADOS bench but it appears to be doing something similar. > > Debug MS is here - [2]http://pastebin.com/3aXbjzqS > > > > It seems like it's very close to working... I must be missing something > > small that's causing some grief. You can see the OSD coming up in the ceph > > monitor and the PG's all become active+clean. When shutting down the > > monitor I get the below which show's it waiting for the pipes to close - > > > > 2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing > > 2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping > > 2013-08-09 15:08:31.382115 7f464bd397c0 20 -- [3]172.16.0.1:6789/0 wait: > > stopped accepter thread > > 2013-08-09 15:08:31.382127 7f464bd397c0 20 -- [4]172.16.0.1:6789/0 wait: > > stopping reaper thread > > 2013-08-09 15:08:31.382146 7f4645500700 10 -- [5]172.16.0.1:6789/0 > > reaper_entry done > > 2013-08-09 15:08:31.382182 7f464bd397c0 20 -- [6]172.16.0.1:6789/0 wait: > > stopped reaper thread > > 2013-08-09 15:08:31.382194 7f464bd397c0 10 -- [7]172.16.0.1:6789/0 wait: > > closing pipes > > 2013-08-09 15:08:31.382200 7f464bd397c0 10 -- [8]172.16.0.1:6789/0 reaper > > 2013-08-09 15:08:31.382205 7f464bd397c0 10 -- [9]172.16.0.1:6789/0 reaper > > done > > 2013-08-09 15:08:31.382210 7f464bd397c0 10 -- [10]172.16.0.1:6789/0 wait: > > waiting for pipes 0x3014c80,0x3015180,0x3015400 to close > > > > The git repo has been updated if anyone has a few spare minutes to take a > > look - [11]https://github.com/funkBuild/ceph-rsockets > > > > Thanks again > > -Matt > > > > On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson > > <[12][email protected]> wrote: > > > > Hi All, > > I've had a few conversations on IRC about getting RDMA support into Ceph > > and thought I would give it a quick attempt to hopefully spur some > > interest. What I would like to accomplish is an RSockets only > > implementation so I'm able to use Ceph, RBD and QEMU at full speed over > > an Infiniband fabric. > > What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by > > replacing the regular socket calls with the rsocket equivalent. > > Unfortunately it doesn't compile and I get an error of - > > CXXLD ceph-osd > > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > > `Accepter::stop()': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243: > > undefined reference to `rshutdown' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251: > > undefined reference to `rclose' > > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > > `Accepter::entry()': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213: > > undefined reference to `raccept' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230: > > undefined reference to `rclose' > > ./.libs/libglobal.a(libcommon_la-Accepter.o): In function > > `Accepter::bind(entity_addr_t const&, int, int)': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61: > > undefined reference to `rsocket' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80: > > undefined reference to `rsetsockopt' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: > > undefined reference to `rbind' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118: > > undefined reference to `rgetsockname' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128: > > undefined reference to `rlisten' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100: > > undefined reference to `rbind' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: > > undefined reference to `rbind' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > > `Pipe::tcp_write(char const*, int)': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined > > reference to `rsend' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined > > reference to `rshutdown' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > > `Pipe::do_sendmsg(msghdr*, int, bool)': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined > > reference to `rsendmsg' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > > `Pipe::tcp_read_nonblocking(char*, int)': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined > > reference to `rrecv' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function > > `Pipe::tcp_read(char*, int)': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined > > reference to `rshutdown' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined > > reference to `rclose' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined > > reference to `rsocket' > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined > > reference to `rconnect' > > ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()': > > /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined > > reference to `rwrite' > > collect2: error: ld returned 1 exit status > > make[3]: *** [ceph-mon] Error 1 > > From the looks of it I need to include the 'rdma/rsocket.h' library > > somewhere else or add librdmacm but I'm not sure where. > > Full disclaimer, I am terrible at C++. If anyone has a few spare minutes > > to have a look into the error messages and can point out where I've gone > > wrong it would be greatly appreciated. > > I've put the code up at - [13]https://github.com/funkBuild/ceph-rsockets > > Thanks again > > -Matt > > > References > > Visible links > 1. http://pastebin.com/WzMJNKZY > 2. http://pastebin.com/3aXbjzqS > 3. http://172.16.0.1:6789/0 > 4. http://172.16.0.1:6789/0 > 5. http://172.16.0.1:6789/0 > 6. http://172.16.0.1:6789/0 > 7. http://172.16.0.1:6789/0 > 8. http://172.16.0.1:6789/0 > 9. http://172.16.0.1:6789/0 > 10. http://172.16.0.1:6789/0 > 11. https://github.com/funkBuild/ceph-rsockets > 12. mailto:[email protected] > 13. https://github.com/funkBuild/ceph-rsockets > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
