Hi Luke,
I actually tested hadoop RPC using SDP. But it did not work for me.
I used hadoop-core 0.20.2
Following is the error I received:
java.io.IOException: Address family not supported by protocol
at sun.net.sdp.SdpSupport.
convert0(Native Method)
at sun.net.sdp.SdpSupport.convertSocket(SdpSupport.java:70)
at sun.net.sdp.SdpProvider.convertTcpToSdpIfMatch(SdpProvider.java:300)
at sun.net.sdp.SdpProvider.implBeforeTcpBind(SdpProvider.java:323)
at sun.net.NetHooks.beforeTcpBind(NetHooks.java:86)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:198)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
My sdp.conf file is like:
bind * *
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
I also tried this for one specific node:
bind 10.80.70.13 50031
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
Thank you very much.
On Thu, Oct 10, 2013 at 4:00 AM, Luke Lu <l...@apache.org> wrote:
> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
> which _should_ work with default hadoop config, where dfs packet size is
> 64KB.
>
> __Luke
>
>
> On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <yasinceli...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I looked around but I think it seems there is no Sockets Direct
> > Protocol(SDP) support in Hadoop RPC.
> > I basically want to take advantage of Infiniband when transferring data.
> > Is there any known support for this? Has anybody tried this before?
> >
> > Best
> >
> > ---------
> > Yasin Celik
> >
>
--
---------
Yasin Celik
