[
https://issues.apache.org/jira/browse/HADOOP-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094500#comment-18094500
]
Ivan Andika edited comment on HADOOP-19926 at 7/8/26 5:44 AM:
--------------------------------------------------------------
Thanks [~fabbri] for your input.
> Interesting project. Is there a demand for this feature?
Currently there was some talks to introduce kernel bypass (DPDK, SPDK, RDMA,
IO_URING) to Apache Ozone which uses both Hadoop RPC and gRPC to improve the
latency, but no concrete plan and commitment yet. Since gRPC requires Netty and
it seems that Netty decides to go for IO_URING, it might be hard to support
RDMA on Netty (although I tried to use AI to protoype an RDMA Netty transport
([https://github.com/ivandika3/netty/pull/1]) to see how it looks like).
Therefore, since Hadoop RPC and network related protocol is contained in the
Hadoop project, it might be easier to support RDMA on Hadoop project without
involving other communities.
I am raising this to start a discussion about the RDMA since I personally am
not familiar with RDMA development. RDMA is currently becomes more used in AI
related storage (i.e. to support GPUDirectStorage RDMA (GDR). For example, 3FS
already supports RDMA and a patch tried to support GDR in
[https://github.com/deepseek-ai/3FS/pull/407] . Other systems like SeaweedFS
([https://github.com/seaweedfs/seaweedfs/pull/7140)] tried to suppor it,
PolarFS also support this. Additionally an old blog
[https://blog.glennklockwood.com/2014/05/hadoops-uncomfortable-fit-in-hpc.html]
was talking about RDMA as a gap in Hadoop ecosystem implementation.
> Also I don't see any need to get MPI involved: Just starting with zero-copy
> kernel bypass support in place of regular network sockets would be a likely
> first step. I assume they have a JNI shim that talks to libibverbs / librdma
> on linux.
Yes, MPI should be out-of-scope. We can focus on writing a JNI shim under
libhadoop to support rdma.
> This is probably better suited to a thread on
> [[email protected].|mailto:[email protected].]
Yes, this is a good idea, but currently I have no concrete plans yet so I might
start a thread when I have a high-level roadmap for this and have had more
experience in RDMA related ecosystem and development.
was (Author: JIRAUSER298977):
Thanks [~fabbri] for your input.
> Interesting project. Is there a demand for this feature?
Currently there was some talks to introduce kernel bypass (DPDK, SPDK, RDMA,
IO_URING) to Apache Ozone which uses both Hadoop RPC and gRPC to improve the
latency, but no concrete plan and commitment yet. Since gRPC requires Netty and
it seems that Netty decides to go for IO_URING approach although I tried to use
AI to protoype an RDMA Netty transport
([https://github.com/ivandika3/netty/pull/1]) to see how it looks like. So
since Hadoop RPC and the HDFS socket is owned by the Hadoop project, it might
be easier to support RDMA on Hadoop project.
I am raising this to start a discussion about the RDMA since I personally am
not familiar with RDMA development. RDMA is currently becomes more used in AI
related storage (i.e. to support GPUDirectStorage RDMA (GDR). For example, 3FS
already supports RDMA and a patch tried to support GDR in
[https://github.com/deepseek-ai/3FS/pull/407] . Other systems like SeaweedFS
([https://github.com/seaweedfs/seaweedfs/pull/7140)] tried to suppor it,
PolarFS also support this. Additionally an old blog
[https://blog.glennklockwood.com/2014/05/hadoops-uncomfortable-fit-in-hpc.html]
was talking about RDMA as a gap in Hadoop ecosystem implementation.
> Also I don't see any need to get MPI involved: Just starting with zero-copy
> kernel bypass support in place of regular network sockets would be a likely
> first step. I assume they have a JNI shim that talks to libibverbs / librdma
> on linux.
Yes, MPI should be out-of-scope. We can focus on writing a JNI shim under
libhadoop to support rdma.
> This is probably better suited to a thread on
> [[email protected].|mailto:[email protected].]
Yes, this is a good idea, but currently I have no concrete plans yet so I might
start a thread when I have a high-level roadmap for this and have had more
experience in RDMA related ecosystem and development.
> Use RDMA in Hadoop project
> --------------------------
>
> Key: HADOOP-19926
> URL: https://issues.apache.org/jira/browse/HADOOP-19926
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Raised this to start a discussion with the Hadoop community.
> While researching RDMA, I encountered the HIBD project
> ([https://hibd.cse.ohio-state.edu/]) that uses RDMA on Hadoop. Technically,
> we can decompile the Hadoop JAR using [https://github.com/java-decompiler]
> and extract the RDMA usage. However, the concern is whether this is compliant
> to ASF policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
