[
https://issues.apache.org/jira/browse/HADOOP-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094500#comment-18094500
]
Ivan Andika commented on HADOOP-19926:
--------------------------------------
Thanks [~fabbri] for your input.
> Interesting project. Is there a demand for this feature?
Currently there was some talks to introduce kernel bypass (DPDK, SPDK, RDMA,
IO_URING) to Apache Ozone which uses both Hadoop RPC and gRPC to improve the
latency. Since gRPC requires Netty and it seems that Netty decides to go for
IO_URING way although I tried to use AI to protoype an RDMA Netty transport
([https://github.com/ivandika3/netty/pull/1]) to see how it looks like. So
since Hadoop RPC and the HDFS socket is owned by the Hadoop project, it might
be easier to support RDMA on Hadoop project.
I am raising this to start a discussion about the RDMA since I personally am
not familiar with RDMA development. RDMA is currently becomes more used in AI
related storage (i.e. to support GPUDirectStorage RDMA (GDR). For example, 3FS
already supports RDMA and a patch tried to support GDR in
[https://github.com/deepseek-ai/3FS/pull/407] . Other systems like SeaweedFS
([https://github.com/seaweedfs/seaweedfs/pull/7140)] tried to suppor it,
PolarFS also support this. Additionally an old blog
[https://blog.glennklockwood.com/2014/05/hadoops-uncomfortable-fit-in-hpc.html]
was talking about RDMA as a gap in Hadoop ecosystem implementation.
> This is probably better suited to a thread on
> [[email protected].|mailto:[email protected].]
Yes, this is important, but currently I have no concrete plans yet so I might
start a thread when I have a high-level roadmap for this and have had some
experienced in RDMA related development.
> Use RDMA in Hadoop project
> --------------------------
>
> Key: HADOOP-19926
> URL: https://issues.apache.org/jira/browse/HADOOP-19926
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Raised this to start a discussion with the Hadoop community.
> While researching RDMA, I encountered the HIBD project
> ([https://hibd.cse.ohio-state.edu/]) that uses RDMA on Hadoop. Technically,
> we can decompile the Hadoop JAR using [https://github.com/java-decompiler]
> and extract the RDMA usage. However, the concern is whether this is compliant
> to ASF policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
