[
https://issues.apache.org/jira/browse/HADOOP-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095429#comment-18095429
]
Aaron Fabbri commented on HADOOP-19926:
---------------------------------------
Thank you for the extra context [~ivanandika] . I've implemented
zero-copy/kernel-bypass/RDMA for a couple of projects including a distributed
filesystem. My suggestion would be to start with instrumentation: We shouldn't
start optimizing network performance until we have profiles which show how much
time is spent there. RDMA APIs and hardware are specialized, and the cost of
that specialization is not worth it unless it has potential for significant
speedups. Having instrumentation in place also helps with the development
process of the updated network stack in addition to helping find other
performance issues that are easier to tackle. If someone can share some open
telemetry / chrome trace format data showing time spent sending and waiting on
receives that would be a good motivator to invest further with a prototype
(assuming someone has time to work on it).
If it looks worthwhile, the next decision is which interface to abstract the
different connection types at. E.g. the at the socket API layer, or RPC API
layer, etc.. Once you do that it is straightforward to prototype multiple
implementations that drive the different lower-level networking APIs.
> Use RDMA in Hadoop project
> --------------------------
>
> Key: HADOOP-19926
> URL: https://issues.apache.org/jira/browse/HADOOP-19926
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Raised this to start a discussion with the Hadoop community.
> While researching RDMA, I encountered the HIBD project
> ([https://hibd.cse.ohio-state.edu/]) that uses RDMA on Hadoop. Technically,
> we can decompile the Hadoop JAR using [https://github.com/java-decompiler]
> and extract the RDMA usage. However, the concern is whether this is compliant
> to ASF policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
