Hi Jun,
The earlier responses to your email reference the JIRA that I opened
about this issue. Short-circuiting the primary HDFS datapath does
improve throughput, and the amount depends on your workload (random
reads especially). Some initial experimental results are posted to that
JIRA. A second advantage is that since the JVM hosting the HDFS client
is doing the reading, the O/S will satisfy future disk requests from the
cache, which isn't really possible when you read over the network (even
to another JVM on the same host).
There are several real disadvantages, the largest of which include 1) it
adds a new datapath, and 2) bypasses various security and auditing
features of HDFS. I would certainly like to think through a more clean
interface for achieving this goal, especially since reading local data
should be the common case. Any thoughts you might have would be
appreciated.
Thanks,
George
Jun Rao wrote:
Hi,
Today, HDFS always reads through a socket even when the data is local to
the client. This adds a lot of overhead, especially for warm reads. It
should be possible for a dfs client to test if a block to be read is local
and if so, bypass socket and read through local FS api directly. This
should improve random access performance significantly (e.g., for HBase).
Has this been considered in HDFS? Thanks,
Jun
