[
https://issues.apache.org/jira/browse/HDFS-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-8765:
----------------------------------
Attachment: short_circuit_demo.patch
Adding a hacked together demo of the short circuit protocol. It won't actually
do short circuit reads; it just tries to ask for and receive file descriptors
over the domain socket. It doesn't work, I'm not sure what all is wrong at the
moment. Just realized that NDEBUG was defined making my asserts useless in
past testing.
Some notes:
-datanode seems to see the REQUEST_SHORT_CIRCUIT_FDS request over the domain
socket.
-if anyone knows the protocol well can can see what I'm doing wrong please
comment. Best places to look are FileHandleImpl::getLocalReader,
send_request_dfs in filehandle.cc, and the tcp/domain socket stuff in
libs/localreader.
-a couple paths for the domain socket are hard coded; grepping for
jclampffer/hadoop_scratch should find these.
-bind path validation minidfs cluster is commented out
-doing basic tests with the 'cat' executable in libhdfspp/lib/utils/cat
With the upcoming holidays + higher priority features in this project I don't
expect to be working on this jira much for the next month.
> Implement local block reader in libhdfspp
> -----------------------------------------
>
> Key: HDFS-8765
> URL: https://issues.apache.org/jira/browse/HDFS-8765
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Reporter: James Clampffer
> Assignee: James Clampffer
> Attachments: short_circuit_demo.patch
>
>
> Implement a block reader that uses the hdfs short circuit protocol to read
> colocated data as efficiently as possible. Implementation will be based on
> BlockReaderLocal.java + the associated JNI bindings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
