[
https://issues.apache.org/jira/browse/HADOOP-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Macdonald updated HADOOP-4:
---------------------------------
Attachment: fuse_dfs.c
Hi Pete,
Have you had a chance to look at FUSE readaheads? I have attached a version of
fuse_dfs.c I have patched, which reads 10MB chunks from DFS, and cache these in
the a struct held in the filehandle.
I'm seeing some improvement (down to 1m 20 compared to "bin/hadoop dfs -cat
file > /dev/null" which takes about 50 seconds). Increasing the buffer size
shows some improvement [I only did some quick tests] - I tried up to 30MB, but
I dont think there's much improvement over 5-10MB
Do you think we're reaching the limit such that the overheads of JNI are making
it impossible to go any faster? Ie Where do we go from here?
Another comment I have is that the configure/makefile asks for a dfs_home. It
might be easier to ask for Hadoop home, then build the appropriate paths from
there (${hadoop_home}/libhdfs and ${hadoop_home}/src/c++/libhdfs). Hadoop has
no include/linux folders etc. Finally, we need a way to detect whether to use
i386 or amd64 to find jvm.so
Craig
> tool to mount dfs on linux
> --------------------------
>
> Key: HADOOP-4
> URL: https://issues.apache.org/jira/browse/HADOOP-4
> Project: Hadoop Core
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.5.0
> Environment: linux only
> Reporter: John Xing
> Assignee: Pete Wyckoff
> Attachments: fuse-dfs.tar.gz, fuse-dfs.tar.gz, fuse-dfs.tar.gz,
> fuse-dfs.tar.gz, fuse-dfs.tar.gz,
> fuse-hadoop-0.1.0_fuse-j.2.2.3_hadoop.0.5.0.tar.gz,
> fuse-hadoop-0.1.0_fuse-j.2.4_hadoop.0.5.0.tar.gz, fuse-hadoop-0.1.1.tar.gz,
> fuse-j-hadoopfs-03.tar.gz, fuse_dfs.c, fuse_dfs.c, fuse_dfs.c, fuse_dfs.c,
> fuse_dfs.c, fuse_dfs.sh, Makefile
>
>
> tool to mount dfs on linux
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
