On Mon, Apr 13, 2009 at 1:07 AM, Todd Lipcon <[email protected]> wrote:
> Hey Brian,
>
> This is really interesting stuff. I'm curious - have you tried these same
> experiments using the Java API? I'm wondering whether this is FUSE-specific
> or inherent to all HDFS reads. I'll try to reproduce this over here as well.
>
I just tried this on a localhost single-node cluster with the following test
program:
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.conf.Configuration;
import java.io.IOException;
import java.net.URI;
public class Test {
public static void main(String[] args) throws Exception {
URI uri = new URI("hdfs://localhost:8020/");
FileSystem fs = FileSystem.get(uri, new Configuration());
Path path = new Path("/testfile");
FSDataInputStream dis = fs.open(path);
for (int size=0; size < 1024*1024; size += 4096) {
for (int i = 0; i < 100; i++) {
long st = System.currentTimeMillis();
byte buf[] = new byte[size];
dis.read(0, buf, 0, size);
long et = System.currentTimeMillis();
System.out.println(String.valueOf(size) + "\t" + String.valueOf(et -
st));
}
}
fs.close();
}
}
I didn't see the same behavior as you're reporting. Can you give this a try
on your cluster and see if it shows the 128K jump?
-Todd
