I'm trying to use a DistributedCache in Hadoop 0.17, but am having problems.
I add a file to the distributed cache with the following code:
DistributedCache.addCacheFile(new
URI("hdfs://ngcwydev102:50023/user/tgibbs/myDir/myOtherDir/part-00000"),
conf);
where conf is the JobConf I've created.
Now, in configure(), I do the following to read the file:
localFiles = DistributedCache.getLocalCacheFiles(conf);
Path path = localFiles[0];
FileSystem fs = path.getFileSystem(conf);
BufferedReader bdr = new BufferedReader(new InputStreamReader(new
FSDataInputStream(fs.open(path))));
This fails and tells me that it can't open:
hdfs://ngcwydev102:50023/user/tgibbs/myDir/myOtherDir/part-00000/part-00000
I've tried leaving off the part-00000 but that didn't work, it always
seems to replace the last part of my file.
Any ideas?
Thanks,
Tanton
