Re: Does anyone have a working example for using MapFiles on the DistributedCache?

[Sean Shanny]

Amareshwari

For part 3 I am trying to open the file to be able to use a  
MapFile.Reader.get() method to look up values based on an input key.

localFiles = DistributedCache.getLocalCacheFiles(conf);

            for (Path localFile : localFiles)
            {
                String sFileName = localFile.getName();

                if (sFileName.equalsIgnoreCase("data"))
                {
                    System.out.println("Full Path: " +  
localFile.toString());
                    System.out.println("Parent: " +  
localFile.getParent().toString());

                    fs = FileSystem.get(localFile.toUri(), conf);
                    myReader = new MapFile.Reader(fs,  
localFile.getParent().toString(), conf);


The code always fails at the myReader.... line with a file not found  
error.  I have triple checked that the file is on the system.  I am  
stumped on how to use a MapReader based on  a file in the  
DistributedCache.


Thanks.

--sean

Sean Shanny
ssha...@tripadvisor.com




On Dec 28, 2008, at 10:59 PM, Amareshwari Sriramadasu wrote:

Sean Shanny wrote:
To all,

Version:  hadoop-0.17.2.1-core.jar

I have created a MapFile.

What I don't seem to be able to do is correctly place the MapFile  
in the DistributedCache and the make use of it in a map method.

I need the following info please:

1.    How and where to place the MapFile directory so that it is  
visible to the hadoop job.
You have to place your files in DFS. If it is directory you can  
place an archive of it.
2.    How to add the files to the DistributedCache.
You can use DistributedCache.addCacheFile or  
DistributedCache.addCacheArchive.
See more documentation @ 
http://hadoop.apache.org/core/docs/r0.17.2/api/org/apache/hadoop/filecache/DistributedCache.html
and
http://hadoop.apache.org/core/docs/r0.17.2/mapred_tutorial.html#DistributedCache
3.    How to create a MapFile.Reader from files in the  
DistributedCache.

I didn't understand what you want to do here. Do you want see the  
files in directory MapFile? or do you want them in classpath etc?
You can use DistributedCache.addFileToClassPath or  
DistributedCache.addArchiveToClassPath

Hope this helps.

Thanks
Amareshwari
I can get this to work with a local file on a single node system  
outside of the DistributedCache but for the life of me cannot get  
it to work within a DistributedCache.

We are trying to load up key value mappings for a Data Warehouse  
ETL process.  The mapper will take an input record, lookup the keys  
based on values and emit the resulting key only record.

Happy to answer any questions to help me make this work.

Thanks.

--sean