Re: Best way to get data out of HDFS to other platforms

[Mac Noland]

There are probably much, much, much better and more standard ways, but in the 
simple push and pulls I've done, I've just rolled my own.

    public void copyFileFromHDFS(String src, String dest, boolean overWrite) 
throws IOException{

        Path hdfsPath = new Path(src);
        FSDataInputStream hdfsFileInPut = fs.open(hdfsPath);
        
        byte[] data = new byte[1024];
        File localFile = new File(dest);
        
        if (overWrite && localFile.exists()){
            localFile.delete();
        }
        
        FileOutputStream localFileOutPut = new FileOutputStream(localFile);
        int off;
        
        while ((off = hdfsFileInPut.read(data)) > -1){
            localFileOutPut.write(data,0,off);
        }
        
        hdfsFileInPut.close();
        localFileOutPut.close();

    }



________________________________
 From: Mapred Learn <mapred.le...@gmail.com>
To: hdfs-user@hadoop.apache.org; CDH Users <cdh-u...@cloudera.org> 
Sent: Friday, April 13, 2012 10:29 AM
Subject: Best way to get data out of HDFS to other platforms 
 

Hi All,

I was trying to find out the best options to get data out of HDFS to other 
platforms like AIX with parameters like reliability, performance and 
parallelism and data integrity in mind.


Thanks,
Sent from my iPhone