Thanks Dan,
I went ahead and changed the ClusterDumper class. It's not a pretty solution
since I just copied the ClusterDumper java code into my own java file and added
a new option
private boolean writeToHadoop;
addOption(WRITE_TO_HADOOP,"wh","Try write to HDFS","false");
if(hasOption(WRITE_TO_HADOOP)){
writeToHadoop = true;
}else{
writeToHadoop = false;
}
and then added an OR to the if
if (outputFile.getName().startsWith("s3n://") || writeToHadoop)
so now I can specify to write to hadoop with a '-wh' arg.
On 12 Mar 2013, at 12:37, Chris Harrington wrote:
> Hi all,
>
> The subject line says it all, ClusterDumper is writing to local file system
> instead of HDFS.
>
> After looking at the source
>
> From the ClusterDumper class
>
> if (this.outputFile == null) {
> shouldClose = false;
> writer = new OutputStreamWriter(System.out);
> } else {
> shouldClose = true;
> if (outputFile.getName().startsWith("s3n://")) {
> Path p = outputPath;
> FileSystem fs = FileSystem.get(p.toUri(), conf);
> writer = new OutputStreamWriter(fs.create(p), Charsets.UTF_8);
> } else {
> writer = Files.newWriter(this.outputFile, Charsets.UTF_8);
> }
> }
>
>
> From the Files class
>
> public static BufferedWriter newWriter(File file, Charset charset)
> throws FileNotFoundException {
> return new BufferedWriter(
> new OutputStreamWriter(new FileOutputStream(file), charset));
> }
>
>
> So a few questions on the above.
>
> 1. Am I correct in saying if the outputFile starts with "s3n://" it writes
> to the HDFS other wise it writes to the local FS?
>
> 2. If the above is true then what is the meaning of a URI starting with
> s3n://
>
> 3. Is there a way to force it to write to the HDFS even if the URI doesn't
> start with s3n:// or am I going to have to modify ClusterDumper class myself?
>
>
>
