The jar file will be uploaded into HDFS which could be shared.
From: xeonmailinglist <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, February 24, 2015 at 11:23 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: "No FileSystem for scheme: hdfs" when submitting a remote job Yes, this was the problem. But I have another question: When I am starting a remote job, the job will run in the remote resource manager. But how the job that is running remotely will know where are the map and reduce functions? On 24-02-2015 19:03, Xuan Gong wrote: Have you added hadoop-hdfs-xxxx.jar into your class path ? From: xeonmailinglist <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, February 24, 2015 at 10:57 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: "No FileSystem for scheme: hdfs" when submitting a remote job Hi, I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening? String host = "192.168.56.100"; Configuration conf = job.getConfiguration(); // this should be like defined in your yarn-site.xml conf.set("yarn.resourcemanager.address", host + ":50001"); // framework is now "yarn", should be defined like this in mapred-site.xm conf.set("mapreduce.framework.name", "yarn"); // like defined in hdfs-site.xml conf.set("fs.defaultFS", "hdfs://" + host + ":9000"); for (Path inputPath : inputPaths) FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString())); FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath)); job.waitForCompletion(true); Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518) at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169) at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101) Thanks,
