Hi,
I am getting following exception while trying to submit a crunch job to a
remote hadoop cluster:
2880 [Thread-15] INFO
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob -
java.lang.IllegalArgumentException: java.net.UnknownHostException: bdatadev
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:
414)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java
:164)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSyste
m.java:124)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFi
les.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1332)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit
(CrunchControlledJob.java:305)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startRead
yJobs(CrunchJobControl.java:180)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobSt
atusAndStartNewOnes(CrunchJobControl.java:209)
at
org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100)
at
org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51)
at
org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.UnknownHostException: bdatadev
... 27 more
However nowhere in my code a host named "bdatadev" is mentioned and also, I
cannot ping this host.
The section of the code where I am setting up the MRPipeline is as follows:
Configuration conf = HBaseConfiguration.create();
conf.set("fs.defaultFS", "hdfs://<server_address>:8020");
conf.set("mapred.job.tracker", "<server_address>:8021");
System.out.println("Hadoop configuration created.");
System.out.println("Initializing crunch pipeline ...");
conf.set("mapred.jar", "<path_to_jar_file>");
pipeline = new MRPipeline(getClass(), "crunchjobtest", conf);
Has anyone faced this issue before and knows how to resolve it/point out if
I am missing anything?
Thanks,
Narlin.
