My map reduce jobs, which were running fine with HBase 0.20.4 with Hadoop
0.20.2 are now failing as I try to upgrade to HBase 0.90.1 with Hadoop
0.20.2-CDH3B4.
Under 0.90.1 I see the following error,
Error initializing attempt_201104252111_0001_m_000002_0:
java.io.FileNotFoundException: File
/tmp/hadoop-peter/mapred/staging/peter/.staging/job_201104252111_0001/job.jar
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
at
org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2176)
My code, which worked fine under 0.20.4 looks like this,
Configuration conf = HBaseConfiguration.create();
conf.set("mapred.job.tracker", "localhost:9001");
Job job = new Job(conf);
JobConf jobConf = job.getConfiguration();
jobConf.setJar( dirName + "/" + jarName);
job.setJarByClass(MyMapper.class);
job.setJobName("My MapReduce Pseudo Distributed");
Scan scan = new Scan();
String tableNameIn = "MyTableIn";
String tableNameOut = "MyTableOut";
TableMapReduceUtil.initTableMapperJob(
tableNameIn,
scan,
MyMapper.class,
ImmutableBytesWritable.class,
ImmutableBytesWritable.class,
job
);
TableMapReduceUtil.initTableReducerJob(
tableNameOut,
MyReducer.class,
job
);
job.setOutputFormatClass(TableOutputFormat.class);
job.setNumReduceTasks(1);
job.waitForCompletion(true);
Any pointers on how to migrate my code so it works on 0.90.1 would be
brilliant. Many thanks.
--
View this message in context:
http://old.nabble.com/Migrating-pseudo-distribute-map-reduce-with-user-jar-file-to-0.90.1-tp31475457p31475457.html
Sent from the HBase User mailing list archive at Nabble.com.
