With CDH3B4, the hadoop processes run as separate users (like hdfs, mapred, etc). Did you set the CDH3B4 directory permissions correctly as described in the install document? See: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3 and search for "permissions". Also see this: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3#UpgradingtoCDH3-ChangesinUserAccountsandGroupsinCDH3DuetoSecurity
In this case, make sure that the /tmp/,,,/staging/... directory is accessible by the task user account (most likely mapred). What seems to be happenning in your case is that the task is not able to access your /tmp directory due to user permissions. If this still doesn't work - try running one of the MR hadoop-*-examples (like "pi") so that you can validate that the cluster is setup correctly. --Suraj On Mon, Apr 25, 2011 at 9:18 PM, Pete Tyler <[email protected]> wrote: > > My map reduce jobs, which were running fine with HBase 0.20.4 with Hadoop > 0.20.2 are now failing as I try to upgrade to HBase 0.90.1 with Hadoop > 0.20.2-CDH3B4. > > Under 0.90.1 I see the following error, > > Error initializing attempt_201104252111_0001_m_000002_0: > java.io.FileNotFoundException: File > /tmp/hadoop-peter/mapred/staging/peter/.staging/job_201104252111_0001/job.jar > does not exist. > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) > at > org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61) > at > org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381) > at > org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) > at > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129) > at > org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055) > at > org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212) > at > org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2176) > > My code, which worked fine under 0.20.4 looks like this, > > Configuration conf = HBaseConfiguration.create(); > conf.set("mapred.job.tracker", "localhost:9001"); > > Job job = new Job(conf); > JobConf jobConf = job.getConfiguration(); > jobConf.setJar( dirName + "/" + jarName); > > job.setJarByClass(MyMapper.class); > job.setJobName("My MapReduce Pseudo Distributed"); > > Scan scan = new Scan(); > > String tableNameIn = "MyTableIn"; > String tableNameOut = "MyTableOut"; > > TableMapReduceUtil.initTableMapperJob( > tableNameIn, > scan, > MyMapper.class, > ImmutableBytesWritable.class, > ImmutableBytesWritable.class, > job > ); > > TableMapReduceUtil.initTableReducerJob( > tableNameOut, > MyReducer.class, > job > ); > > job.setOutputFormatClass(TableOutputFormat.class); > job.setNumReduceTasks(1); > > job.waitForCompletion(true); > > Any pointers on how to migrate my code so it works on 0.90.1 would be > brilliant. Many thanks. > -- > View this message in context: > http://old.nabble.com/Migrating-pseudo-distribute-map-reduce-with-user-jar-file-to-0.90.1-tp31475457p31475457.html > Sent from the HBase User mailing list archive at Nabble.com. > >
