I'll try again, can anyone tell me should it be possible to run hadoop in a pseudo-distributed mode (i.e. everything on one machine) and then submit a mapred job using the ToolRunner from another machine on that hadoop configuration?
Cheers Arv -----Original Message----- From: Arv Mistry [mailto:[EMAIL PROTECTED] Sent: Thursday, July 31, 2008 2:32 PM To: [email protected] Subject: Running mapred job from remote machine to a pseudo-distributed hadoop I have hadoop setup in a pseudo-distributed mode i.e. everything on one machine, And I'm trying to submit a hadoop mapred job from another machine to that hadoop setup. At the point that I run the mapred job I get the following error. Any ideas as to what I'm doing wrong? Is this possible in a pseudo-distributed mode? Cheers Arv INFO | jvm 1 | 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR [HadoopJobTool] java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1 | 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1 | 2008/07/31 14:01:00 | INFO | jvm 1 | 2008/07/31 14:01:00 | org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1 | 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1 | 2008/07/31 14:01:00 | INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Client.call(Client.java:557) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) INFO | jvm 1 | 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1 | 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1 | 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo cationHandler.java:82) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation Handler.java:59) INFO | jvm 1 | 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973) INFO | jvm 1 | 2008/07/31 14:01:00 | at com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.run(Hadoop JobTool.java:129) INFO | jvm 1 | 2008/07/31 14:01:00 | at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) INFO | jvm 1 | 2008/07/31 14:01:00 | at com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.launchJob( HadoopJobTool.java:142) INFO | jvm 1 | 2008/07/31 14:01:00 | at com.rialto.profiler.profiler.clickstream.RawStreamGenerator.run(RawStrea mGenerator.java:138)
