Arv Mistry wrote:
I'll try again, can anyone tell me should it be possible to run hadoop
in a pseudo-distributed mode (i.e. everything on one machine) and then
submit a mapred job using the ToolRunner from another machine on that
hadoop configuration?
Cheers Arv
Yes. It is possible to do. You can start hadoop cluster on single node.
Documentation available at
http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed
Once the cluster is up, you can submit jobs from any client, but the
client configuration should be aware of Namenode and JobTracker nodes.
You can use the generic options *-fs* and *-jt* on commandline for the same.
Thanks
Amareshwari
-----Original Message-----
From: Arv Mistry [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 31, 2008 2:32 PM
To: [email protected]
Subject: Running mapred job from remote machine to a pseudo-distributed
hadoop
I have hadoop setup in a pseudo-distributed mode i.e. everything on one
machine, And I'm trying to submit a hadoop mapred job from another
machine to that hadoop setup.
At the point that I run the mapred job I get the following error. Any
ideas as to what I'm doing wrong?
Is this possible in a pseudo-distributed mode?
Cheers Arv
INFO | jvm 1 | 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR
[HadoopJobTool] java.io.IOException:
/tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such
file or directory
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
java.lang.reflect.Method.invoke(Method.java:597)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
INFO | jvm 1 | 2008/07/31 14:01:00 |
INFO | jvm 1 | 2008/07/31 14:01:00 |
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
/tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such
file or directory
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
java.lang.reflect.Method.invoke(Method.java:597)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
INFO | jvm 1 | 2008/07/31 14:01:00 |
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.Client.call(Client.java:557)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
$Proxy5.submitJob(Unknown Source)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
java.lang.reflect.Method.invoke(Method.java:597)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
$Proxy5.submitJob(Unknown Source)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.run(Hadoop
JobTool.java:129)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.launchJob(
HadoopJobTool.java:142)
INFO | jvm 1 | 2008/07/31 14:01:00 | at
com.rialto.profiler.profiler.clickstream.RawStreamGenerator.run(RawStrea
mGenerator.java:138)
