Thanks for the answer. It looks like the values are setup correctly.
I see /mapred/local directory created successfully as well. Do I need
to explicity define a value in hadoop-site.xml?
hadoop-default.xml:
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
</property>
and in hadoop-site.xml:
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-datastore</value>
<description>A base for other temporary directories.</description>
</property>
On Oct 30, 2008, at 2:40 PM, pvvpr wrote:
Hi Scott,
Your reducer classes are unable to read map outputs. You may check
"mapred.local.dir" property in your conf/hadoop-default.xml and
conf/hadoop-site.xml . These directories should be valid directories
in
your slaves. You can give multiple folders with comma separated
values.
- Prasad.
On Thursday 30 October 2008 11:32:02 pm Scott Whitecross wrote:
So its not just at 16%, but depends on the task:
2008-10-30 13:58:29,702 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_200810301345_0001_r_000000_0 0.25675678% reduce > copy (57 of
74 at 13.58 MB/s) >
2008-10-30 13:58:29,357 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_200810301345_0001_m_000048_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_200810301345_0001/
attempt_200810301345_0001_m_000048_0/output/file.out.index in any of
the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator
$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:359)
at
org
.apache
.hadoop
.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:
138)
at org.apache.hadoop.mapred.TaskTracker
$MapOutputServlet.doGet(TaskTracker.java:2402)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
I'm out of thoughts on what the problem could be..
On Oct 30, 2008, at 12:35 PM, Scott Whitecross wrote:
I'm growing very frustrated with a simple cluster setup. I can get
the cluster setup on two machines, but have troubles when trying to
extend the installation to 3 or more boxes. I keep seeing the below
errors. It seems the reduce tasks can't get access to the data.
I can't seem to figure out how to fix this error. What amazes me
is that file not found issues appear on the master box, as well as
the slaves. What causes the reduce tasks to not read find
information via the localhost?
Setup/Errors:
My basic setup comes from:
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-No
de_Cluster) (Michael Noll's setup). I've put the following in the
my
/etc/ hosts file:
127.0.0.1 localhost
10.1.1.12 master
10.1.1.10 slave
10.1.1.13 slave1
And have setup transparent ssh to all boxes (and it works). All
boxes can see each other, etc.
My base level hadoop-site.xml is:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-datastore</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>master:54311</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
Errors:
WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_200810301206_0004_m_000001_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find taskTracker/jobcache/job_200810301206_0004/
attempt_200810301206_0004_m_000001_0/output/file.out.index in any of
the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator
$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:359)
at
org
.apache
.hadoop
.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:
138)...
and in the userlog of the attempt:
2008-10-30 12:28:00,806 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://localhost:50060/mapOutput?job=job_200810301206_0004&map=attempt_20
0810301206_0004_m_000001_0&reduce=0 at
sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown
Source)
at
sun
.reflect
.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:27)
