Sources are from October 31st. Sun Standard Edition 1.5.0_02-b09 for amd64 Every segment that I fetch seems to be missing a part when stored on the filesystem. The stranger thing is it is always the same part (very reproducible).
If I have mapred.reduce.tasks set to 20, the hole is at part 13. That
is, the part-00013 directory is empty while the remainder (0 through 12,
14 through 19) all have data.
If I have mapred.reduce.tasks set to 19, the hole is at part 11.
content/part-00011 is empty.
Attached are my site configuration (reduce.tasks is 19), task log for a
failing task and the output from the job tracker.
Below is a snippet from the datanode log (the only errors that exist are
related to this task or others which process the above part #) and below
that the output from localhost:7845 on the jobtracker machine for the
job.
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.read(DataInputStream.java:134)
at org.apache.nutch.ndfs.DataNode
$DataXceiver.run(DataNode.java:369)
at java.lang.Thread.run(Thread.java:595)
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.read(DataInputStream.java:134)
at org.apache.nutch.ndfs.DataNode
$DataXceiver.run(DataNode.java:369)
at java.lang.Thread.run(Thread.java:595)
Job 'job_k1p80p'
Job File: /home/sitesell/system/submit_2pgex8/job.xml
Start time: Thu Nov 03 12:04:43 EST 2005
The job failed at: Thu Nov 03 16:00:42 EST 2005
__________________________________________________________________________________________________
Map Tasks
Map Task Id Pct Complete State
Diagnostic Text
task_m_2m1twe 1.0 103189 pages, 5045 errors, 13.1
pages/s, 1000 kb/s,
task_m_4nzguk 1.0 103141 pages, 5193 errors, 12.9
pages/s, 988 kb/s,
task_m_5aprs2 1.0 103427 pages, 4756 errors, 13.4
pages/s, 1027 kb/s,
task_m_6pd5q7 1.0 102650 pages, 5081 errors, 12.6
pages/s, 962 kb/s,
task_m_8qzj8p 1.0 103610 pages, 4539 errors, 13.6
pages/s, 1039 kb/s,
task_m_aev1di 1.0 102666 pages, 4997 errors, 13.2
pages/s, 1007 kb/s,
task_m_f2zfyw 1.0 103235 pages, 4662 errors, 13.6
pages/s, 1045 kb/s,
task_m_f84hfi 1.0 103746 pages, 4657 errors, 13.0
pages/s, 991 kb/s,
task_m_hhv9b9 1.0 102909 pages, 4972 errors, 13.5
pages/s, 1026 kb/s,
task_m_kijqqx 1.0 103439 pages, 4858 errors, 13.4
pages/s, 1024 kb/s,
task_m_n5mxax 1.0 102894 pages, 4953 errors, 13.3
pages/s, 1017 kb/s,
task_m_p45m8c 1.0 103705 pages, 4969 errors, 13.1
pages/s, 1007 kb/s,
task_m_qfevss 1.0 102640 pages, 5006 errors, 13.2
pages/s, 1011 kb/s,
task_m_qg3816 1.0 103658 pages, 5039 errors, 13.3
pages/s, 1014 kb/s,
task_m_rlxmuw 1.0 103609 pages, 4491 errors, 13.6
pages/s, 1038 kb/s,
task_m_t9ksdc 1.0 103053 pages, 5287 errors, 12.9
pages/s, 994 kb/s,
task_m_wt3oyf 1.0 103006 pages, 5168 errors, 13.3
pages/s, 1014 kb/s,
task_m_xk3gxz 1.0 103294 pages, 5216 errors, 13.0
pages/s, 996 kb/s,
task_m_yjrejy 1.0 103158 pages, 4787 errors, 13.5
pages/s, 1038 kb/s,
__________________________________________________________________________________________________
Reduce Task Id Pct Complete State Diagnostic Text
task_r_2ktith 1.0 reduce > reduce
task_r_6hwvi0 1.0 reduce > reduce
task_r_8bi6h5 1.0 reduce > reduce
task_r_bpisbi 1.0 reduce > reduce
task_r_cfoo7z 1.0 reduce > reduce
task_r_cmy1r3 1.0 reduce > reduce
task_r_efnd4k 1.0 reduce > reduce
task_r_ervlp5 1.0 reduce > reduce
task_r_kvmno7 1.0 reduce > reduce
task_r_n4q36e 1.0 reduce > reduce
task_r_o4st5w 1.0 reduce > reduce
task_r_ow0sul 1.0 reduce > reduce
task_r_r7u152 1.0 reduce > reduce
task_r_ra99xx 1.0 reduce > reduce
task_r_ush85v 1.0 reduce > reduce
task_r_vbmkfw 1.0 reduce > reduce
task_r_wbirax 1.0 reduce > reduce
task_r_z17yss 1.0 reduce > reduce
task_r_o9mv91 0.9153447 reduce > reduce Timed
out.java.io.IOException: Task process exit with nonzero status.
at org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139)
at
org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
out.java.io.IOException: Task process exit
with nonzero status. at
org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
out.java.io.IOException: Task process exit
with nonzero status. at
org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
out.java.io.IOException: Task process exit
with nonzero status. at
org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92)
--
Rod Taylor <[EMAIL PROTECTED]>
051103 142051 Adding task 'task_r_o4st5w' to set for tracker 'tracker_23756' 051103 142051 Adding task 'task_r_8bi6h5' to set for tracker 'tracker_20790' 051103 142051 Adding task 'task_r_o9mv91' to set for tracker 'tracker_45593' 051103 142054 Adding task 'task_r_wbirax' to set for tracker 'tracker_13766' 051103 142054 Adding task 'task_r_r7u152' to set for tracker 'tracker_35947' 051103 142054 Adding task 'task_r_cmy1r3' to set for tracker 'tracker_23756' 051103 142054 Adding task 'task_r_efnd4k' to set for tracker 'tracker_20790' 051103 142054 Adding task 'task_r_ush85v' to set for tracker 'tracker_45593' 051103 142057 Adding task 'task_r_kvmno7' to set for tracker 'tracker_13766' 051103 142057 Adding task 'task_r_z17yss' to set for tracker 'tracker_35947' 051103 144502 Task 'task_r_efnd4k' has finished successfully. 051103 144531 Task 'task_r_ow0sul' has finished successfully. 051103 144538 Task 'task_r_bpisbi' has finished successfully. 051103 144549 Task 'task_r_o4st5w' has finished successfully. 051103 144602 Task 'task_r_cfoo7z' has finished successfully. 051103 144604 Task 'task_r_cmy1r3' has finished successfully. 051103 144604 Task 'task_r_kvmno7' has finished successfully. 051103 144613 Task 'task_r_vbmkfw' has finished successfully. 051103 144628 Task 'task_r_wbirax' has finished successfully. 051103 144629 Task 'task_r_8bi6h5' has finished successfully. 051103 144646 Task 'task_r_6hwvi0' has finished successfully. 051103 144917 Task 'task_r_ra99xx' has finished successfully. 051103 144938 Task 'task_r_2ktith' has finished successfully. 051103 145011 Task 'task_r_n4q36e' has finished successfully. 051103 145050 Task 'task_r_z17yss' has finished successfully. 051103 145117 Task 'task_r_r7u152' has finished successfully. 051103 145123 Task 'task_r_ervlp5' has finished successfully. 051103 145622 Task 'task_r_o9mv91' has been lost. 051103 145622 Adding task 'task_r_o9mv91' to set for tracker 'tracker_45593' 051103 150050 Task 'task_r_ush85v' has finished successfully. 051103 151743 Task 'task_r_o9mv91' has been lost. 051103 151743 Adding task 'task_r_o9mv91' to set for tracker 'tracker_45593' 051103 153856 Task 'task_r_o9mv91' has been lost. 051103 153856 Adding task 'task_r_o9mv91' to set for tracker 'tracker_45593'
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="nutch-conf.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<nutch-conf>
<!-- CONFIGURED PER NUTCH INSTANCE -->
<property>
<name>ndfs.name.dir</name>
<value>/opt/sitesell/sbider_data/nutch/ndfs/system</value>
<description>Determines where on the local filesystem the NDFS name node
should store the name table.</description>
</property>
<property>
<name>ndfs.data.dir</name>
<value>/opt/sitesell/sbider_data/nutch/ndfs/data</value>
<description>Determines where on the local filesystem an NDFS data node
should store its blocks. If this is a comma- or space-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.</description>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/sitesell/local</value>
<description>The local directory where MapReduce stores intermediate
data files.
</description>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/sitesell/system</value>
<description>The local directory where MapReduce stores control files.
</description>
</property>
<!-- HTTP User-Agent settings -->
<property>
<name>http.agent.name</name>
<value>SBIder</value>
<description>Our HTTP 'User-Agent' request header.</description>
</property>
<property>
<name>http.agent.description</name>
<value>SBIder</value>
<description>Further description of our bot- this text is used in
the User-Agent header. It appears in parenthesis after the agent name.
</description>
</property>
<property>
<name>http.robots.agents</name>
<value>SBIder,NutchCVS,Nutch,*</value>
<description>The agent strings we'll look for in robots.txt files,
comma-separated, in decreasing order of precedence.</description>
</property>
<property>
<name>http.agent.url</name>
<value>http://www.sitesell.com/sbider.html</value>
<description>A URL to advertise in the User-Agent header. This will
appear in parenthesis after the agent name.
</description>
</property>
<property>
<name>http.agent.email</name>
<value>http://support.sitesell.com/contact-support.html</value>
<description>An email address to advertise in the HTTP 'From' request
header and User-Agent header.</description>
</property>
<property>
<name>http.timeout</name>
<value>10000</value>
<description>The default network timeout, in milliseconds.</description>
</property>
<property>
<name>http.max.delays</name>
<value>3</value>
<description>The number of times a thread will delay when trying to
fetch a page. Each time it finds that a host is busy, it will wait
fetcher.server.delay. After http.max.delays attepts, it will give
up on the page for now.</description>
</property>
<property>
<name>http.content.limit</name>
<value>20480</value>
<description>The length limit for downloaded content, in bytes.
If this value is nonnegative (>=0), content longer than it will be truncated;
otherwise, no truncation at all.
</description>
</property>
<!-- ndfs properties -->
<property>
<name>ndfs.replication</name>
<value>1</value>
<description>The target number of times to replicate a data block.</description>
</property>
<!-- generate properties -->
<property>
<name>generate.max.per.host</name>
<value>5</value>
<description>The maximum number of urls per host in a single
fetchlist. -1 if unlimited.</description>
</property>
<!-- Fetcher properties -->
<property>
<name>fetcher.threads.fetch</name>
<value>40</value>
<description>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection).</description>
</property>
<!-- Filesystem properties -->
<property>
<name>fs.default.name</name>
<value>master1.sitebuildit.com:5466</value>
<description>The name of the default file system. Either the
literal string "local" or a host:port for NDFS.</description>
</property>
<!-- Map Reduce properties -->
<property>
<name>mapred.job.tracker</name>
<value>master1.sitebuildit.com:5464</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>mapred.map.tasks</name>
<value>193</value>
<description>The default number of map tasks per job. Typically set
to a prime several times greater than number of available hosts.
Ignored when mapred.job.tracker is "local".
</description>
</property>
<property>
<name>mapred.tasktracker.tasks.maximum</name>
<value>4</value>
<description>The number of tasks to run on a given tasktracker</description>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>19</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>
<!-- Plugin Inclusions -->
<property>
<name>plugin.includes</name>
<value>nutch-extensionpoints|protocol-(httpclient|http)|urlfilter-regex|parse-(text|html|js)</value>
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded.
In any case you need at least include the nutch-extensionpoints plugin. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins.
</description>
</property>
</nutch-conf>
task_r_o9mv91.log.gz
Description: GNU Zip compressed data
