I forgot to provide this earlier. Here is nutch ndfs -ls output for the
directory structure of a segment with a failed part-00013.
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls /opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133
051103 162002 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162003 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162003 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162003 Client connection to 192.168.100.15:5466: starting
Found 6 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text
<dir>
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content
051103 162010 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162011 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162011 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162011 Client connection to 192.168.100.15:5466: starting
Found 20 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00000
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00001
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00002
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00003
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00004
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00005
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00006
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00007
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00008
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00009
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00010
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00011
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00012
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00013
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00014
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00015
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00016
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00017
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00018
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00019
<dir>
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00012
051103 162017 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162017 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162017 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162017 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00012/data
439524693
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00012/index
56208
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00013
051103 162019 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162019 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162019 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162020 Client connection to 192.168.100.15:5466: starting
Found 0 items
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00014
051103 162021 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162022 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162022 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162022 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00014/data
440339945
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/content/part-00014/index
56183
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch
051103 162033 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162034 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162034 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162034 Client connection to 192.168.100.15:5466: starting
Found 20 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00000
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00001
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00002
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00003
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00004
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00005
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00006
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00007
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00008
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00009
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00010
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00011
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00012
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00013
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00014
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00015
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00016
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00017
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00018
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00019
<dir>
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00013
051103 162039 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162039 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162039 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162039 Client connection to 192.168.100.15:5466: starting
Found 0 items
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00012
051103 162041 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162041 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162042 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162042 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00012/data
8784520
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00012/index
56208
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00014
051103 162043 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162043 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162044 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162044 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00014/data
8788470
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_fetch/part-00014/index
56183
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate
051103 162055 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162055 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162055 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162055 Client connection to 192.168.100.15:5466: starting
Found 20 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00000
9531698
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00001
9684746
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00002
9762019
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00003
9715727
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00004
9518134
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00005
9676499
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00006
9722801
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00007
9715404
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00008
9514007
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00009
9668149
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00010
9649085
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00011
9726466
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00012
9534012
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00013
9744911
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00014
9694646
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00015
9652845
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00016
9505674
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00017
9700052
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00018
9714650
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_generate/part-00019
9714743
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse
051103 162108 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162109 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162109 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162109 Client connection to 192.168.100.15:5466: starting
Found 19 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00000
155306656
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00001
163093258
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00002
155290671
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00003
163551019
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00004
156198582
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00005
163963632
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00006
155873286
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00007
162752185
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00008
155215446
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00009
163084991
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00010
154982905
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00011
164212118
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00012
154450623
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00014
155279291
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00015
163724449
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00016
154542758
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00017
162865027
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00018
154375952
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/crawl_parse/part-00019
162991584
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data
051103 162121 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162122 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162122 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162122 Client connection to 192.168.100.15:5466: starting
Found 20 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00000
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00001
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00002
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00003
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00004
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00005
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00006
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00007
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00008
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00009
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00010
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00011
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00012
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00013
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00014
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00015
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00016
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00017
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00018
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00019
<dir>
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00012
051103 162127 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162127 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162127 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162127 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00012/data
128385655
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00012/index
56509
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00013
051103 162129 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162129 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162129 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162129 Client connection to 192.168.100.15:5466: starting
Found 0 items
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00014
051103 162131 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162131 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162131 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162131 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00014/data
128731018
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_data/part-00014/index
55566
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text
051103 162139 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162140 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162140 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162140 Client connection to 192.168.100.15:5466: starting
Found 20 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00000
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00001
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00002
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00003
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00004
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00005
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00006
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00007
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00008
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00009
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00010
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00011
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00012
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00013
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00014
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00015
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00016
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00017
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00018
<dir>
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00019
<dir>
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00012
051103 162145 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162145 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162145 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162145 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00012/data
111853821
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00012/index
56509
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00013
051103 162147 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162147 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162147 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162147 Client connection to 192.168.100.15:5466: starting
Found 0 items
[EMAIL PROTECTED] ~]$ /opt/nutch/bin/nutch ndfs
-ls
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00014
051103 162149 parsing file:/opt/nutch-0.8_7/conf/nutch-default.xml
051103 162149 parsing file:/opt/nutch-0.8_7/conf/nutch-site.xml
051103 162149 No FS indicated, using
default:master1.sitebuildit.com:5466
051103 162149 Client connection to 192.168.100.15:5466: starting
Found 2 items
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00014/data
111121278
/opt/sitesell/sbider_data/nutch/segments/20051102031132/20051102031133/parse_text/part-00014/index
55566
On Thu, 2005-11-03 at 15:32 -0500, Rod Taylor wrote:
> Sources are from October 31st. Sun Standard Edition 1.5.0_02-b09 for
> amd64
>
> Every segment that I fetch seems to be missing a part when stored on the
> filesystem. The stranger thing is it is always the same part (very
> reproducible).
>
> If I have mapred.reduce.tasks set to 20, the hole is at part 13. That
> is, the part-00013 directory is empty while the remainder (0 through 12,
> 14 through 19) all have data.
>
> If I have mapred.reduce.tasks set to 19, the hole is at part 11.
> content/part-00011 is empty.
>
> Attached are my site configuration (reduce.tasks is 19), task log for a
> failing task and the output from the job tracker.
>
> Below is a snippet from the datanode log (the only errors that exist are
> related to this task or others which process the above part #) and below
> that the output from localhost:7845 on the jobtracker machine for the
> job.
>
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.read(DataInputStream.java:134)
> at org.apache.nutch.ndfs.DataNode
> $DataXceiver.run(DataNode.java:369)
> at java.lang.Thread.run(Thread.java:595)
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.read(DataInputStream.java:134)
> at org.apache.nutch.ndfs.DataNode
> $DataXceiver.run(DataNode.java:369)
> at java.lang.Thread.run(Thread.java:595)
>
>
> Job 'job_k1p80p'
>
> Job File: /home/sitesell/system/submit_2pgex8/job.xml
> Start time: Thu Nov 03 12:04:43 EST 2005
> The job failed at: Thu Nov 03 16:00:42 EST 2005
>
> __________________________________________________________________________________________________
>
> Map Tasks
>
> Map Task Id Pct Complete State
> Diagnostic Text
> task_m_2m1twe 1.0 103189 pages, 5045 errors, 13.1
> pages/s, 1000 kb/s,
> task_m_4nzguk 1.0 103141 pages, 5193 errors, 12.9
> pages/s, 988 kb/s,
> task_m_5aprs2 1.0 103427 pages, 4756 errors, 13.4
> pages/s, 1027 kb/s,
> task_m_6pd5q7 1.0 102650 pages, 5081 errors, 12.6
> pages/s, 962 kb/s,
> task_m_8qzj8p 1.0 103610 pages, 4539 errors, 13.6
> pages/s, 1039 kb/s,
> task_m_aev1di 1.0 102666 pages, 4997 errors, 13.2
> pages/s, 1007 kb/s,
> task_m_f2zfyw 1.0 103235 pages, 4662 errors, 13.6
> pages/s, 1045 kb/s,
> task_m_f84hfi 1.0 103746 pages, 4657 errors, 13.0
> pages/s, 991 kb/s,
> task_m_hhv9b9 1.0 102909 pages, 4972 errors, 13.5
> pages/s, 1026 kb/s,
> task_m_kijqqx 1.0 103439 pages, 4858 errors, 13.4
> pages/s, 1024 kb/s,
> task_m_n5mxax 1.0 102894 pages, 4953 errors, 13.3
> pages/s, 1017 kb/s,
> task_m_p45m8c 1.0 103705 pages, 4969 errors, 13.1
> pages/s, 1007 kb/s,
> task_m_qfevss 1.0 102640 pages, 5006 errors, 13.2
> pages/s, 1011 kb/s,
> task_m_qg3816 1.0 103658 pages, 5039 errors, 13.3
> pages/s, 1014 kb/s,
> task_m_rlxmuw 1.0 103609 pages, 4491 errors, 13.6
> pages/s, 1038 kb/s,
> task_m_t9ksdc 1.0 103053 pages, 5287 errors, 12.9
> pages/s, 994 kb/s,
> task_m_wt3oyf 1.0 103006 pages, 5168 errors, 13.3
> pages/s, 1014 kb/s,
> task_m_xk3gxz 1.0 103294 pages, 5216 errors, 13.0
> pages/s, 996 kb/s,
> task_m_yjrejy 1.0 103158 pages, 4787 errors, 13.5
> pages/s, 1038 kb/s,
>
> __________________________________________________________________________________________________
>
> Reduce Task Id Pct Complete State Diagnostic Text
> task_r_2ktith 1.0 reduce > reduce
> task_r_6hwvi0 1.0 reduce > reduce
> task_r_8bi6h5 1.0 reduce > reduce
> task_r_bpisbi 1.0 reduce > reduce
> task_r_cfoo7z 1.0 reduce > reduce
> task_r_cmy1r3 1.0 reduce > reduce
> task_r_efnd4k 1.0 reduce > reduce
> task_r_ervlp5 1.0 reduce > reduce
> task_r_kvmno7 1.0 reduce > reduce
> task_r_n4q36e 1.0 reduce > reduce
> task_r_o4st5w 1.0 reduce > reduce
> task_r_ow0sul 1.0 reduce > reduce
> task_r_r7u152 1.0 reduce > reduce
> task_r_ra99xx 1.0 reduce > reduce
> task_r_ush85v 1.0 reduce > reduce
> task_r_vbmkfw 1.0 reduce > reduce
> task_r_wbirax 1.0 reduce > reduce
> task_r_z17yss 1.0 reduce > reduce
> task_r_o9mv91 0.9153447 reduce > reduce Timed
> out.java.io.IOException: Task process exit with nonzero status.
> at org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139)
> at
> org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
> out.java.io.IOException: Task process exit
> with nonzero status. at
> org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
> org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
> out.java.io.IOException: Task process exit
> with nonzero status. at
> org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
> org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92) Timed
> out.java.io.IOException: Task process exit
> with nonzero status. at
> org.apache.nutch.mapred.TaskRunner.runChild(TaskRunner.java:139) at
> org.apache.nutch.mapred.TaskRunner.run(TaskRunner.java:92)
>
>
--
Rod Taylor <[EMAIL PROTECTED]>
