We have been playing with a SegmentReader (we have a working copy) and
may have run into a limitation or bug within NDFS. Occurs on mapred code
for both the 18th and 20th.
When running the attached SegmentReader.java on a small segment (10k
urls or less) it runs problem free but on larger segments the NameNode
seems to go a little crazy (gets stuck in some kind of a loop).
The exact same code writing to stdout functions as expected even when a
segment is several million URLs or tens of GBs.
segread.op is the client output
Below is the NameNode output. These exceptions continue until you kill
the daemon.
java.io.IOException: Cannot abandon block during write
to
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
051019 174056 Server handler on 5466 call error:
java.io.IOException: Cannot abandon block during write
to
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
java.io.IOException: Cannot abandon block during write
to
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
051019 174056 Server handler on 5466 call error:
java.io.IOException: Cannot abandon block during write
to
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
java.io.IOException: Cannot abandon block during write
to
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
--
Rod Taylor <[EMAIL PROTECTED]>
051019 135031 Reader: segment:
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121
051019 135032 parsing file:/opt/nutch-0.8_4/conf/nutch-default.xml
051019 135032 parsing file:/opt/nutch-0.8_4/conf/mapred-default.xml
051019 135032 parsing file:/opt/nutch-0.8_4/conf/nutch-site.xml
051019 135032 parsing file:/opt/nutch-0.8_4/conf/nutch-default.xml
051019 135032 parsing file:/opt/nutch-0.8_4/conf/nutch-site.xml
051019 135032 Client connection to 192.168.100.14:5464: starting
051019 135032 Client connection to 192.168.100.14:5466: starting
051019 135113 Running job: job_ja0rtg
051019 135114 map 0%
051019 135141 map 1%
051019 135143 map 2%
051019 135146 map 4%
051019 135150 map 5%
051019 135153 map 7%
051019 135155 map 9%
051019 135158 map 10%
051019 135202 map 11%
051019 135205 map 13%
051019 135208 map 14%
051019 135210 map 16%
051019 135213 map 17%
051019 135217 map 19%
051019 135220 map 20%
051019 135222 map 21%
051019 135225 map 23%
051019 135229 map 24%
051019 135231 map 25%
051019 135232 map 26%
051019 135234 map 27%
051019 135237 map 29%
051019 135241 map 30%
051019 135243 map 31%
051019 135244 map 32%
051019 135246 map 33%
051019 135249 map 34%
051019 135253 map 35%
051019 135256 map 37%
051019 135258 map 39%
051019 135301 map 40%
051019 135305 map 41%
051019 135308 map 43%
051019 135311 map 44%
051019 135314 map 45%
051019 135317 map 46%
051019 135320 map 47%
051019 135326 map 49%
051019 135329 map 50%
051019 135334 map 51%
051019 135338 map 52%
051019 135341 map 53%
051019 135344 map 54%
051019 135350 map 55%
051019 135353 map 56%
051019 135356 map 57%
051019 135358 map 58%
051019 135409 map 59%
051019 135410 map 61%
051019 135412 map 62%
051019 135416 map 63%
051019 135420 map 64%
051019 135424 map 65%
051019 135428 map 66%
051019 135431 map 67%
051019 135434 map 68%
051019 135437 map 69%
051019 135443 map 70%
051019 135446 map 71%
051019 135449 map 72%
051019 135453 map 73%
051019 135457 map 74%
051019 135500 map 75%
051019 135504 map 76%
051019 135507 map 78%
051019 135510 map 79%
051019 135512 map 80%
051019 135513 map 81%
051019 135517 map 83%
051019 135519 map 85%
051019 135522 map 86%
051019 135525 map 87%
051019 135528 map 88%
051019 135529 map 89%
051019 135531 map 90%
051019 135534 map 92%
051019 135536 map 93%
051019 135537 map 94%
051019 135540 map 95%
051019 135541 map 96%
051019 135544 map 97%
051019 135546 map 99%
051019 135549 map 100%
051019 140628 reduce 0%
051019 140632 reduce 1%
051019 140644 reduce 2%
051019 140650 reduce 3%
051019 140659 reduce 4%
051019 140705 reduce 5%
051019 140714 reduce 6%
051019 140723 reduce 7%
051019 140732 reduce 8%
051019 140738 reduce 9%
051019 140746 reduce 10%
051019 140750 reduce 11%
051019 140755 reduce 12%
051019 140759 reduce 13%
051019 140805 reduce 14%
051019 140811 reduce 15%
051019 140816 reduce 16%
051019 140822 reduce 17%
051019 140832 reduce 18%
051019 140833 reduce 19%
051019 140837 reduce 20%
051019 140843 reduce 21%
051019 140849 reduce 22%
051019 140853 reduce 23%
051019 140900 reduce 24%
051019 140906 reduce 25%
051019 140909 reduce 26%
051019 140912 reduce 27%
051019 140917 reduce 28%
051019 140920 reduce 29%
051019 140922 reduce 30%
051019 140925 reduce 31%
051019 140928 reduce 32%
051019 140932 reduce 33%
051019 140934 reduce 34%
051019 140938 reduce 35%
051019 140942 reduce 36%
051019 140946 reduce 37%
051019 140949 reduce 38%
051019 140952 reduce 39%
051019 140957 reduce 40%
051019 141001 reduce 41%
051019 141005 reduce 42%
051019 141012 reduce 43%
051019 141018 reduce 44%
051019 141022 reduce 45%
051019 141024 reduce 46%
051019 141027 reduce 47%
051019 141033 reduce 48%
051019 141042 reduce 49%
051019 141118 reduce 50%
051019 141203 reduce 51%
051019 141208 reduce 52%
051019 141232 reduce 53%
051019 141311 reduce 54%
051019 141418 reduce 55%
051019 141500 reduce 56%
051019 141509 reduce 57%
051019 141515 reduce 58%
051019 141603 reduce 59%
051019 141712 reduce 60%
051019 151521 reduce 61%
051019 151557 reduce 62%
051019 151937 reduce 63%
051019 152010 reduce 64%
051019 152639 reduce 65%
051019 152716 reduce 66%
051019 152929 reduce 67%
051019 153011 reduce 68%
051019 153138 reduce 69%
051019 153205 reduce 70%
051019 153252 reduce 71%
051019 153256 reduce 72%
051019 153317 reduce 73%
051019 153405 reduce 74%
051019 154309 reduce 75%
051019 154348 reduce 76%
051019 154741 reduce 77%
051019 154820 reduce 78%
051019 155256 reduce 79%
051019 155333 reduce 80%
051019 155500 reduce 81%
051019 155535 reduce 82%
051019 155539 reduce 83%
051019 155624 reduce 84%
051019 155633 reduce 85%
051019 155641 reduce 86%
051019 155657 reduce 87%
051019 155720 reduce 88%
051019 155732 reduce 89%
051019 155818 reduce 90%
051019 160006 reduce 91%
051019 160018 reduce 92%
051019 160030 reduce 93%
051019 160106 reduce 94%
051019 160136 reduce 95%
051019 160227 reduce 96%
051019 160416 reduce 92%
051019 160436 reduce 93%
051019 160500 reduce 95%
051019 160536 reduce 96%
051019 160603 reduce 97%
051019 160639 reduce 98%
051019 170417 reduce 99%
051019 170450 reduce 100%
051019 170529 Job complete: job_ja0rtg
051019 172819 Client connection to 192.168.100.14:5466: closing
051019 172821 Client connection to 192.168.100.14:5464: closing
051019 172919 Client connection to 192.168.100.14:5466: starting
051019 173152 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00002
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 173152 Checksum error:
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00002
051019 174229 Client connection to 192.168.100.14:5466: closing
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00004
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00005
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00006
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00007
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00008
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00009
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00010
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00011
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00012
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00013
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00014
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00015
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00016
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00017
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00018
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00019
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00020
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00021
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00022
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00023
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Couldn't copy the content of
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/part-00024
into
/opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/dump
051019 174330 Connection refused
051019 174330 Reader: done
/**
* Copyright 2005 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nutch.crawl;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.Iterator;
import java.util.logging.Logger;
import org.apache.nutch.fs.NutchFileSystem;
import org.apache.nutch.io.ObjectWritable;
import org.apache.nutch.io.UTF8;
import org.apache.nutch.io.Writable;
import org.apache.nutch.io.WritableComparable;
import org.apache.nutch.mapred.FileSplit;
import org.apache.nutch.mapred.JobClient;
import org.apache.nutch.mapred.JobConf;
import org.apache.nutch.mapred.OutputCollector;
import org.apache.nutch.mapred.RecordReader;
import org.apache.nutch.mapred.RecordWriter;
import org.apache.nutch.mapred.Reducer;
import org.apache.nutch.mapred.Reporter;
import org.apache.nutch.mapred.SequenceFileInputFormat;
import org.apache.nutch.mapred.SequenceFileRecordReader;
import org.apache.nutch.parse.ParseData;
import org.apache.nutch.parse.ParseText;
import org.apache.nutch.protocol.Content;
import org.apache.nutch.util.LogFormatter;
import org.apache.nutch.util.NutchConf;
import org.apache.nutch.util.NutchConfigured;
/* Dump the content of a segment. */
public class SegmentReader extends NutchConfigured implements Reducer {
public static final String DIR_NAME = "segdump";
public static final Logger LOG =
LogFormatter.getLogger(SegmentReader.class.getName());
long recNo = 0L;
/** Wraps inputs in an [EMAIL PROTECTED] ObjectWritable}, to permit merging different
* types in reduce. */
public static class InputFormat extends SequenceFileInputFormat {
public RecordReader getRecordReader(NutchFileSystem fs, FileSplit split,
JobConf job, Reporter reporter)
throws IOException {
reporter.setStatus(split.toString());
return new SequenceFileRecordReader(fs, split) {
public synchronized boolean next(Writable key, Writable value)
throws IOException {
ObjectWritable wrapper = (ObjectWritable)value;
try {
wrapper.set(getValueClass().newInstance());
} catch (Exception e) {
throw new IOException(e.toString());
}
return super.next(key, (Writable)wrapper.get());
}
};
}
}
/** Implements a text output format*/
public static class TextOutputFormat
implements org.apache.nutch.mapred.OutputFormat {
public RecordWriter getRecordWriter(final NutchFileSystem fs, JobConf job,
String name) throws IOException {
final File segmentDumpFile =
new File(new File(job.getOutputDir(), SegmentReader.DIR_NAME), name);
// Get the old copy out of the way
fs.delete(segmentDumpFile);
final PrintStream printStream = new PrintStream(fs.create(segmentDumpFile, true));
return new RecordWriter() {
public synchronized void write(WritableComparable key, Writable value)
throws IOException {
ObjectWritable writable = (ObjectWritable)value;
printStream.println((String)writable.get());
}
public synchronized void close(Reporter reporter) throws IOException {
printStream.close();
}
};
}
}
public SegmentReader() { super(null); }
public SegmentReader(NutchConf conf) {
super(conf);
}
public void configure(JobConf job) {
}
public void reduce(WritableComparable key, Iterator values,
OutputCollector output, Reporter reporter)
throws IOException {
StringBuffer dump = new StringBuffer();
dump.append("\nRecno:: ").append(recNo++).append("\n");
while (values.hasNext()) {
Object value = ((ObjectWritable)values.next()).get(); // unwrap
if (value instanceof CrawlDatum) {
dump.append("\nCrawlDatum::\n").append(((CrawlDatum)value).toString());
} else if (value instanceof Content) {
dump.append("\nContent::\n").append(((Content)value).toString());
} else if (value instanceof ParseData) {
dump.append("\nParseData::\n").append(((ParseData)value).toString());
} else if (value instanceof ParseText) {
dump.append("\nParseText::\n").append(((ParseText)value).toString());
} else {
LOG.warning("Unrecognized type: " + value.getClass());
}
}
output.collect(key, new ObjectWritable(dump.toString()));
}
public void reader(File segment) throws IOException {
LOG.info("Reader: segment: " + segment);
JobConf job = new JobConf(getConf());
job.addInputDir(new File(segment, CrawlDatum.FETCH_DIR_NAME));
job.addInputDir(new File(segment, Content.DIR_NAME));
job.addInputDir(new File(segment, ParseData.DIR_NAME));
job.addInputDir(new File(segment, ParseText.DIR_NAME));
job.setInputFormat(InputFormat.class);
job.setInputKeyClass(UTF8.class);
job.setInputValueClass(ObjectWritable.class);
job.setReducerClass(SegmentReader.class);
job.setOutputDir(segment);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(UTF8.class);
job.setOutputValueClass(ObjectWritable.class);
JobClient.runJob(job);
// concatenate the output
NutchFileSystem nfs = NutchFileSystem.get(job);
File directory = new File(job.getOutputDir(), SegmentReader.DIR_NAME);
File dumpFile = new File(directory, job.get("segment.dump.filename", "dump"));
File[] files = nfs.listFiles(directory);
DataOutputStream outputStream = null;
if (files.length > 0) {
outputStream = new DataOutputStream(nfs.create(dumpFile, true));
try {
for (int i = 0 ; i < files.length; i++) {
File partFile = (File)files[i];
try {
append(nfs, job, partFile, outputStream);
} catch (IOException exception) {
LOG.warning("Couldn't copy the content of " + partFile.toString() + " into " + dumpFile.toString());
LOG.warning(exception.getMessage());
}
}
}
finally {
outputStream.close();
}
}
LOG.info("Reader: done");
}
/** Appends two files TODO: A better place for this method is in FileUtil class*/
public void append(NutchFileSystem nfs, NutchConf conf, File src, OutputStream outputStream) throws IOException {
DataInputStream in = new DataInputStream(nfs.open(src));
try {
byte buf[] = new byte[conf.getInt("io.file.buffer.size", 4096)];
int readBytes = in.read(buf);
while (readBytes >= 0) {
outputStream.write(buf, 0, readBytes);
readBytes = in.read(buf);
}
} finally {
in.close();
}
}
public static void main(String[] args) throws Exception {
SegmentReader segmentReader = new SegmentReader(NutchConf.get());
String usage = "Usage: SegmentReader <segment>";
if (args.length == 0) {
System.err.println(usage);
System.exit(-1);
}
segmentReader.reader(new File(args[0]));
}
}
