HELP - Error on migrate from 0.8 to 0.9 following the procedures
outlined in the Wiki. This is at the mergesegs step, crawldb convert
went fine - trying to deal with segments now as I'm on a slow connection
and would be painful to re-crawl. Anyone seen this or have any ideas on
how to get past this..?? Nutch on fedora, using Java 1.5 Initial
crawls on .9 seem to work just fine but I haven't tried a mergesegs on
the .9 fetched segments yet....
Also this one -2006-12-15 00:28:22,061 WARN util.NativeCodeLoader -
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable.
I can see how to build native library from Hadoop source but where does
it go in Nutch world - we've only got the Hadoop.jar in lib..?? Does
the native library bring any performance boost to the table..??
Std Out:
SegmentMerger: adding crawl/segments.old/20061209202120
SegmentMerger: adding crawl/segments.old/20061212194145
SegmentMerger: using segment data from: content crawl_generate crawl_fetch
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:399)
at
org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:547)
at
org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:595)
From Hadoop log:
2006-12-15 00:08:51,133 INFO segment.SegmentMerger - Merging 29
segments to crawl/converted8/20061215000851
2006-12-15 00:08:51,345 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-0
2006-12-15 00:08:51,350 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-1
2006-12-15 00:08:51,389 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-10
2006-12-15 00:08:51,390 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-11
2006-12-15 00:08:51,391 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-12
2006-12-15 00:08:51,392 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-13
2006-12-15 00:08:51,394 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-14
2006-12-15 00:08:51,395 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-15
2006-12-15 00:08:51,396 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-16
2006-12-15 00:08:51,397 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-17
2006-12-15 00:08:51,400 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-18
2006-12-15 00:08:51,401 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-19
2006-12-15 00:08:51,405 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-2
2006-12-15 00:08:51,407 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-20
2006-12-15 00:08:51,408 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-21
2006-12-15 00:08:51,409 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-22
2006-12-15 00:08:51,442 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-23
2006-12-15 00:08:51,444 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-24
2006-12-15 00:08:51,445 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-25
2006-12-15 00:08:51,453 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-26
2006-12-15 00:08:51,455 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-3
2006-12-15 00:08:51,456 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-4
2006-12-15 00:08:51,457 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-5
2006-12-15 00:08:51,458 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-6
2006-12-15 00:08:51,459 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-7
2006-12-15 00:08:51,500 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-8
2006-12-15 00:08:51,501 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061208184550-9
2006-12-15 00:08:51,502 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061209202120
2006-12-15 00:08:51,503 INFO segment.SegmentMerger - SegmentMerger:
adding crawl/segments.old/20061212194145
2006-12-15 00:08:51,505 INFO segment.SegmentMerger - SegmentMerger:
using segment data from: content crawl_generate crawl_fetch
2006-12-15 00:28:22,061 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2006-12-15 00:52:43,895 WARN mapred.LocalJobRunner - job_dokmpz
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocalFileSystem.reportChecksumFailure(LocalFileSystem.java:380)
at
org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:136)
at
org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:114)
at
org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.read(DataInputStream.java:80)
at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:200)
at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:192)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:75)
at
org.apache.hadoop.fs.LocalFileSystem.renameRaw(LocalFileSystem.java:212)
at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:373)
at
org.apache.hadoop.mapred.PhasedFileSystem.commit(PhasedFileSystem.java:181)
at
org.apache.hadoop.mapred.PhasedFileSystem.commit(PhasedFileSystem.java:211)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:315)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)
--
rp
