Re: [Nutch-general] Is the Database toast?

Jason Boss Tue, 21 Sep 2004 15:40:50 -0700

Thanks for the help Doug.

This is what I am getting after adding that little line.


[EMAIL PROTECTED] nutch-nightly]# bin/nutch dedup segments dedup.tmp
040922 062233 Clearing old deletions in segments/20040829092114/index
040922 062234 Clearing old deletions in segments/20040829122947/index
040922 062234 Clearing old deletions in segments/20040829124357/index
040922 062234 Clearing old deletions in segments/20040829130541/index
040922 062234 Clearing old deletions in segments/20040829212107/index
040922 062234 Clearing old deletions in segments/20040829225928/index
040922 062234 Clearing old deletions in segments/20040830042947/index
040922 062234 Clearing old deletions in segments/20040830043001/index
040922 062234 Clearing old deletions in segments/20040830065943/index
040922 062234 Clearing old deletions in segments/20040830111830/index
040922 062237 Reading url hashes...
040922 062237 loading file:/root/nutch-nightly/conf/nutch-default.xml
040922 062238 loading file:/root/nutch-nightly/conf/nutch-site.xml
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index:
12243, Size: 12
  at java.util.ArrayList.RangeCheck(ArrayList.java:507)
  at java.util.ArrayList.get(ArrayList.java:324)
  at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:155)
  at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:66)
  at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:237)
  at
net.nutch.indexer.DeleteDuplicates.computeHashes(DeleteDuplicates.java:182)
  at
net.nutch.indexer.DeleteDuplicates.deleteUrlDuplicates(DeleteDuplicates.java
:149)
  at net.nutch.indexer.DeleteDuplicates.main(DeleteDuplicates.java:264)
[EMAIL PROTECTED] nutch-nightly]#

Something very strange is going on.  I posted this in the dev group, but I
think it has to do with everything.  On this same box, I am trying to update
the db and get this error:

Å°)á»±ï¿½)Õ¢

ï¿½ï¿½>ï¿½>!ï¿½>ï¿½ï¿½0.http://www.angelfire.com/ga/Jannat/Shadow.html.http://www.angelf
ire.com/ga/Jannat/Shadow.htmlï¿½Ñ¦Ö½ï¿½ï¿½ï¿½

ï¿½ï°³9R>p>19ï¿½ï¿½/-http://www.angelfire.com/ga/JayR>ï¿½z>ï¿½m#!http://www.angelfire
.com/ga/KSBA/!http://www.angelfire.com/ga/KSBA/ï¿½}ï¿½
ï¿½Â©ï¿½ï¿½Ä½Ï£ï¿½î¸µ9ï¿½>&Ç®>ï¿½}+)http://www.angelfire.com/ga/KingGhidorah/)http://www
.angelfire.com/ga/KingGhidorah/Ê©oÃ±Ó»é£¬yï¿½lï¿½>ï¿½>ï¿½ï¿½53http://www.angelfire.co
m/ga/LBAEß¯Å¹Û¾Û¶NWà¬¸ï¿½ï¿½>ï¿½>ï¿½ï¿½20http://www.angelfire.com/ga/Link64/Z64Enter.
html0http://www.angelfire.com/ga/Link64/Z64Enter.htmlï¿½Û¢xï¿½`\ï¿½é¬«(ï¿½Ê©>ï¿½>,*ht
tp://www.angelfire.com/ga/MAP/bogey.html*http://www.angelfire.com/ga/MAP/bog
ey.htmlï¿½Ô§ï¿½5ï¿½LÄ¿ï¿½ï¿½>ï¿½>ï¿½{*(http://www.angelfire.com/ga/MattSteiner/(http://
www.angelfire.com/ga/MattSteiner/K^Ü»Tr.Çªk85Ù¼Lï¿½ï¿½
ï¿½Ûª>ï¿½>ï¿½o$"http://www.angelfire.com/ga/RRCOC/"http://www.angelfire.com/ga/RR
COC/Z!JjË´]VÙ³%ï¿½ï¿½hï¿½>[EMAIL PROTECTED];http://www.angelfire.com/ga/SGRLibraries/HahiraL
ibrary.html;http://www.angelfire.com/ga/SGRLibraries/HahiraLibrary.htmlNä¼¢^
PÍ§k×²ï¿½ï¿½î¥¡ï¿½>ï¿½>ï¿½ï¿½86http://www.angelfire.com/ga/SGsÞ¼'t^Ü»ï¿½s/Lakeland.html6ht
tp://www.angelfire.com/ga/SGRLibraries/Lakeland.htmlï¿½
ï¿½î¥¡ï¿½>ï¿½>ï¿½ï¿½97http://www.angelfire.com/ga/SGRLibraries/Southside.html7http://
www.angelfire.com/ga/SGRLibraries/Southside.htmlï¿½Ñ£ï¿½r-ï¿½~ï¿½É½î¥¡ï¿½>ï¿½>ï¿½ï¿½;9http:/
/www.angelfire.com/ga/SGRLibraries/Statenville.html9http://www.angelfire.com
/ga/SGRLibraries/Statenville.html6Þ¬×¼ï¿½Ì¯;ï¿½ï¿½î¥¡ï¿½>ï¿½>ï¿½{*(http://www.angelfire
.com/ga/SYKOTEKNEON/(http://www.angelfire.com/ga/SYKOTEKNEON/Ê·bï¿½Ó¤zã¾¨ï¿½iï¿½
ï¿½ï¿½>ï¿½>,*http://wwï¿½qï¿½gelfire.com/ga/SkerriesHarps/*httpï¿½ï¿½>ï¿½>ï¿½s&$http://www.a
ngelfire.com/ga/Solomon/$http://www.angelfire.com/ga/Solomon/ï¿½Nï¿½Ò¶u=sï¿½ï¿½ï¿½ï¿½>
ï¿½>ï¿½ï¿½20http://www.angelfire.com/ga/Starcheer/index.html0http://www.angelfire.
com/ga/Starcheer/index.htmlï¿½hu&Ö¶Ã¥ï¿½PVï¿½ï¿½
>ï¿½>ï¿½ï¿½20http://www.angelfire.com/ga/TauBetaSigmaZetaTau/0http://www.angelfire
.com/ga/TauBetaSigmaZetaTau/vï¿½Gï¿½ï¿½vs!|ï¿½ï¿½ï¿½>ï¿½>ï¿½s&$http://www.angelfire.com/
ga/TheCat0/$http://www.angelfire.com/ga/TheCat0/Î¾ï¿½ï¿½ï¿½Ev\;
ï¿½

ï¿½D>ï¿½ï¿½ï¿½0.http://www.angelfire.com/ga/ThirdAgeTreasures/.http://www.angelfire
.com/ga/ThirdAgeTreasures/
RV
gÇ¢Cï¿½×¥ï¿½ï¿½>ï¿½>ï¿½o$"http://www.angelfire.com/ga/Verma/"http://www.angelfire.com
/ga/Verma/ÓUÉ«ï¿½Yè²«MÍ²ï¿½î¥¦Ï¾>ï¿½>ï¿½ï¿½31http://www.angelfire.com/ga/ZetaTalpha/i
ndex.html1http://www.angelfire.com/ga/ZetaTalpha/index.htmlÍ¤Jsï¿½ï¿½ï¿½ß²ï¿½ï¿½ï¿½>ï¿½>
ï¿½ï¿½64http://www.angelfire.com/ga/ZoonPolitikon/index.html4http://www.angelfir
e.com/ga/ZoonPolitikon/index.htmlÃ²ï¿½

è¶¥fhÐµÛ¡ï¿½ï¿½>ãµ¾ï¿½ï¿½0.http://www.angelfire.com/ga/achamtb/clubs.html.http://ww
w.angelfire.com/ga/achamtb/clubs.htmlï¿½
Ã¼
]Ï½Úªï¿½r3}ï¿½cï¿½>"ìº¾ï¿½ï¿½31http://www.angelfire.com/ga/achamtb/downhill.html1http
://www.angelfire.com/ga/achamtb/downhill.htmlI7É¿cÛ¸ï¿½nï¿½cï¿½>"ìº¾ï¿½ï¿½.,http://www
.angelfire.com/ga/achamtb/faq.html,http://www.angelfire.com/ga/achamtb/faq.h
tml'å©©ï¿½ï¿½ï¿½txï¿½ï¿½"Ùºï¿½ï¿½cï¿½>"ìº¾ï¿½ï¿½53http://www.angelfire.com/ga/achamtb/new_photos
.html3http://www.angelfire.com/ga/achamtb/new_photos.htmlÒ¿ï¿½

ï¿½ ï¿½cÇ¼#ï¿½cï¿½>"ìº¾ï¿½ï¿½31http://www.angelfire.com/ga/achamtb/overview.html1http://
www.angelfire.com/ga/achamtb/overview.htmlÓ¤tÕ¯]
,`ï¿½ï¿½cï¿½>"ìº¾ï¿½ï¿½31http://www.angelfire.com/ga/achamtb/pictures.html1http://www.
angelfire.com/ga/achamtb/pictures.htmleZ{ï¿½ï¿½}ï¿½ï¿½oï¿½cï¿½>"ìº¾ï¿½ï¿½31http://www.angel
fire.com/ga/achamtb/survival.html1http://www.angelfire.com/ga/achamtb/surviv
al.htmlï¿½Õ®
ï¿½Öº=ï¿½cï¿½>"ìº¾ï¿½ï¿½1/http://www.angelfire.com/ga/achamtb/trails.html/http://www.a
ngelfire.com/ga/achamtb/trails.htmlï¿½ã¥½W{j;g(Ó¡bï¿½ï¿½cï¿½>"ìº¾ï¿½s&$http://www.ange
lfire.com/ga/aeontrix$http://www.angelfire.com/ga/aeontrixr)fä¨¹!?lî¥¨ï¿½>ï¿½>ï¿½ï¿½
[EMAIL PROTECTED]://www.angelfire.com/ga/angelhugspage/[EMAIL PROTECTED]://ww
w.angelfire.com/ga/angelhugspage/hugsforsinglemoms.htmlNOï¿½
ï¿½$ï¿½ï¿½ï¿½ï¿½ï¿½
ï¿½ï¿½ï¿½^U>)C>ï¿½ï¿½53http://www.angelfire.com/ga/batwentyone/Attack.html3http://www
.angelfire.com/ga/batwentyone/Attack.htmlÜ¯ï¿½
Ñºc&Å¤Ò³ï¿½ï¿½^U>
<>ï¿½ï¿½42http://www.angelfire.com/ga/batwentyone/Bases.html2http://www.angelfir
e.com/ga/batwentyone/Bases.htmlË¨J/ï¿½È£$c%ï¿½ï¿½^U>
<>ï¿½ï¿½64http://www.angelfire.com/ga/batwentyone/Schools.html4http://www.angelf
ire.com/ga/batwentyone/Schools.htmlØ¦Aï¿½3fË»Vï¿½Ðµï¿½^U>
<>ï¿½ï¿½64http://www.angelfire.com/ga/batwentyone/Spe.Msn.html4http://www.angelf
ire.com/ga/batwentyone/Spe.Msn.html
ï¿½
 7Ö¬ï¿½ï¿½?}ï¿½ï¿½^U>
<>ï¿½ï¿½75http://www.angelfire.com/ga/batwenEï¿½
J8ï¿½^U>ners.html5http://www.angelfire<>ï¿½ï¿½75http://www.angelfire.com/ga/batwe
ntyone/figthers.html5http://www.angelfire.com/ga/batwentyone/figthers.htmlï¿½
ï¿½
ï¿½:é£»Ó³ï¿½ï¿½^U>
<>ï¿½ï¿½64http://www.angelfire.com/ga/batwentyone/gallery.html4http://www.angelf
ire.com/ga/batwentyone/gallery.htmlï¿½Ú¹ {
/_d}×¶ï¿½^U>
<>ï¿½ï¿½75http://www.angelfire.com/ga/batwentyone/thankyou.html5http://www.angel
fire.com/ga/batwentyone/thankyou.htmlÏ¬Ï©
fÍ¤Ã©Æ³Â½_ï¿½^U>
<>ï¿½ï¿½97http://www.angelfire.com/ga/batwentyone/transports.html7http://www.ang
elfire.com/ga/batwentyone/transports.htmlï¿½Uï¿½MnÆ¨Ö¡.ï¿½ï¿½^U>
<>ï¿½ï¿½64http://www.angelfire.com/ga/batwentyone/utility.html4http://www.angelf
ire.com/ga/batwentyone/utility.htmlPÂ¼rÒ«$.D;È£Ò¾ï¿½^U>
<>ï¿½ï¿½42http://www.angelfire.com/ga/batwentyone/valor.html2http://www.angelfir
e.com/ga/batwentyone/valor.htmlÒ¥ï¿½ï¿½ï¿½ï¿½xï¿½ï¿½^U>
<>ï¿½q%#http://www.angelfire.com/ga/bazuka/#http://www.angelfire.com/ga/bazuka
//#Êµï¿½UÐ«ï¿½ï¿½ï¿½ï¿½>$zï¿½ï¿½k&î¥¨ï¿½>ï¿½>ï¿½w read 26994 bytes, should read
779577707lfire.com/ga/b^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c^[[?6
c^[[?6cPuTTY^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c^[[?6c   at
net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:192)
  at net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:205)
  at net.nutch.io.MapFile$Reader.next(MapFile.java:300)
  at
net.nutch.db.WebDBWriter$PagesByURLProcessor.mergeEdits(WebDBWriter.java:623
)
  at net.nutch.db.WebDBWriter$CloseProcessor.closeDown(WebDBWriter.java:543)
  at net.nutch.db.WebDBWriter.close(WebDBWriter.java:1534)
  at net.nutch.tools.UpdateDatabaseTool.close(UpdateDatabaseTool.java:297)
  at net.nutch.tools.UpdateDatabaseTool.main(UpdateDatabaseTool.java:342)
You have new mail in /var/spool/mail/root
[EMAIL PROTECTED] nutch-nightly]#

Do I start the database over or is this some segment that is corrupting
everything?

Thanks,

Jason

----- Original Message ----- 
From: "Doug Cutting" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, September 21, 2004 10:43 AM
Subject: Re: [Nutch-general] Is the Database toast?


> It actually looks more like a segment's index is trashed.
>
> Try using the following patch to identify the troubled segment, then
> re-index it.
>
> Doug
>



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Nutch-general mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-general

Re: [Nutch-general] Is the Database toast?

Reply via email to