Hi Doug,
Just got done using the new nutch-nightly build. I FINALLY got through 4
segments, but still have issues with segments that didn't "finish" normally.
Meaning for some reason either the crawler died or I had to do a CTRL C on a
crawl. How do you merge segments that have the bad EOF at the end?
This is what my m3 directory looks like.
[EMAIL PROTECTED] nutch-nightly]# du -h segments/m3
70M segments/m3/20040829124357/fetchlist
84M segments/m3/20040829124357/fetcher
2.2G segments/m3/20040829124357/content
803M segments/m3/20040829124357/parse_text
1.5G segments/m3/20040829124357/parse_data
1.1G segments/m3/20040829124357/index
5.6G segments/m3/20040829124357
2.0G segments/m3/20040929022352/fetchlist
390M segments/m3/20040929022352/fetcher
7.1G segments/m3/20040929022352/content
2.5G segments/m3/20040929022352/parse_text
6.5G segments/m3/20040929022352/parse_data
3.1G segments/m3/20040929022352/index
22G segments/m3/20040929022352
19M segments/m3/20040829092114/fetchlist
21M segments/m3/20040829092114/fetcher
244M segments/m3/20040829092114/content
82M segments/m3/20040829092114/parse_text
143M segments/m3/20040829092114/parse_data
119M segments/m3/20040829092114/index
625M segments/m3/20040829092114
28G segments/m3
[EMAIL PROTECTED] nutch-nightly]#
When trying to merge these segments it trips out with this error.
[EMAIL PROTECTED] nutch-nightly]# bin/nutch mergesegs segments/m3/ -o
segments/20041004000b -cm
041005 041415 loading file:/root/nutch-nightly/conf/nutch-default.xml
041005 041415 loading file:/root/nutch-nightly/conf/nutch-site.xml
041005 041422 Segment 20040829124357: 550000 entries.
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:268)
at
net.nutch.io.NFSRandomAccessFile.readFully(BufferedRandomAccessFile.java:288
)
at
net.nutch.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:42)
at net.nutch.io.DataOutputBuffer.write(DataOutputBuffer.java:76)
at net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:236)
at net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:191)
at net.nutch.io.SequenceFile$Reader.next(SequenceFile.java:213)
at net.nutch.io.MapFile$Reader.next(MapFile.java:299)
at net.nutch.io.ArrayFile$Reader.next(ArrayFile.java:47)
at
net.nutch.tools.SegmentMergeTool$SegmentReader.<init>(SegmentMergeTool.java:
86)
at net.nutch.tools.SegmentMergeTool.run(SegmentMergeTool.java:148)
at net.nutch.tools.SegmentMergeTool.main(SegmentMergeTool.java:322)
041005 041452 SEVERE null
[EMAIL PROTECTED] nutch-nightly]#
Any help would be awesome on this!
Thanks,
Jason
----- Original Message -----
From: "Doug Cutting" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Monday, October 04, 2004 1:37 PM
Subject: Re: [Nutch-dev] mergesegs errors
> Have you tried the new nightly that I just built?
>
> Doug
>
> Jason Boss wrote:
> > Doug and Andrzej,
> >
> > What do I need to do to get my local system working? Can I use the new
> > version or do I still need to wait for a revised patch?
> >
> > Thanks,
> >
> > Jason
> >
> > ----- Original Message -----
> > From: "Andrzej Bialecki" <[EMAIL PROTECTED]>
> > To: <[EMAIL PROTECTED]>
> > Sent: Monday, October 04, 2004 11:00 AM
> > Subject: Re: [Nutch-dev] mergesegs errors
> >
> >
> >
> >>Doug Cutting wrote:
> >>
> >>
> >>>Andrzej Bialecki wrote:
> >>>
> >>>
> >>>>Please be patient - I just got back from a trip, and I have a patch
> >>>>for this, but I need to test it first before committing - another
> >>>>couple of hours.
> >>>
> >>>
> >>>Mike just submitted a revision of the NutchFS API & implementation that
> >>>should fix this. Sorry, Andrzej, if this replicates work that you're
> >>>doing! Mike's actively developing the Nutch Distributed File System
> >>>(NDFS), but doesn't always follow the Nutch mailing lists. I spoke to
> >>>him on the phone Friday, urging him to look at this bug.
> >>
> >>I just realized that there were some incoming changes... or rather CVS
> >>did it for me ;-) No problem, I'll re-create the patch.
> >>
> >>
> >>>Mike said he has over 5 terabytes of data in an NDFS on a cluster at
the
> >>>University of Washington. So it really does work!
> >>
> >>I don't doubt it - however, there are border cases where the
> >>LocalFileSystem cannot work due to java.io.File limitations (which IMHO
> >>should be considered bugs in Sun implementation of the
> >>platform-independent FS interface).
> >>
> >>--
> >>Best regards,
> >>Andrzej Bialecki
> >>
> >>-------------------------------------------------
> >>Software Architect, System Integration Specialist
> >>CEN/ISSS EC Workshop, ECIMF project chair
> >>EU FP6 E-Commerce Expert/Evaluator
> >>-------------------------------------------------
> >>FreeBSD developer (http://www.freebsd.org)
> >>
> >>
> >>
> >>-------------------------------------------------------
> >>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> >>Use IT products in your business? Tell us what you think of them. Give
us
> >>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
> >
> > more
> >
> >>http://productguide.itmanagersjournal.com/guidepromo.tmpl
> >>_______________________________________________
> >>Nutch-developers mailing list
> >>[EMAIL PROTECTED]
> >>https://lists.sourceforge.net/lists/listinfo/nutch-developers
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give
us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Nutch-developers mailing list
> > [EMAIL PROTECTED]
> > https://lists.sourceforge.net/lists/listinfo/nutch-developers
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Nutch-developers mailing list
> [EMAIL PROTECTED]
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
