Removing a plugin after fetch but before indexing causes errors
---------------------------------------------------------------
Key: NUTCH-51
URL: http://issues.apache.org/jira/browse/NUTCH-51
Project: Nutch
Type: Bug
Components: indexer
Environment: Linux/Jdk 1.4.2
Reporter: byron miller
I created a segment of 4 million pages with a normal fetch. I created an index
on this segment and it took over 30 hours to complete - when it did i had
errors running a search that the nutchbean couldn't find the title value.
I did some researching and it looks like i should have ran a dedup or something
else.
So to debug i tried to re-index without some of the plugins enabled, and that
is when i get this error:
[EMAIL PROTECTED] [/home2/mozdex/nutch]# bin/nutch segread -fix
segments/20050420222001/
run with heapsize 1500
-Xmx1500m
050424 180125 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml
050424 180126 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml
050424 180126 No FS indicated, using default:local
[EMAIL PROTECTED] [/home2/mozdex/nutch]# bin/nutch index
segments/20050420222001 -dir /home/work
run with heapsize 1500
-Xmx1500m
050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml
050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml
050424 180152 No FS indicated, using default:local
050424 180152 indexing segment: segments/20050420222001
050424 180152 * Opening segment 20050420222001
050424 180153 * Indexing segment 20050420222001
050424 180153 Plugins: looking in: /home2/mozdex/nutch/build/plugins
050424 180154 parsing:
/home2/mozdex/nutch/build/plugins/protocol-http/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-html/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-text/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-pdf/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-msword
050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-ext
050424 180154 not including: /home2/mozdex/nutch/build/plugins/index-basic
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-basic/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/query-more
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-site/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-url/plugin.xml
050424 180154 parsing:
/home2/mozdex/nutch/build/plugins/urlfilter-regex/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/urlfilter-prefix
050424 180154 not including: /home2/mozdex/nutch/build/plugins/creativecommons
050424 180154 not including:
/home2/mozdex/nutch/build/plugins/language-identifier
050424 180154 parsing:
/home2/mozdex/nutch/build/plugins/clustering-carrot2/plugin.xml
050424 180155 parsing: /home2/mozdex/nutch/build/plugins/ontology/plugin.xml
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:145)
at org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:254)
Caused by: java.lang.RuntimeException: org.apache.nutch.indexer.IndexingFilter
not found.
at
org.apache.nutch.indexer.IndexingFilters.<clinit>(IndexingFilters.java:36)
... 2 more
It seems when you generate a fetchlist with the plugin you can't process the
index without the plugin.
If this is true, a friendly error message would be nice and perhaps a utility
to "fix" the segment so it can be processed without the plugin.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
