[ http://issues.apache.org/jira/browse/NUTCH-51?page=comments#action_63657 ] byron miller commented on NUTCH-51: -----------------------------------
I have index more setup. protocol-http|urlfilter-regex|parse-(text|html|pdf)|index-more|query-(basic|site|url)|clustering-carrot2|ontolog should i have index-(more|basic) for all functionality to work during indexing? Whenever i had this issue, i only removed language-indentifier and i don't believe i have ever had idnex-basic enabled during any segment/indexing phase of this db. > Removing a plugin after fetch but before indexing causes errors > --------------------------------------------------------------- > > Key: NUTCH-51 > URL: http://issues.apache.org/jira/browse/NUTCH-51 > Project: Nutch > Type: Bug > Components: indexer > Environment: Linux/Jdk 1.4.2 > Reporter: byron miller > > I created a segment of 4 million pages with a normal fetch. I created an > index on this segment and it took over 30 hours to complete - when it did i > had errors running a search that the nutchbean couldn't find the title value. > I did some researching and it looks like i should have ran a dedup or > something else. > So to debug i tried to re-index without some of the plugins enabled, and that > is when i get this error: > [EMAIL PROTECTED] [/home2/mozdex/nutch]# bin/nutch segread -fix > segments/20050420222001/ > run with heapsize 1500 > -Xmx1500m > 050424 180125 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml > 050424 180126 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml > 050424 180126 No FS indicated, using default:local > [EMAIL PROTECTED] [/home2/mozdex/nutch]# bin/nutch index > segments/20050420222001 -dir /home/work > run with heapsize 1500 > -Xmx1500m > 050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml > 050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml > 050424 180152 No FS indicated, using default:local > 050424 180152 indexing segment: segments/20050420222001 > 050424 180152 * Opening segment 20050420222001 > 050424 180153 * Indexing segment 20050420222001 > 050424 180153 Plugins: looking in: /home2/mozdex/nutch/build/plugins > 050424 180154 parsing: > /home2/mozdex/nutch/build/plugins/protocol-http/plugin.xml > 050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-html/plugin.xml > 050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-text/plugin.xml > 050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-pdf/plugin.xml > 050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-msword > 050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-ext > 050424 180154 not including: /home2/mozdex/nutch/build/plugins/index-basic > 050424 180154 parsing: > /home2/mozdex/nutch/build/plugins/query-basic/plugin.xml > 050424 180154 not including: /home2/mozdex/nutch/build/plugins/query-more > 050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-site/plugin.xml > 050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-url/plugin.xml > 050424 180154 parsing: > /home2/mozdex/nutch/build/plugins/urlfilter-regex/plugin.xml > 050424 180154 not including: > /home2/mozdex/nutch/build/plugins/urlfilter-prefix > 050424 180154 not including: /home2/mozdex/nutch/build/plugins/creativecommons > 050424 180154 not including: > /home2/mozdex/nutch/build/plugins/language-identifier > 050424 180154 parsing: > /home2/mozdex/nutch/build/plugins/clustering-carrot2/plugin.xml > 050424 180155 parsing: /home2/mozdex/nutch/build/plugins/ontology/plugin.xml > Exception in thread "main" java.lang.ExceptionInInitializerError > at > org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:145) > at org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:254) > Caused by: java.lang.RuntimeException: > org.apache.nutch.indexer.IndexingFilter not found. > at > org.apache.nutch.indexer.IndexingFilters.<clinit>(IndexingFilters.java:36) > ... 2 more > It seems when you generate a fetchlist with the plugin you can't process the > index without the plugin. > If this is true, a friendly error message would be nice and perhaps a utility > to "fix" the segment so it can be processed without the plugin. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
