[ http://issues.apache.org/jira/browse/NUTCH-242?page=comments#action_12372592 ]
Rod Taylor commented on NUTCH-242: ---------------------------------- "Shouldn't you use the returned value of the filter?" I forgot about URL Normalization (focused on expunging the data only). I suppose normalization rules could change as well. Okay. I will look at Injector.java and essentially run the same steps when urlFiltering is enabled for updateDb. > Add optional -urlFiltering to updatedb > -------------------------------------- > > Key: NUTCH-242 > URL: http://issues.apache.org/jira/browse/NUTCH-242 > Project: Nutch > Type: New Feature > Versions: 0.8-dev > Reporter: Rod Taylor > Attachments: nutch_crawldb_filtering.patch > > Allow filtering the URLs completely out of the database during an updatedb > run. This allows the regex-urlfilter.xml rules to be changed and the > non-matching entries to be expunged from the database. > Merging the functionality with updatedb was done for efficiency reasons since > this eats up CPU time only where a separate job would use IO and CPU time. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
