[ http://issues.apache.org/jira/browse/NUTCH-242?page=comments#action_12372592 ]
Rod Taylor commented on NUTCH-242: ---------------------------------- "Shouldn't you use the returned value of the filter?" I forgot about URL Normalization (focused on expunging the data only). I suppose normalization rules could change as well. Okay. I will look at Injector.java and essentially run the same steps when urlFiltering is enabled for updateDb. > Add optional -urlFiltering to updatedb > -------------------------------------- > > Key: NUTCH-242 > URL: http://issues.apache.org/jira/browse/NUTCH-242 > Project: Nutch > Type: New Feature > Versions: 0.8-dev > Reporter: Rod Taylor > Attachments: nutch_crawldb_filtering.patch > > Allow filtering the URLs completely out of the database during an updatedb > run. This allows the regex-urlfilter.xml rules to be changed and the > non-matching entries to be expunged from the database. > Merging the functionality with updatedb was done for efficiency reasons since > this eats up CPU time only where a separate job would use IO and CPU time. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
