[
https://issues.apache.org/jira/browse/NUTCH-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doğacan Güney updated NUTCH-505:
--------------------------------
Attachment: NUTCH-505_draft.patch
Initial draft patch.
* Uses UrlValidator class from apache commons validator.
* ParseOutputFormat first checks if an outlink is valid. If it is, then it runs
normalizers and urlfilters on url.
This patch is tested very lightly, so it probably doesn't work great yet.
Comments, reviews, suggestions are welcome.
> Outlink urls should be validated
> --------------------------------
>
> Key: NUTCH-505
> URL: https://issues.apache.org/jira/browse/NUTCH-505
> Project: Nutch
> Issue Type: Improvement
> Reporter: Doğacan Güney
> Priority: Minor
> Attachments: NUTCH-505_draft.patch
>
>
> See discussion here:
> http://www.nabble.com/fetching-http%3A--www.variety.com-%3C-div%3E%3C-a%3E-tf3961692.html
> Parse plugins may extract garbage urls from pages. We need a url validation
> system that tests these urls and filters out garbage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
