[
https://issues.apache.org/jira/browse/NUTCH-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903096#comment-17903096
]
ASF GitHub Bot commented on NUTCH-3087:
---------------------------------------
sebastian-nagel opened a new pull request, #845:
URL: https://github.com/apache/nutch/pull/845
…ght require it
- strip the userinfo from the authority only for HTTP and HTTPS
Thanks for your contribution to [Apache Nutch](https://nutch.apache.org/)!
Your help is appreciated!
Before opening the pull request, please verify that
* there is an open issue on the [Nutch issue
tracker](https://issues.apache.org/jira/projects/NUTCH) which describes the
problem or the improvement. We cannot accept pull requests without an issue
because the change wouldn't be listed in the release notes.
* the issue ID (`NUTCH-XXXX`)
- is referenced in the title of the pull request
- and placed in front of your commit messages surrounded by square
brackets (`[NUTCH-XXXX] Issue or pull request title`)
* commits are squashed into a single one (or few commits for larger changes)
* Java source code follows [Nutch Eclipse Code Formatting
rules](https://github.com/apache/nutch/blob/master/eclipse-codeformat.xml)
* Nutch is successfully built and unit tests pass by running `ant clean
runtime test`
* there should be no conflicts when merging the pull request branch into the
*recent* master branch. If there are conflicts, please try to rebase the pull
request branch on top of a freshly pulled master branch.
* if new dependencies are added,
- are these dependencies licensed in a way that is compatible for
inclusion under [ASF
2.0](https://www.apache.org/legal/resolved.html#category-a)?
- are `LICENSE-binary` and `NOTICE-binary` updated accordingly?
We will be able to faster integrate your pull request if these conditions
are met. If you have any questions how to fix your problem or about using Nutch
in general, please sign up for the [Nutch mailing
list](https://nutch.apache.org/mailing_lists.html). Thanks!
> Nutch crawling inconsistent on URLs with userinfo
> -------------------------------------------------
>
> Key: NUTCH-3087
> URL: https://issues.apache.org/jira/browse/NUTCH-3087
> Project: Nutch
> Issue Type: Bug
> Components: urlnormalizer
> Affects Versions: 1.21
> Reporter: Hiran Chaudhuri
> Priority: Major
> Fix For: 1.21
>
>
> I am trying to scan the URL
> smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/
> Note the userinfo 'hiran', which is used for authentication on the server.
> (The smb plugin pulls credentials from another configuration file, but this
> is irrelevant here).
> The URL is fetched, parsed, updated in the crawldb and sent to the indexer.
> So far so good. But the outlinks that are detected are of different quality:
> some have the userinfo preserved, some are missing that information.
> Dumping the segment I can see the below data. Note that some of the outlinks
> start with smb://hi...@nas.fritz.box, while others start with
> smb://nas.fritz.box. The impact is that on the next fetch run authentication
> information is missing and the URLs cannot be fetched further.
>
> {code:java}
> Recno:: 0
> URL:: smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/
> CrawlDatum::
> Version: 7
> Status: 1 (db_unfetched)
> Fetch time: Tue Oct 29 22:56:58 CET 2024
> Modified time: Thu Jan 01 01:00:00 CET 1970
> Retries since fetch: 0
> Retry interval: 86400 seconds (1 days)
> Score: 1.0
> Signature: null
> Metadata:
> _ngt_=1730239026566
> Content::
> Version: -1
> url: smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/
> base: smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/.
> contentType: text/html
> metadata: nutch.segment.name=20241029225708 _fst_=33 nutch.crawl.score=1.0
> Content:
> <html><head><title>Index of
> /Documents/Hiran/Monitoring/</title></head><body><h1>Index of
> /Documents/Hiran/Monitoring/</h1><pre><a href=".svn/">.svn/ Tue Oct 24
> 13:32:32 CEST 2017</a>
> <a href="architektur.dia">architektur.dia Mon Feb 22 21:30:33 CET 2010</a>
> <a href="architektur.dia%7E">architektur.dia~ Mon Feb 22 21:20:42 CET
> 2010</a>
> <a href="architektur.png">architektur.png Mon Feb 22 21:34:27 CET 2010</a>
> <a href="deployment.dia">deployment.dia Mon Feb 22 22:56:15 CET 2010</a>
> <a href="deployment.dia%7E">deployment.dia~ Mon Feb 22 22:51:21 CET
> 2010</a>
> <a href="deployment.png">deployment.png Mon Feb 22 23:00:34 CET 2010</a>
> <a href="Monitoring+strategy.odt">Monitoring strategy.odt Fri Aug 01
> 13:38:04 CEST 2014</a>
> </pre></body></html>
> ParseData::
> Version: 5
> Status: success(1,0)
> Title: Index of /Documents/Hiran/Monitoring/
> Outlinks: 5
> outlink: toUrl:
> smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/architektur.dia anchor:
> architektur.dia Mon Feb 22 21:30:33 CET 2010
> outlink: toUrl:
> smb://nas.fritz.box/Documents/Hiran/Monitoring/architektur.dia~ anchor:
> architektur.dia~ Mon Feb 22 21:20:42 CET 2010
> outlink: toUrl:
> smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/deployment.dia anchor:
> deployment.dia Mon Feb 22 22:56:15 CET 2010
> outlink: toUrl:
> smb://nas.fritz.box/Documents/Hiran/Monitoring/deployment.dia~ anchor:
> deployment.dia~ Mon Feb 22 22:51:21 CET 2010
> outlink: toUrl:
> smb://hi...@nas.fritz.box/Documents/Hiran/Monitoring/Monitoring+strategy.odt
> anchor: Monitoring strategy.odt Fri Aug 01 13:38:04 CEST 2014
> Content Metadata:
> nutch.segment.name = 20241029225708
> nutch.content.digest = a794c6675cb2f9e460e7771060ed2dfc
> _fst_ = 33
> nutch.crawl.score = 1.0
> Parse Metadata:
> CharEncodingForConversion = windows-1252
> OriginalCharEncoding = windows-1252
> language = en
> CrawlDatum::
> Version: 7
> Status: 65 (signature)
> Fetch time: Tue Oct 29 22:57:25 CET 2024
> Modified time: Thu Jan 01 01:00:00 CET 1970
> Retries since fetch: 0
> Retry interval: 0 seconds (0 days)
> Score: 0.0
> Signature: a794c6675cb2f9e460e7771060ed2dfc
> Metadata:
>
> CrawlDatum::
> Version: 7
> Status: 33 (fetch_success)
> Fetch time: Tue Oct 29 22:57:17 CET 2024
> Modified time: Thu Jan 01 01:00:00 CET 1970
> Retries since fetch: 0
> Retry interval: 86400 seconds (1 days)
> Score: 1.0
> Signature: null
> Metadata:
> _ngt_=1730239026566
> _pst_=success(1), lastModified=0
> Content-Type=text/html
> ParseText::
> Index of /Documents/Hiran/Monitoring/
> Index of /Documents/Hiran/Monitoring/
> .svn/ Tue Oct 24 13:32:32 CEST 2017
> architektur.dia Mon Feb 22 21:30:33 CET 2010
> architektur.dia~ Mon Feb 22 21:20:42 CET 2010
> architektur.png Mon Feb 22 21:34:27 CET 2010
> deployment.dia Mon Feb 22 22:56:15 CET 2010
> deployment.dia~ Mon Feb 22 22:51:21 CET 2010
> deployment.png Mon Feb 22 23:00:34 CET 2010
> Monitoring strategy.odt Fri Aug 01 13:38:04 CEST 2014
> {code}
>
> Addendum: It is ok to have only 5 outlinks from a document with 8 anchors.
> The .svn and the two .png links are ignored. My regex-urlfilter.txt looks
> like this:
> {code:java}
> # skip file: ftp: and mailto: urls
> -^(?:file|ftp|mailto):
> # skip image and other suffixes we can't yet parse
> # for a more extensive coverage use the urlfilter-suffix plugin
> -(?i)\.(?:gif|jpg|png|ico|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$
> # skip URLs containing certain characters as probable queries, etc.
> -[?*!=]
> # skip URLs with slash-delimited segment that repeats 3+ times, to break loops
> -.*(/[^/]+)/[^/]+\1/[^/]+\1/
> # skip version control internal files
> -(?i)\.(?:git|svn|cvs)$
> # skip recycle bin URLs
> -(?i)/%23recycle/$
> -/\.svn/
> -/\.git/
> # accept anything else
> +.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
