OK, many thanks for the clarification. One thing I'm still uncertain of,
is do I need a config line for each specific domain that is in need of
normalizing? For example, if I notice I have 12 different sites with
duplicate pages indexed, all of them www. and non-www versions, do I
need 12 lines of config in this text file?
e.g.
www.apples.com apples.com
www.oranges.com oranges.com
www.pears.com pears.com
www.blackberries.com blackberries.com
www.blueberries.com blueberries.com
www.bananas.com bananas.com
...
and so on.
That is, from a long term maintainability point of view, will I be
continuous adding new lines to this host-urlnormalizer.txt file as and
when I notice another site has content duplicated? Or is there some way
to write a www. => no-www rule to cover any domain Nutch happens to
encounter in the future?
Arthur.
On 2015-07-07 12:59, Markus Jelsma wrote:
Hello, i added an example to the issue, Hope it helps.
# Force all sub domains to non-www.
*.example.com example.com
# Force www sub domain to non-www.
www.example.net example.net
# Force non-www. sub domain to www
example.org www.example.org
-----Original message-----
From:Arthur Yarwood <[email protected]>
Sent: Tuesday 7th July 2015 11:14
To: [email protected]
Subject: Re: Duplicate pages with and without www. prefix being
indexed
Cool, thanks, that looks like it may do the job.
Am I correct in saying I need to define mapping for each specific
domain
in host-urlnormalizer.txt? Or can I add a generic rule to cover any
host?
i.e.
www.example1.org .example1.org
www.example2.org .example2.org
www.example3.org .example3.org
...
or can I do something along the lines of:-
www.* .($1)
Arthur.
On 07/07/2015 09:01, Markus Jelsma wrote:
> You can use the host normalizer for this.
> https://issues.apache.org/jira/browse/NUTCH-1319
>
> -----Original message-----
>> From:Arthur Yarwood <[email protected]>
>> Sent: Tuesday 7th July 2015 0:02
>> To: [email protected]
>> Subject: Duplicate pages with and without www. prefix being indexed
>>
>> I have a Nutch 1.10 and Solr 5.2 setup. Still playing around with it all
>> and quite new to me, but I've noticed for one site I have crawled, I'm
>> getting content indexed twice in Solr, once with the www. domain prefix
>> and once without. E.g.
>>
>> http://www.example.com/somepage.html
>> http://example.com/somepage.html
>>
>> How can I avoid this duplication? At least from being indexed into Solr.
>> And preferable a generic solution that will work with any other site I
>> crawl in the future, some of which may default to www.xyz.com and some
>> xyz.com. i.e. I know I could add a regex-urlfilter for this one domain,
>> but I'd like to avoid this duplication in call instances in arise.
>>
>> Thanks!
>>
>>
>> --
>> Arthur Yarwood
>>
>>
--
Arthur Yarwood
--
Arthur Yarwood
