Hi Semyon,
> It seems like Nutch takes the anchor name as an URL for the crawling a store
> it in database with
the key equals to name.
if you look into the page HTML you can see that it's the href attribute:
<p><a href="http://#Are there any places to eat onsite during the show?"
target="_self"
title="http://#Are there any places to eat onsite during the show?">Are there
any places to eat
onsite during the show?</a></p>
How are URL filters configured? Normally, a URL
"http://#Are there any places to eat onsite during the show?"
should not make it into the CrawlDb.
Best,
Sebastian
On 11/28/2017 02:17 PM, Semyon Semyonov wrote:
> Hello all,
>
> I have launched a crawling process for 100 websites with external links
> equals to true.
> After several hours, I run the crawlcomplete command with mode equals host.
>
> The crawlcomplete output file contains(apart from the proper host names) the
> following lines.
>
> 1 #Are there any places to eat onsite during the show#Are there any places
> to eat onsite during the show UNFETCHED
> 1 #Are there any points where I can access the internet at the show#Are
> there any points where I can access the internet at the show UNFETCHED
> 1 #Can I register onsite#Can I register onsite UNFETCHED
> 1 #Can children attend the show#Can children attend the show UNFETCHED
> 1 #Can you recommend any site-seeing attractions in Amsterdam#Can you
> recommend any site-seeing attractions in Amsterdam UNFETCHED
> 1 #Do I need a visa#Do I need a visa UNFETCHED
> 1 #How do I get to IBC2018 at the Amsterdam RAI#How do I get to IBC2018 at
> the Amsterdam RAI UNFETCHED
> 1 #Is there anywhere for me to practice my religion#Is there anywhere for
> me to practice my religion UNFETCHED
> 1 #Is there parking#Is there parking UNFETCHED
> 1 #Want to exhibit at IBC2018#Want to exhibit at IBC2018 UNFETCHED
> 1 #What do I have access to at IBC#What do I have access to at IBC
> UNFETCHED
> 1 #What do I need to bring to IBC#What do I need to bring to IBC UNFETCHED
> 1 #What is the IBC Big Screen Experience#What is the IBC Big Screen
> Experience UNFETCHED
> 1 #When and where is IBC#When and where is IBC UNFETCHED
> 1 #Who attends IBC#Who attends IBC UNFETCHED
>
> After googling I found the webpage where it came from:
> https://show.ibc.org/about-ibc/faqs
>
> It seems like Nutch takes the anchor name as an URL for the crawling a store
> it in database with the key equals to name.
>
> For example.
> <a class="anchor" name="Are there any places to eat onsite during the
> show?"></a>
>
> Any suggestion what is it and how to fix it?
> Thanks.
>
> Semyon.
>
