Hi all,
I am trying to crawl a single website and while it works for one
website, it doesn't for another:
The one that doesn't work is https://www.saturn.de/ (the one that works
with the same configuration is https://www.gamestop.de/)
What I do is using the following commands:
bin/nutch inject crawlDir/crawldb conf/seed.txt (with content
https://www.saturn.de/)
bin/nutch generate crawlDir/crawldb crawlDir/segments -topN 10
segment=`ls -d crawlDir/segments/* | tail -1`
bin/nutch fetch $segment -threads 2
bin/nutch parse $segment -threads 2
bin/nutch updatedb crawlDir/crawldb $segment
after that a bin/nutch readdb -stats says:
root@d12560375098:~/nutch# bin/nutch readdb crawlDir/crawldb/ -stats
CrawlDb statistics start: crawlDir/crawldb/
Statistics for CrawlDb: crawlDir/crawldb/
TOTAL urls: 93
shortest fetch interval: 15 days, 00:00:00
avg fetch interval: 15 days, 00:00:00
longest fetch interval: 15 days, 00:00:00
earliest fetch time: Mon Oct 22 15:59:00 UTC 2018
avg of fetch times: Mon Oct 22 19:51:00 UTC 2018
latest fetch time: Tue Nov 06 15:59:00 UTC 2018
retry 0: 93
score quantile 0.01: 0.00917431153357029
score quantile 0.05: 0.00917431153357029
score quantile 0.1: 0.00917431153357029
score quantile 0.2: 0.00917431153357029
score quantile 0.25: 0.00917431153357029
score quantile 0.3: 0.00917431153357029
score quantile 0.4: 0.00917431153357029
score quantile 0.5: 0.00917431153357029
score quantile 0.6: 0.00917431153357029
score quantile 0.7: 0.00917431153357029
score quantile 0.75: 0.00917431153357029
score quantile 0.8: 0.00917431153357029
score quantile 0.9: 0.01834862306714058
score quantile 0.95: 0.01834862306714058
score quantile 0.99: 0.5922935494221679
min score: 0.00917431153357029
avg score: 0.021505396853211105
max score: 1.0183485746383667
status 1 (db_unfetched): 92
status 2 (db_fetched): 1
okay! So it crawled one page (the one from seed.txt and found 92 links,
right?)
after that I do:
bin/nutch generate crawlDir/crawldb crawlDir/segments -topN 10
segment=`ls -d crawlDir/segments/* | tail -1`
bin/nutch fetch $segment -threads 2
bin/nutch parse $segment -threads 2
bin/nutch updatedb crawlDir/crawldb $segment
The output seems all okay, I see that 10 pages are fetched and parsed
etc. The output of the fetcher e.g. looks like:
root@d12560375098:~/nutch# bin/nutch fetch $segment -threads 2
Fetcher: starting at 2018-10-22 15:59:59
Fetcher: segment: crawlDir/segments/20181022155950
Fetcher: threads: 2
Fetcher: time-out divisor: 2
QueueFeeder finished: total 10 records hit by time limit : 0
FetcherThread 37 Using queue mode : byHost
FetcherThread 37 Using queue mode : byHost
FetcherThread 43 fetching
https://www.saturn.de/de/category/_festplatten-speichermedien-286920.html
(queue crawl delay=5000ms)
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
robots.txt whitelist not configured.
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=9,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=9,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=9,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=9,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=9,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_computer-tablet-235588.html (queue
crawl delay=5000ms)
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=8,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=8,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=8,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=8,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=8,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_pc-241041.html (queue crawl delay=5000ms)
-activeThreads=2, spinWaiting=1, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=7,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_smartphones-tarife-235592.html (queue
crawl delay=5000ms)
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=6,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=6,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=6,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=6,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=6,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_tablets-252064.html (queue crawl
delay=5000ms)
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=5,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=5,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=5,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=5,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=5,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_notebooks-241042.html (queue crawl
delay=5000ms)
-activeThreads=2, spinWaiting=1, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=4,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_apple-322511.html (queue crawl
delay=5000ms)
-activeThreads=2, spinWaiting=1, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=3,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_monitore-241044.html (queue crawl
delay=5000ms)
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/category/_netzwerk-241046.html (queue crawl
delay=5000ms)
-activeThreads=2, spinWaiting=1, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1
FetcherThread 43 fetching
https://www.saturn.de/de/shop/cardinformation.html (queue crawl
delay=5000ms)
FetcherThread 44 has no more work available
FetcherThread 44 -finishing thread FetcherThread, activeThreads=1
FetcherThread 43 has no more work available
FetcherThread 43 -finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0,
fetchQueues.getQueueCount=0
-activeThreads=0
Fetcher: finished at 2018-10-22 16:00:50, elapsed: 00:00:51
parse says:
root@d12560375098:~/nutch# bin/nutch parse $segment -threads 2
ParseSegment: starting at 2018-10-22 16:01:01
ParseSegment: segment: crawlDir/segments/20181022155950
Parsed (89ms):https://www.saturn.de/de/category/_apple-322511.html
Parsed (16ms):https://www.saturn.de/de/category/_computer-tablet-235588.html
Parsed
(14ms):https://www.saturn.de/de/category/_festplatten-speichermedien-286920.html
Parsed (14ms):https://www.saturn.de/de/category/_monitore-241044.html
Parsed (8ms):https://www.saturn.de/de/category/_netzwerk-241046.html
Parsed (12ms):https://www.saturn.de/de/category/_notebooks-241042.html
Parsed (7ms):https://www.saturn.de/de/category/_pc-241041.html
Parsed
(7ms):https://www.saturn.de/de/category/_smartphones-tarife-235592.html
Parsed (6ms):https://www.saturn.de/de/category/_tablets-252064.html
Parsed (8ms):https://www.saturn.de/de/shop/cardinformation.html
ParseSegment: finished at 2018-10-22 16:01:03, elapsed: 00:00:01
but when I do another readdb -stats it says:
Statistics for CrawlDb: crawlDir/crawldb/
TOTAL urls: 101
shortest fetch interval: 15 days, 00:00:00
avg fetch interval: 15 days, 00:00:00
longest fetch interval: 15 days, 00:00:00
earliest fetch time: Mon Oct 22 15:59:00 UTC 2018
avg of fetch times: Wed Oct 24 07:11:00 UTC 2018
latest fetch time: Tue Nov 06 16:00:00 UTC 2018
retry 0: 101
score quantile 0.01: 1.7152949061710387E-4
score quantile 0.05: 1.73100212123245E-4
score quantile 0.1: 0.010499933175742628
score quantile 0.2: 0.010984613560140133
score quantile 0.25: 0.010984613560140133
score quantile 0.3: 0.010984613560140133
score quantile 0.4: 0.010984613560140133
score quantile 0.5: 0.010984613560140133
score quantile 0.6: 0.010984613560140133
score quantile 0.7: 0.010984613560140133
score quantile 0.75: 0.010984613560140133
score quantile 0.8: 0.010984613560140133
score quantile 0.9: 0.021969228982925415
score quantile 0.95: 0.021969228982925415
score quantile 0.99: 0.5157972617447326
min score: 1.6989465802907944E-4
avg score: 0.021709534201291528
max score: 1.0183485746383667
status 1 (db_unfetched): 90
status 2 (db_fetched): 11
CrawlDb statistics: done
As you can see, 10 more pages where fetched (before 1, now after -topN
10 --> 11) as intended but db_unfetched decreased from 92 to 90.. I have
no clue why this happens. How can that be? It seems like all links from
the 10 fetched pages are ignored instead of putting it into the crawldb
as unfetched.
After one more crawling cycle:
bin/nutch generate crawlDir/crawldb crawlDir/segments -topN 10
segment=`ls -d crawlDir/segments/* | tail -1`
bin/nutch fetch $segment -threads 2
bin/nutch parse $segment -threads 2
bin/nutch updatedb crawlDir/crawldb $segment
readdb says:
root@d12560375098:~/nutch# bin/nutch readdb crawlDir/crawldb/ -stats
CrawlDb statistics start: crawlDir/crawldb/
Statistics for CrawlDb: crawlDir/crawldb/
TOTAL urls: 109
shortest fetch interval: 15 days, 00:00:00
avg fetch interval: 15 days, 03:18:09
longest fetch interval: 22 days, 12:00:00
earliest fetch time: Mon Oct 22 15:59:00 UTC 2018
avg of fetch times: Thu Oct 25 23:18:00 UTC 2018
latest fetch time: Wed Nov 14 04:16:00 UTC 2018
retry 0: 109
score quantile 0.01: 0.0
score quantile 0.05: 1.694382077403134E-4
score quantile 0.1: 1.73100212123245E-4
score quantile 0.2: 0.012107410468161106
score quantile 0.25: 0.012107410468161106
score quantile 0.3: 0.012107410468161106
score quantile 0.4: 0.012107410468161106
score quantile 0.5: 0.012107410468161106
score quantile 0.6: 0.012107410468161106
score quantile 0.7: 0.012107410468161106
score quantile 0.75: 0.012107410468161106
score quantile 0.8: 0.012107410468161106
score quantile 0.9: 0.02421482279896736
score quantile 0.95: 0.02421482279896736
score quantile 0.99: 0.4389530326798525
min score: 0.0
avg score: 0.02122468467152447
max score: 1.0183485746383667
status 1 (db_unfetched): 86
status 2 (db_fetched): 19
status 3 (db_gone): 2
status 4 (db_redir_temp): 2
CrawlDb statistics: done
db_unfetched decreased again.. this will continue until db_unfetched
reached 0 and my crawler does nothing anymore.
my nutch-site.xml looks like:
<property>
<name>http.agent.name</name>
<value>ENTER_NAME_HERE</value>
</property>
<property>
<name>db.fetch.interval.default</name>
<value>1296000</value>
</property>
<property>
<name>db.fetch.schedule.adaptive.inc_rate</name>
<value>0.4</value>
<description>If a page is unmodified, its fetchInterval will be
increased by this rate. This value should not
exceed 0.5, otherwise the algorithm becomes unstable.</description>
</property>
<property>
<name>db.fetch.schedule.adaptive.dec_rate</name>
<value>0.2</value>
<description>If a page is modified, its fetchInterval will be
decreased by this rate. This value should not
exceed 0.5, otherwise the algorithm becomes unstable.</description>
</property>
<property>
<name>db.fetch.schedule.adaptive.min_interval</name>
<value>86400.0</value>
<description>Minimum fetchInterval, in seconds.</description>
</property>
<property>
<name>db.fetch.schedule.adaptive.max_interval</name>
<value>31536000.0</value>
<description>Maximum fetchInterval, in seconds (365 days).
NOTE: this is limited by db.fetch.interval.max. Pages with
fetchInterval larger than db.fetch.interval.max
will be fetched anyway.</description>
</property>
<property>
<name>db.fetch.schedule.adaptive.sync_delta</name>
<value>false</value>
<description>If true, try to synchronize with the time of page change.
by shifting the next fetchTime by a fraction (sync_rate) of the difference
between the last modification time, and the last fetch time.</description>
</property>
<property>
<name>db.fetch.schedule.class</name>
<value>org.apache.nutch.crawl.AdaptiveFetchSchedule</value>
<description>The implementation of fetch schedule. DefaultFetchSchedule simply
adds the original fetchInterval to the last fetch time, regardless of
page changes.</description>
</property>
<property>
<name>db.signature.class</name>
<value>org.apache.nutch.crawl.TextProfileSignature</value>
</property>
<property>
<name>db.max.outlinks.per.page</name>
<value>-1</value>
<description>The maximum number of outlinks that we'll process for a page.
If this value is nonnegative (>=0), at most db.max.outlinks.per.page
outlinks
will be processed for a page; otherwise, all outlinks will be processed.
</description>
</property>
<!-- link-navigation --> <property>
<name>db.ignore.internal.links</name>
<value>false</value>
</property>
<property>
<name>db.ignore.external.links</name>
<value>true</value>
</property>
<!-- sitemap properties --> <property>
<name>sitemap.strict.parsing</name>
<value>false</value>
<description>
If true (default) the Sitemap parser rejects URLs not sharing the same
prefix with the sitemap: a sitemap
`http://example.com/catalog/sitemap.xml'
may only contain URLs starting with `http://example.com/catalog/'.
All other URLs are skipped. If false the parser will allow any URLs
contained
in the sitemap.
</description>
</property>
<property>
<name>sitemap.url.filter</name>
<value>false</value>
<description>
Filter URLs from sitemaps.
</description>
</property>
<property>
<name>sitemap.url.normalize</name>
<value>false</value>
<description>
Normalize URLs from sitemaps.
</description>
</property>
<!-- etc --> <property>
<name>http.redirect.max</name>
<value>6</value>
<description>The maximum number of redirects the fetcher will follow when
trying to fetch a page. If set to negative or 0, fetcher won't immediately
follow redirected URLs, instead it will record them for later fetching.
</description>
</property>
<property>
<name>fetcher.verbose</name>
<value>true</value>
<description>If true, fetcher will log more verbosely.</description>
</property>
and my regex-urlfilter.txt only contains:
-(?i)\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|svg|SVG|mp3|MP3|mp4|MP4|pdf|PDF|json|JSON)$
+.
First I had lots of filters in my regex-urlfilter.txt.. but because I
thought that the fetched links are filtered out because of a wrong
filter I commented out nearly everything..but now that I have nothing
left in my regex-urlfilter.txt it still continues to decrease
db_unfetched-counter. So this doesn't seem to be the problem.
I have no more idea why this happens. Does anyone have a hint for me? I
would really really appreciate it! Thanks a lot in advance!
Greetings,
Marco
