Two things to be aware of with fetching:
1) The number of urls from a specific host. I think it is defined by
the generate.max.per.host configuration variable. If this is lower (say
10-50) your fetches will be much faster and will not hang on single
large sites.
2) The max.crawl.delay variable sets the limit at which site will be
ignored if they have a crawl delay in robots.txt that is greater than
the variable. By default this is 30 seconds and is usually not that big
of a problem, especially when there is a limit to the number of pages
per site as in 1. The problem comes when you have 1000 pages from a
single site with a crawl delay of 20 seconds per page or something similar.
Dennis
POIRIER David wrote:
Thank you Dennis.
I just re-lunched a complete crawl of one source (one host), modifying the
fetcher.threads.per.host limit from 1 to 4, expecting this to speed up the
fetch process... and not be blacklisted.
I did check the cpu and memory status before killing the prior fetching of the
same source, and everything was pretty ok. CPU charge was toping at 45%-50% and
I still had 1GO of RAM left. The java process itself was not eating more then
150MO of RAM. HD access was also good.
I just finished the fetch/parse/index of a filesystem using the protocol-file
plugin and all my usual homemade plugins. The index was ready in about 1h30. I
fetched 55 000 URIs.
I guess the slowing down of my regular internet sources is due to network
latency problems.
Anyway, I'll keep you posted about the performance difference when using 4
threads instead of 1 per host.
David
-----Original Message-----
From: Dennis Kubes [mailto:[EMAIL PROTECTED]
Sent: mercredi, 4. juin 2008 18:18
To: [email protected]
Subject: Re: Can I parse more than once fetched segments?
Usually it is not logging (80M wouldn't really be that much, we have
some logs that do ~G a day). Not having enough memory set in either the
hadoop servers or the child opts could cause lots of swapping which
could slow it down. Also having to many active tasks at once eating up
CPU could also do it. If you added custom parsing plugins it may be
that either their processing takes a lot of cpu or that they eat up
memory eventually causing swapping.
From what you describe my first thought would be memory leak or url
with a lot of content.
Dennis
POIRIER David wrote:
Dennis,
Thank you. The parse was over the second I received your mail. More than 24
hours... I wonder if this is because I added two more parser plugins, plugins
writing a lot to the hadoop.log file. Actually this file get usually bigger
then 80MO every day. Can that cause performance problems?
I also have performance problems when crawling a fairly big source (+30 000
urls). The fetching of the first 10 000 urls goes fairly rapidly, but then it
takes forever for the last 20 000 urls. Can it be my parser plugins? The log
file? Not enough fetching threads?
If you have any idea.
Thank you,
David
-----------------------------------------
David Poirier
E-business Consultant - Software Engineer
Direct: +41 (0)22 596 10 35
Cross Systems - Groupe Micropole Univers
Route des Acacias 45 B
1227 Carouge / Genève
Tél: +41 (0)22 308 48 60
Fax: +41 (0)22 308 48 68
-----Original Message-----
From: Dennis Kubes [mailto:[EMAIL PROTECTED]
Sent: mercredi, 4. juin 2008 16:27
To: [email protected]
Subject: Re: Can I parse more than once fetched segments?
You can if you remove the crawl_parse, parse_text, and parse_data
directories and then run the parse command. Don't know why it would be
taking so long.
Dennis
POIRIER David wrote:
Hello,
Can I parse more than once fetched segments without having to fetch
everything again?
When I first tried to use the "./bin nutch parse
./path/to/an/already/parsed/segment" command I got a java exception
explaining that the segment involved had already be parsed. Indeed the
following subdirectories could be found under the segment directory:
segment/content
segment/crawl_fetch
segment/crawl_generate
segment/crawl_parse
segment/parse_data
segment/parse_text
To try and force the parsing process I renamed the last 3 subdirectories
to something else and re-lunched the "./bin nutch parse" command. It has
been running for more than 24 hours... and it is still not over.
My idea is to afterward recreate an index with the newly parsed segment.
Is this the way to do it? Isn't there a simpler, and maybe quicker, way
to reparsed segments?
Thank you,
David
