Usually it is not logging (80M wouldn't really be that much, we have
some logs that do ~G a day). Not having enough memory set in either the
hadoop servers or the child opts could cause lots of swapping which
could slow it down. Also having to many active tasks at once eating up
CPU could also do it. If you added custom parsing plugins it may be
that either their processing takes a lot of cpu or that they eat up
memory eventually causing swapping.
From what you describe my first thought would be memory leak or url
with a lot of content.
Dennis
POIRIER David wrote:
Dennis,
Thank you. The parse was over the second I received your mail. More than 24
hours... I wonder if this is because I added two more parser plugins, plugins
writing a lot to the hadoop.log file. Actually this file get usually bigger
then 80MO every day. Can that cause performance problems?
I also have performance problems when crawling a fairly big source (+30 000
urls). The fetching of the first 10 000 urls goes fairly rapidly, but then it
takes forever for the last 20 000 urls. Can it be my parser plugins? The log
file? Not enough fetching threads?
If you have any idea.
Thank you,
David
-----------------------------------------
David Poirier
E-business Consultant - Software Engineer
Direct: +41 (0)22 596 10 35
Cross Systems - Groupe Micropole Univers
Route des Acacias 45 B
1227 Carouge / Genève
Tél: +41 (0)22 308 48 60
Fax: +41 (0)22 308 48 68
-----Original Message-----
From: Dennis Kubes [mailto:[EMAIL PROTECTED]
Sent: mercredi, 4. juin 2008 16:27
To: [email protected]
Subject: Re: Can I parse more than once fetched segments?
You can if you remove the crawl_parse, parse_text, and parse_data
directories and then run the parse command. Don't know why it would be
taking so long.
Dennis
POIRIER David wrote:
Hello,
Can I parse more than once fetched segments without having to fetch
everything again?
When I first tried to use the "./bin nutch parse
./path/to/an/already/parsed/segment" command I got a java exception
explaining that the segment involved had already be parsed. Indeed the
following subdirectories could be found under the segment directory:
segment/content
segment/crawl_fetch
segment/crawl_generate
segment/crawl_parse
segment/parse_data
segment/parse_text
To try and force the parsing process I renamed the last 3 subdirectories
to something else and re-lunched the "./bin nutch parse" command. It has
been running for more than 24 hours... and it is still not over.
My idea is to afterward recreate an index with the newly parsed segment.
Is this the way to do it? Isn't there a simpler, and maybe quicker, way
to reparsed segments?
Thank you,
David
