From: Dennis Kubes <[EMAIL PROTECTED]>
Subject: Re: Nutch generate and fetch very slow after a few crawls (results)
To: [email protected]
Date: Thursday, November 27, 2008, 2:22 PM
I think you have more than one problem going on here to
create the situation you are facing.
1) The biggest problem I see is the NFS mount. The
generate step is still going to write out intermediate
output and is going to be doing lots of read input. With
NFS it is having to do lots of network roundtrips. Even
when using DFS, data is downloaded to local disk first
before being processed in MapReduce.
2) 512M is low. I don't think that is the biggest
problem but it could definitely be causing some slowdown
because of the amount of data that can be read and processed
at one time.
3) CPU is at 100% and if it stays that way for a long time
there may be other issues going on beside just processing
data. These could be filter regex related or something
else. I don't think this is the main problem though.
Dennis
ML mail wrote:
Dear Dennis
Here a few lines to explain better our current
configuration: Nutch 0.9 is running in an OpenVZ Virtual
Private Server with 512 MB RAM allocated to it and full CPU
usage (a Pentium IV Xeon 2.8 GHz, the machine itself has 1
GB physical RAM). NUTCH_HEAPSIZE in the nutch shell script
is set to 350 MB instead of the original 1000 MB.
Right now it is running the generate step since approx
3 hours for a topN of 40000 pages and usually it takes
between 4 and 5 hours. The command top shows the following:
PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
16268 nutch 18 0 471m 159m 5808 S 100 31.1
171:03.82 java
So it looks like the CPU is the limiting factor in our
case, when I interpret this correctly. This is quite strange
because a Pentium IV 2.8 GHz is not so old...
Regarding the swapping, I can't see any swapping
in the VPS as there is no swap really in OpenVZ, here is the
output of "free -m" during the generate step:
total used free shared
buffers cached
Mem: 512 477 34 0
0 0
-/+ buffers/cache: 477 34
Swap: 0 0 0
Nutch binaries is stored and running from a Linux/MD
software RAID 1 set with two Seagate ATA 7200 rpm harddisks.
The crawl directory and all its data is stored on a 8 TB
Linux RAID 6 NAS mounted via NFS with the following NFS
mount options:
rw,tcp,rsize=8192,wsize=8192
But if I understand correctly during the generate step
of Nutch there is only very low I/O activity because it is
only querying the crawldb, am I correct ?
About the URL filter we use pretty much the default
crawl-urlfilter.txt file except that we added maybe 2-3 more
extensions to skip and changed the "accept hosts"
to only index those ending in .be. You will find our
crawl-urlfilter.txt file at the end of this mail.
So I hope I have provided you enough information, if
you need more just let me know.
Thanks again and best regards
---------- crawl-urlfilter.txt -------
# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP|js)$
# skip URLs containing certain characters as probable
queries, etc.
[EMAIL PROTECTED]
# skip URLs with slash-delimited segment that repeats
3+ times, to break loops
-.*(/.+?)/.*?\1/.*?\1/
# accept hosts in MY.DOMAIN.NAME
+^http://([a-z0-9]*\.).*\.be/
# skip everything else
-.
---------- crawl-urlfilter.txt -------
--- On Wed, 11/26/08, Dennis Kubes
<[EMAIL PROTECTED]> wrote:
From: Dennis Kubes <[EMAIL PROTECTED]>
Subject: Re: Nutch generate and fetch very slow
after a few crawls (results)
To: [email protected]
Date: Wednesday, November 26, 2008, 4:46 AM
Well, generate will still have to go through all
of the
urls, although skipping after 25 per domain should
be really quick. It
really depends on your hardware and any regexes
you may be running in
urlfilters for the generate step. A 2.8Ghz Xeon
with 1G ram should be
pretty quick. 100,000 pages on a core2duo on my
laptop (4G Ram) takes
less than an hour if I remember correctly. What
type of hard drive
(speed) are you using and are you swapping a lot
during generate? It may
be the amount of RAM.
Dennis
ML mail wrote:
Dear Richard and others interested,
Just wanted to post the results of reducing
generate.max.per.host to 25 (instead of -1:
unlimited) as
recommended by Richard.
So actually to summarize, the fetch step has
been
greatly reduced to 1 hour instead of 6 hours (for
topN set
at 25000) but unfortunately the generate step is
still quite
slow and takes around 4 hours (for the same topN
amount).
Is this normal for the generate step to still
be so
slow ? The whole index is only around 170'000
pages big.
Is there maybe also an option in nutch-default.xml
config
file where one can optimize the generate process ?
Best regards
--- On Fri, 11/21/08, Richard Cyganiak
<[EMAIL PROTECTED]> wrote:
From: Richard Cyganiak
<[EMAIL PROTECTED]>
Subject: Re: Nutch generate and fetch very
slow
after a few crawls
To: [email protected]
Date: Friday, November 21, 2008, 2:42 AM
On 21 Nov 2008, at 09:47, ML mail wrote:
What would then be the solution to
this
problem ?
Shall I simply set generate.max.per.host
to
something like 5
? Or is there another way to make run
Nutch at a
good speed
again ?
I spent some time trying different values
for
generate.max.per.host and I found that
this is a
good rule
of thumb:
generate.max.per.host = topN /
numberOfNodes /
1000
Where topN is the size of your segments,
numberOfNodes is
the number of machines in your cluser.
This keeps
the fetch
rate close to maximum.
Check the log of the fetch job -- if the
last few
pages
consist of request to just one or a few
hosts,
then your
value for generate.max.per.host is too
large. You
want to
fetch from many hosts in parallel
throughout the
entire
fetch job. On the other hand, if you set
it too
low, then
you will never make progress on these
large sites.
I fetched the same segment repeatedly to
find out
what
values work best.
Hope that helps,
Richard
Best regards
--- On Thu, 11/20/08, Dennis Kubes
<[EMAIL PROTECTED]> wrote:
From: Dennis Kubes
<[EMAIL PROTECTED]>
Subject: Re: Nutch generate and
fetch very
slow
after a few crawls
To: [email protected]
Date: Thursday, November 20, 2008,
2:40 PM
Off the top of my head I would
guess that
you hit
a patch of
urls all from the same domain and
that
would slow
down
fetching on a single host because
only one
thread
would be
active? The generate.max.per.host
config
variable
can limit
that.
But that is just a guess. What
job is it
slowing
down on?
Yes Nutch will take more time with
more
data, but
that is
too much of a difference.
Dennis
ML mail wrote:
Hello,
I am currently using the
recrawl
script from
the Nutch
Wiki for crawling all websites
from a
specific
small top
level domain and have configured
the
recrawl
script to run
with THREADS=50, DEPTH=4,
TOPN=25000.
Which means
that each
time I run the script 100'000
pages
will get
crawled.
The first time I ran the
script it
took 6
hours for
the whole process with mergesegs,
inverlinks,
index, merge
and so on. The second time it took
just 3
hours
more so 9
hours, then the 4th time 12 hours
but now
the
fourth time it
is actually still running after 22
hours
and
it's only
at the 64'000 page to be
crawled. It
looks
like that it
is especially the fetch step and
the index
step
which are
running much more slowly, the
other steps
look
normal.
So is this actually a normal
behavior
of Nutch
? I
would expect Nutch to be each time
a tiny
little
bit more
slower due to updating an always
growing
database/index/segment but never
so much
slower as
I am
currently experiencing. Especially
when
right now
there are
only 144'915 pages indexed and
the
whole crawl
directly
with everything is only around 2
GB big.
Nutch is running on a quite
good
Pentium 4
Xeon
computer 2.8 GHz with 1 GB RAM and
nothing
else
mutch
running on it, also I didn't
change
much in
the config
of Nutch itself so it's pretty
much
default.
Does anyone have an idea ? I
can
provide more
info if
you desire, just let me know what
you
need.
Many thanks in advance and
best
regards
