Performance problem with widely used keywords
---------------------------------------------
Key: NUTCH-180
URL: http://issues.apache.org/jira/browse/NUTCH-180
Project: Nutch
Type: Wish
Reporter: Mike Alulin
It looks like Nutch is very slow when the search phrase includes a few widely
used keywords. For example "I 1 2 3 4 5 6 7 8 9 0" typed without the quotes to
Yahoo, Google, or MSN is processed in less than a second. Nutch on the other
hand requires much more time for this even on smaller databases. For example
this phrase made objectssearch.com think more than 1 minute although their DB
is much smaller than DBs of the big 3 guys. On my test Nutch DB with only 3M
pages this phrase took a few seconds to process.
Unfortunately I do not know much about search algorithms, but it looks like
Nutch do have some space to improve the search performance. The current
implementation can be easily "killed" by a few search requests like this. Just
a couple of dozen of such requests makes my server with 2 Opterons think for a
minute or two with 100% CPU utilization.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
