You could incorporate memcached or OSCache on frontend servers to try and cache as much as possible.
-byron --- Mike Alulin <[EMAIL PROTECTED]> wrote: > Thank you for your response. After reading this > document > http://wiki.apache.org/nutch/HardwareRequirements I > thought Nutch has its own cache. :o( > > RAMDirectory will help only in case one has enough > memory to store whole 100% of index files. It would > be better to have something like SQL Server has when > it caches in RAM only recently used pages. Although > I do not know much about the Nutch architecture and > have no idea whether any application specific > caching can be implemented. > > > Byron Miller <[EMAIL PROTECTED]> wrote: > Using that large of a JVM won't do anything but > slow > you down. I would warm up your index by throwing > queries at it to get the blocks cached on an OS > level > or work on implementing RAMDirectory instead of > FSDirectory to store your index in ram if you have > the > resources to do so. > > > > --- Mike Alulin wrote: > > > How to make Nutch use more RAM? I've added -Xmx3g > to > > the Tomcat command line, but Nutch does not seems > to > > use this memory. It does not go above 170MB. No > > matter whether I reserve 300MB or 3GB of RAM for > > JVM, search works with similar speed and requires > > similar amount of HDD reads. My test DB includes > > about 3M pages that according to the Nutch > hardware > > requirements article > > > (http://wiki.apache.org/nutch/HardwareRequirements) > > should take up to 6GB of RAM. How can I load more > > indexes to RAM? > > > > Test system info: > > OS: Windows Server 2K3 64bit > > RAM: 4GB > > JDK 1.5 64 bit > > Nutch DB ~3M pages > > > > > > > > --------------------------------- > > Yahoo! Photos > > Ring in the New Year with Photo Calendars. Add > > photos, events, holidays, whatever. > > > > > > --------------------------------- > Yahoo! Photos > Got holiday prints? See all the ways to get quality > prints in your hands ASAP. ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
