I'm running a blade server with 16GB of memory and run 5 simultaneous crawls at a time when I'm crawling big sites otherwise, I use the Xmx of 1GB and run 15 crawls at a time. Most of my big crawls seem to run out of memory even with a setting like 3GB. What settings do you use for sites like wikipedia.com?
--- On Thu, 10/30/08, Alexander Aristov <[EMAIL PROTECTED]> wrote: > From: Alexander Aristov <[EMAIL PROTECTED]> > Subject: Re: Xmx settings > To: [email protected], [EMAIL PROTECTED] > Date: Thursday, October 30, 2008, 12:58 AM > Try to inclrease swap size as the message states. Even if > you specify that > java should use 3G of RAM you might run out of memory if > you OS doesn't > support such sizes. > > How many memory do you have on the machine? > > Alex > > 2008/10/29 Alex Basa <[EMAIL PROTECTED]> > > > I'm getting several OutOfMemoryError messages from > large sites like this > > one and gutenberg.com > > > > JAVA_HEAP_MAX=-Xmx3528m > > JAVA_PERM_HEAP=-XX:MaxPermSize=128m > > > > Anyone care to share their settings for doing a crawl > level 1 on a huge > > site? > > > > fetching > http://www.azlyrics.com/lyrics/3oh3/coloradosunrise.html > > fetching > http://www.azlyrics.com/lyrics/unloco/facedown.html > > fetching > http://www.azlyrics.com/lyrics/unfinishedthought/starfighter.html > > # > > # An unexpected error has been detected by Java > Runtime Environment: > > # > > # java.lang.OutOfMemoryError: requested 145320 bytes > for Chunk::new. Out of > > swap > > space? > > # > > # Internal Error (allocation.cpp:218), pid=29949, > tid=12 > > # Error: Chunk::new > > > > > > > > > > > > > -- > Best Regards > Alexander Aristov
