Sorry. Yes, I need all the data at one time, because the Compass API seems to be meant in that way.
Larry Meadors <[EMAIL PROTECTED]> wrote: OK, so the answer to the question is...[ yes, i need all the data at one time | no, i do not need all of the data at one time ]. Larry On Thu, Mar 13, 2008 at 9:04 AM, nch wrote: > > Hi! > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap space in > both cases. I am using a profiler to look into what's happening during > execution of the indexing process and I can tell that's the exact amount it > displays for the VM. > > As I describe in the Compass user forum (see link bellow), I'm indexing a > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB of > data. > > Hibernate can handle all that data, I guess it's using some sort of > pagination. iBatis seems to divide the select statement into several select > statements, but it also seems each of such statements surpasses the limit of > 1024MB when trying to index only 300000 of the database records. > > Thank you > > > > Chris Lamey wrote: > What is you JVM heap size set to and how much data is returned by select * > from table? > > If you're trying to pull back 1G worth of data into a JVM with a heap size > set to 64M, you will hit the heap limit pretty quick. > > > -----Original Message----- > From: nch [mailto:[EMAIL PROTECTED] > Sent: Thu 3/13/2008 8:03 AM > To: user-java@ibatis.apache.org > Subject: select * causing " OutOfMemoryError: Java heap space" > > > Hi, everybody. > I'm testing the Compass search engine and, in particular, how to index a big > set of documents from a table in a MySQL database. In order to do this I > issue a "select * from table_name" using iBatis, but this seems to be > causing the application to use all heap space available. > I added a new entry into Compass user forum with details: > http://forum.compass-project.org/thread.jspa?threadID=215278 > > Can you figure out why is this happening? > > Many thanks > > > --------------------------------- > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it > now. > > > > > ________________________________ > Never miss a thing. Make Yahoo your homepage. --------------------------------- Never miss a thing. Make Yahoo your homepage.
