this is only my 2 cents but I would throw that class out. Write your own implementation which uses a row handler. This is the type of situation in which a row handler could really help. almost no memory would be used.
On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote: > > Yes, I agree that dividing the query into several queries should do as > long as none of them uses more memory than available. Gracias, Carlos. > > Nevertheless, I looked into > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and, > fundamentally, what it does is a queryForPaginatedList and iterates > through the resulting PaginatedList with nextPage() > > I guess that the reason why pages are kept in memory after performing > paginatedList.nextPage() is because a paginatedList.previousPage() can be > issued but, should't it free pages if heap space is running out? > > *Carlos de Luna Saenz <[EMAIL PROTECTED]>* wrote: > > Since we have a similar trouble around here i must say that you have lot > of options when going to compass.. > The first one is to index "object by object" by hand, the second one is to > use the Gps wich will use the complete bunch of data and make subindexes > (making parts of your "main index" with diferent "maps" for each subindex in > your gpsDriver. > if your indexing is as strong as ours i should look forward for numer 1 or > number 3 depending of your needs. (More a Compass than iBatis issue) > Greetings > Carlos de Luna > > ----- Mensaje original ---- > De: nch > Para: user-java@ibatis.apache.org; [EMAIL PROTECTED] > Enviado: jueves, 13 de marzo, 2008 9:37:04 > Asunto: Re: select * causing " OutOfMemoryError: Java heap space" > > > Sorry. Yes, I need all the data at one time, because the Compass API seems > to be meant in that way. > > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i need > all the data at > > one time | no, i do not need all of the data at one time ]. > > Larry > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote: > > > > Hi! > > > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap space > in > > both cases. I am using a profiler to look into what's happening during > > execution of the indexing process and I can tell that's the exact amount > it > > displays for the VM. > > > > As I describe in the Compass user forum (see link bellow), I'm indexing > a > > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB of > > data. > > > > Hibernate can handle all that data, I guess it's using some sort of > > pagination. iBatis seems to divide the select statement into several > select > > statements, but it also seems each of such statements surpasses the > limit of > > 1024MB when trying to index only 300000 of the database records. > > > > Thank you > > > > > > > > Chris Lamey wrote: > > What is you JVM heap size set to and how much data is returned by select > * > > from table? > > > > If you're trying to pull back 1G worth of data into a JVM with a heap > size > > set to 64M, you will hit the heap limit pretty quick. > > > > > > -----Original Message----- > > From: nch [mailto:[EMAIL PROTECTED] > > Sent: Thu 3/13/2008 8:03 AM > > To: user-java@ibatis.apache.org > > Subject: select * causing " OutOfMemoryError: Java heap space" > > > > > > Hi, everybody. > > I'm testing the Compass search engine and, in particular, how to index a > big > > set of documents from a table in a MySQL database. In order to do this I > > issue a "select * from table_name" using iBatis, but this seems to be > > causing the application to use all heap space available. > > I added a new entry into Compass user forum with details: > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > Can you figure out why is this happening? > > > > Many thanks > > > > > > --------------------------------- > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try > it > > now. > > > > > > > > > > ________________________________ > > Never miss a thing. Make Yahoo your homepage. > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > ____________________________________________________________________________________ > ¡Capacidad ilimitada de almacenamiento en tu correo! > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!: > http://correo.espanol.yahoo.com/ > > > ------------------------------ > Never miss a thing. Make Yahoo your > homepage.<http://us.rd.yahoo.com/evt=51438/*http://www.yahoo.com/r/hs> >
