Sure. Please, see attached.
I don't think the problem is in the RowHandler, though, because the
OutOfMemoryError occurs before invoking RowHandle#handleRow.
Cheers
----- Original Message ----
From: Larry Meadors <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 3:13:27 AM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
Can you post the row handler you are using.
Larry
On Sat, Mar 15, 2008 at 1:14 PM, nch <[EMAIL PROTECTED]> wrote:
>
>
> Hi, Nathan.
> I did so, but I'm still having the same issue. Perhaps I'm not using it
> correctly?
> See my last post to this forum entry:
>
> http://forum.compass-project.org/thread.jspa?threadID=215278
>
> See the stack trace:
>
> MemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> at
> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> at
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> at
> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
> at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
> at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
> at
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
> at
> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
> at
> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
> at
> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
> at
> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
> at
> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
> at
> org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> at
> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
> at
> org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
> at
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
> at
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
> at
> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
> at
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
> at
> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
> at
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
> at
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
>
>
>
>
>
> ----- Original Message ----
> From: Nathan Maves <[EMAIL PROTECTED]>
> To: user-java@ibatis.apache.org
>
> Sent: Saturday, March 15, 2008 5:43:04 PM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> this is only my 2 cents but I would throw that class out. Write your own
> implementation which uses a row handler. This is the type of situation in
> which a row handler could really help. almost no memory would be used.
>
>
>
> On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote:
> >
> > Yes, I agree that dividing the query into several queries should do as
> long as none of them uses more memory than available. Gracias, Carlos.
> >
> > Nevertheless, I looked into
> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> fundamentally, what it does is a queryForPaginatedList and iterates through
> the resulting PaginatedList with nextPage()
> >
> > I guess that the reason why pages are kept in memory after performing
> paginatedList.nextPage() is because a paginatedList.previousPage() can be
> issued but, should't it free pages if heap space is running out?
> >
> >
> > Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote:
> >
> > Since we have a similar trouble around here i must say that you have lot
> of options when going to compass..
> > The first one is to index "object by object" by hand, the second one is to
> use the Gps wich will use the complete bunch of data and make subindexes
> (making parts of your "main index" with diferent "maps" for each subindex in
> your gpsDriver.
> > if your indexing is as strong as ours i should look forward for numer 1 or
> number 3 depending of your needs. (More a Compass than iBatis issue)
> > Greetings
> > Carlos de Luna
> >
> > ----- Mensaje original ----
> > De: nch
> >
> > Para: user-java@ibatis.apache.org; [EMAIL PROTECTED]
> > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> >
> > Sorry. Yes, I need all the data at one time, because the Compass API seems
> to be meant in that way.
> >
> > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i need
> all the data at
> >
> >
> >
> > one time | no, i do not need all of the data at one time ].
> >
> > Larry
> >
> >
> > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > >
> > > Hi!
> > >
> > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap space
> in
> > > both cases. I am using a profiler to look into what's happening during
> > > execution of the indexing process and I can tell that's the exact amount
> it
> > > displays for the VM.
> > >
> > > As I describe in the Compass user forum (see link bellow), I'm indexing
> a
> > > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB of
> > > data.
> > >
> > > Hibernate can handle all that data, I guess it's using some sort of
> > > pagination. iBatis seems to divide the select statement into several
> select
> > > statements, but it also seems each of such statements surpasses the
> limit of
> > > 1024MB when trying to index only 300000 of the database records.
> > >
> > > Thank you
> > >
> > >
> > >
> > > Chris Lamey wrote:
> > > What is you JVM heap size set to and how much data is returned by select
> *
> > > from table?
> > >
> > > If you're trying to pull back 1G worth of data into a JVM with a heap
> size
> > > set to 64M, you will hit the heap limit pretty quick.
> > >
> > >
> > > -----Original Message-----
> > > From: nch [mailto:[EMAIL PROTECTED]
> > > Sent: Thu 3/13/2008 8:03 AM
> > > To: user-java@ibatis.apache.org
> > > Subject: select * causing " OutOfMemoryError: Java heap space"
> > >
> > >
> > > Hi, everybody.
> > > I'm testing the Compass search engine and, in particular, how to index a
> big
> > > set of documents from a table in a MySQL database. In order to do this I
> > > issue a "select * from table_name" using iBatis, but this seems to be
> > > causing the application to use all heap space available.
> > > I added a new entry into Compass user forum with details:
> > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > >
> > > Can you figure out why is this happening?
> > >
> > > Many thanks
> > >
> > >
> > > ---------------------------------
> > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it
> > > now.
> > >
> > >
> > >
> > >
> > > ________________________________
> > > Never miss a thing. Make Yahoo your homepage.
> >
> >
> > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >
> >
> >
> >
> >
> ____________________________________________________________________________________
> > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > http://correo.espanol.yahoo.com/
> >
> >
> >
> >
> >
> >
> >
> > ________________________________
> Never miss a thing. Make Yahoo your homepage.
>
>
>
>
> ________________________________
> Never miss a thing. Make Yahoo your homepage.
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
package org.myorg.compass;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.compass.core.CompassSession;
import com.ibatis.sqlmap.client.event.RowHandler;
public class SqlMapClientGpsDeviceRowHandler implements RowHandler {
private static final Log log = LogFactory.getLog(SqlMapClientGpsDeviceRowHandler.class);
private CompassSession session;
private int pageSize;
private int pageCount= 0;
private int currentItem= 1;
public SqlMapClientGpsDeviceRowHandler(CompassSession session, int pageSize){
this.session= session;
this.pageSize= pageSize;
}
public void handleRow(Object o) {
session.create(o);
if(currentItem== pageSize){
if (log.isDebugEnabled()) {
log.debug("Indexing page number ["+ pageCount++ + "]");
}
session.evictAll();
}
currentItem= (currentItem% pageSize)+ 1;
}
}
