Hi, Nathan.
I did so, but I'm still having the same issue. Perhaps I'm not using it
correctly?
See my last post to this forum entry:
http://forum.compass-project.org/thread.jspa?threadID=215278
See the stack trace:
MemoryError: Java heap space:
java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
at
com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
at
com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at
com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at
com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
at
com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
at
com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
at
org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
at
org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
at
org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
at org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
at
org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
at
org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
at
org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
at
org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
at
org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
at
org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
at
org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
at
org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
----- Original Message ----
From: Nathan Maves <[EMAIL PROTECTED]>
To: [email protected]
Sent: Saturday, March 15, 2008 5:43:04 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
this is only my 2 cents but I would throw that class out. Write your own
implementation which uses a row handler. This is the type of situation in
which a row handler could really help. almost no memory would be used.
On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote:
Yes, I agree that dividing the query into several queries should do as long as
none of them uses more memory than available. Gracias, Carlos.
Nevertheless, I looked into
org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and, fundamentally,
what it does is a queryForPaginatedList and iterates through the resulting
PaginatedList with nextPage()
I guess that the reason why pages are kept in memory after performing
paginatedList.nextPage() is because a paginatedList.previousPage() can be
issued but, should't it free pages if heap space is running out?
Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote:
Since we have a similar trouble around here i must say that you have lot of
options when going to compass..
The first one is to index "object by object" by hand, the second one is to use
the Gps wich will use the complete bunch of data and make subindexes (making
parts of your "main index" with diferent "maps" for each subindex in your
gpsDriver.
if your indexing is as strong as ours i should look forward for numer 1 or
number 3 depending of your needs. (More a Compass than iBatis issue)
Greetings
Carlos de Luna
----- Mensaje original ----
De: nch
Para: [email protected]; [EMAIL PROTECTED]
Enviado: jueves, 13 de marzo, 2008 9:37:04
Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
Sorry. Yes, I need all the data at one time, because the Compass API seems to
be meant in that way.
Larry Meadors wrote: OK, so the answer to the question is...[ yes, i need all
the data at
one time | no, i do not need all of the data at one time ].
Larry
On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
>
> Hi!
>
> I'm doing tests both under tomcat and jetty. I set 1024MB of heap space in
> both cases. I am using a profiler to look into what's happening during
> execution of the indexing process and I can tell that's the exact amount it
> displays for the VM.
>
> As I describe in the Compass user forum (see link bellow), I'm indexing a
> Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB of
> data.
>
> Hibernate can handle all that data, I guess it's using some sort of
> pagination. iBatis seems to divide the select statement into several select
> statements, but it also seems each of such statements surpasses the limit of
> 1024MB when trying to index only 300000 of the database records.
>
> Thank you
>
>
>
> Chris Lamey wrote:
> What is you JVM heap size set to and how much data is returned by select *
> from table?
>
> If you're trying to pull back 1G worth of data into a JVM with a heap size
> set to 64M, you will hit the heap limit pretty quick.
>
>
> -----Original Message-----
> From: nch [mailto:[EMAIL PROTECTED]
> Sent: Thu 3/13/2008 8:03 AM
> To: [email protected]
> Subject: select * causing " OutOfMemoryError: Java heap space"
>
>
> Hi, everybody.
> I'm testing the Compass search engine and, in particular, how to index a big
> set of documents from a table in a MySQL database. In order to do this I
> issue a "select * from table_name" using iBatis, but this seems to be
> causing the application to use all heap space available.
> I added a new entry into Compass user forum with details:
> http://forum.compass-project.org/thread.jspa?threadID=215278
>
> Can you figure out why is this happening?
>
> Many thanks
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.
>
>
>
>
> ________________________________
> Never miss a thing. Make Yahoo your homepage.
Never miss a thing. Make Yahoo your homepage.
____________________________________________________________________________________
¡Capacidad ilimitada de almacenamiento en tu correo!
No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
http://correo.espanol.yahoo.com/
Never miss a thing. Make Yahoo your homepage.
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
