Wow, this thread is interesting. I suppose I could read the past emails, but could someone summarize the problem and progress to date? I may be able to shed some light on what is happening.
Clinton On Sun, Mar 16, 2008 at 11:49 PM, nch <[EMAIL PROTECTED]> wrote: > > Another thing I forgot. This works perfectly well if we replace iBatis by > Hibernate, so I don't think this is being caused by the MySQL driver. > > ----- Original Message ---- > From: nch <[EMAIL PROTECTED]> > To: user-java@ibatis.apache.org > Sent: Sunday, March 16, 2008 10:18:42 PM > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > Well, I modified the mediumblob into a mediumtext and removed all other > fields except the id one. > > ----- Original Message ---- > From: nch <[EMAIL PROTECTED]> > To: user-java@ibatis.apache.org > Sent: Sunday, March 16, 2008 10:06:34 PM > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > You're probably right. I'm using the table called "text" which contains a > field of type "mediumblob". > > CREATE TABLE /*$wgDBprefix*/text ( > old_id int unsigned NOT NULL auto_increment, > old_text mediumblob NOT NULL, > old_flags tinyblob NOT NULL, > PRIMARY KEY old_id (old_id) > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240; > > You can find it's definition here: > > http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup > > I'll try to modify that field, first, and see what happens. > I've already tried "select * from your_wikipedia_dump limit $start$, > $size$" and it seems to work fine, but I see a possible issue with this, you > can not remove any of the records you've already read during the whole > process. This might not be an option. > > Thank you. > > ----- Original Message ---- > From: Larry Meadors <[EMAIL PROTECTED]> > To: user-java@ibatis.apache.org > Sent: Sunday, March 16, 2008 4:19:43 PM > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > Yeah, I just noticed that - it looks like it's failing in the jdbc > driver when it tries to read a blob, maybe? > > I know that some versions of the postgresql jdbc driver load the > entire contents of a resultset into a massive byte[][] structure. I > wonder if that is what is happening here? I think you can get the > source for the MySQL driver and check that if you want to. > > If it is, you may need to structure your app to take smaller bites of > this elephant. > > Try "select * from your_wikipedia_dump limit $start$, $size$" instead > - that will limit the size of the results to $size$ rows, starting on > row $start$ (the $start$ value is zero-based). If you experiment to > see how big $size$ can get, you should be able to get pretty decent > performance out of this. > > Psuedo-code would be like this... > > int start = 0; > int size = 1000; // see how big you can make this to improve performance > while(still_more_data){ > still_more_data = index(start, size); > start += size; > } > > Your index method will return true if the query returned any data. If > the query returns no data, it'll return false. > > I can't imagine what Hibernate is doing differently here to make this > work if the JDBC driver is failing when executing the query. > > Larry > > > On Sun, Mar 16, 2008 at 4:12 AM, nch <[EMAIL PROTECTED]> wrote: > > > > > > Sure. Please, see attached. > > I don't think the problem is in the RowHandler, though, because the > > OutOfMemoryError occurs before invoking RowHandle#handleRow. > > > > Cheers > > > > > > ----- Original Message ---- > > From: Larry Meadors <[EMAIL PROTECTED]> > > To: user-java@ibatis.apache.org > > > > Sent: Sunday, March 16, 2008 3:13:27 AM > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > Can you post the row handler you are using. > > > > Larry > > > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > Hi, Nathan. > > > I did so, but I'm still having the same issue. Perhaps I'm not using > it > > > correctly? > > > See my last post to this forum entry: > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > > > See the stack trace: > > > > > > MemoryError: Java heap space: > > > java.lang.OutOfMemoryError: Java heap space > > > at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198) > > > at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318) > > > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366) > > > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333) > > > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435) > > > at > > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040) > > > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443) > > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777) > > > at com.mysql.jdbc.Connection.execSQL(Connection.java:3249) > > > at > > > > > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java > :1268) > > > at > > > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843) > > > at > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute( > DelegatingPreparedStatement.java:169) > > > at > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute( > DelegatingPreparedStatement.java:169) > > > at > > > > > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery( > SqlExecutor.java:186) > > > at > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery > (GeneralStatement.java:205) > > > at > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback > (GeneralStatement.java:173) > > > at > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler > (GeneralStatement.java:133) > > > at > > > > > com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler > (SqlMapExecutorDelegate.java:649) > > > at > > > > > com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler( > SqlMapSessionImpl.java:156) > > > at > > > > > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex( > SqlMapClientGpsDeviceWithRowHandler.java:33) > > > at > > > > > org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult( > AbstractGpsDevice.java:93) > > > at > > > > > org.compass.core.CompassCallbackWithoutResult.doInCompass( > CompassCallbackWithoutResult.java:29) > > > at > > > org.compass.core.CompassTemplate.execute(CompassTemplate.java:132) > > > at > > > > > org.compass.gps.impl.SingleCompassGps.executeForIndex( > SingleCompassGps.java:161) > > > at > > > org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java > :91) > > > at > > > > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult > (SpringSyncTransactionGpsDeviceWrapper.java:98) > > > at > > > > > > org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction > (TransactionCallbackWithoutResult.java:33) > > > at > > > > > org.springframework.transaction.support.TransactionTemplate.execute( > TransactionTemplate.java:128) > > > at > > > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index( > SpringSyncTransactionGpsDeviceWrapper.java:96) > > > at > > > > > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded( > SingleCompassGps.java:133) > > > at > > > > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep > (DefaultLuceneSearchEngineIndexManager.java:233) > > > at > > > > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate > (DefaultLuceneSearchEngineIndexManager.java:182) > > > > > > > > > > > > > > > > > > ----- Original Message ---- > > > From: Nathan Maves <[EMAIL PROTECTED]> > > > To: user-java@ibatis.apache.org > > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > this is only my 2 cents but I would throw that class out. Write your > own > > > implementation which uses a row handler. This is the type of > situation in > > > which a row handler could really help. almost no memory would be > used. > > > > > > > > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > Yes, I agree that dividing the query into several queries should do > as > > > long as none of them uses more memory than available. Gracias, Carlos. > > > > > > > > Nevertheless, I looked into > > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and, > > > fundamentally, what it does is a queryForPaginatedList and iterates > > through > > > the resulting PaginatedList with nextPage() > > > > > > > > I guess that the reason why pages are kept in memory after > performing > > > paginatedList.nextPage() is because a paginatedList.previousPage() can > be > > > issued but, should't it free pages if heap space is running out? > > > > > > > > > > > > Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote: > > > > > > > > Since we have a similar trouble around here i must say that you have > lot > > > of options when going to compass.. > > > > The first one is to index "object by object" by hand, the second one > is > > to > > > use the Gps wich will use the complete bunch of data and make > subindexes > > > (making parts of your "main index" with diferent "maps" for each > subindex > > in > > > your gpsDriver. > > > > if your indexing is as strong as ours i should look forward for > numer 1 > > or > > > number 3 depending of your needs. (More a Compass than iBatis issue) > > > > Greetings > > > > Carlos de Luna > > > > > > > > ----- Mensaje original ---- > > > > De: nch > > > > > > > > Para: user-java@ibatis.apache.org; [EMAIL PROTECTED] > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04 > > > > Asunto: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > Sorry. Yes, I need all the data at one time, because the Compass API > > seems > > > to be meant in that way. > > > > > > > > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i > > need > > > all the data at > > > > > > > > > > > > > > > > one time | no, i do not need all of the data at one time ]. > > > > > > > > Larry > > > > > > > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote: > > > > > > > > > > Hi! > > > > > > > > > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap > > space > > > in > > > > > both cases. I am using a profiler to look into what's happening > during > > > > > execution of the indexing process and I can tell that's the exact > > amount > > > it > > > > > displays for the VM. > > > > > > > > > > As I describe in the Compass user forum (see link bellow), I'm > > indexing > > > a > > > > > Wikipedia dump of about 650000-675000 records. That's sort of > 1-1.5GB > > of > > > > > data. > > > > > > > > > > Hibernate can handle all that data, I guess it's using some sort > of > > > > > pagination. iBatis seems to divide the select statement into > several > > > select > > > > > statements, but it also seems each of such statements surpasses > the > > > limit of > > > > > 1024MB when trying to index only 300000 of the database records. > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > Chris Lamey wrote: > > > > > What is you JVM heap size set to and how much data is returned by > > select > > > * > > > > > from table? > > > > > > > > > > If you're trying to pull back 1G worth of data into a JVM with a > heap > > > size > > > > > set to 64M, you will hit the heap limit pretty quick. > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: nch [mailto:[EMAIL PROTECTED] > > > > > Sent: Thu 3/13/2008 8:03 AM > > > > > To: user-java@ibatis.apache.org > > > > > Subject: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > > > > Hi, everybody. > > > > > I'm testing the Compass search engine and, in particular, how to > index > > a > > > big > > > > > set of documents from a table in a MySQL database. In order to do > this > > I > > > > > issue a "select * from table_name" using iBatis, but this seems to > be > > > > > causing the application to use all heap space available. > > > > > I added a new entry into Compass user forum with details: > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > > > > > > > Can you figure out why is this happening? > > > > > > > > > > Many thanks > > > > > > > > > > > > > > > --------------------------------- > > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. > Try > > > it > > > > > now. > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > ¡Capacidad ilimitada de almacenamiento en tu correo! > > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!: > > > > http://correo.espanol.yahoo.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > ________________________________ > > > Never miss a thing. Make Yahoo your homepage. > > > > > > ________________________________ > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try > it > > now. > > > ------------------------------ > Looking for last minute shopping deals? Find them fast with Yahoo! > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping> > > > ------------------------------ > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it > now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ%20> > > > ------------------------------ > Looking for last minute shopping deals? Find them fast with Yahoo! > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping> >
