SOLUTION? A quick Google search yielded this thread from the MySQL boards which details the exact problem your having.
http://lists.mysql.com/java/9137 It's actually quite silly and definitely a MySQL driver issue. I imagine that Hibernate's MySQL dialect configuration is aware of this problem, which is why it works. Recall what I said earlier regarding the fact that the driver throws the exception from the PreparedStatement.execute() method... thus it's not even getting to the point of returning a result set. This is why: // MySQL Driver code.... protected boolean createStreamingResultSet() { return ((this.resultSetType == java.sql.ResultSet.TYPE_FORWARD_ONLY) && (this.resultSetConcurrency == java.sql.ResultSet.CONCUR_READ_ONLY) && (this.fetchSize == Integer.MIN_VALUE)); // NOTICE THIS.... WTF!? } Turns out that for some reason the MySQL driver will always load all of the results in advance unless the fetchSize is set to Integer.MIN_VALUE.... wow. Try setting fetchSize="-2147483648" Silly, but wow... Clinton On Tue, Mar 18, 2008 at 9:13 AM, nch <[EMAIL PROTECTED]> wrote: > > > > > Hi! > > Clinton, answering to your last mail: > - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currently using) > with MySQL 5.0.45 (Debian). Tried also on Windows with the same drivers, but > can not remember the MySQL version. > - Bellow are, I think I don't miss any, the different relevant configurations > (some through Spring): > > - jdbc.properties: Database connection config > > jdbc.driverClassName=com.mysql.jdbc.Driver > jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf-8 > jdbc.username=x > jdbc.password=y > > > - sql-map-config.xml: iBatis config file > > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map Config > 2.0//EN" > "http://ibatis.apache.org/dtd/sql-map-config-2.dtd";> > > <sqlMapConfig> > <settings enhancementEnabled="true" maxTransactions="20" maxRequests="32" > maxSessions="10"/> > > <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/> > > </sqlMapConfig> > > > - TestingPojoSQL.xml: The operations needed for indexing: > > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" > "http://ibatis.apache.org/dtd/sql-map-2.dtd";> > > <sqlMap namespace="TestingPojoSQL"> > <typeAlias alias="article" type="org.myorg.model.TestingPojo"/> > > <resultMap id="articleResult" class="article"> > <result property="id" column="id"/> > <result property="contents" column="contents"/> > </resultMap> > > <select id="getAllTestingPojos" resultMap="articleResult" fetchSize="100" > resultSetType="FORWARD_ONLY"> > <![CDATA[ > select * from articles > ]]> > </select> > > </sqlMap> > > > - applicationContext-resources.xml: Here a DBCP data source is defined (I > tried also C3P0. Made no difference): > > <?xml version="1.0" encoding="UTF-8"?> > > xmlns:jee="http://www.springframework.org/schema/jee"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://www.springframework.org/schema/jee > http://www.springframework.org/schema/jee/spring-jee-2.0.xsd";> > > <bean id="propertyConfigurer" > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <list> > <value>classpath:jdbc.properties</value> > </list> > </property> > </bean> > > <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" > destroy-method="close"> > <property name="driverClassName" value="${jdbc.driverClassName}"/> > <property name="url" value="${jdbc.url}"/> > <property name="username" value="${jdbc.username}"/> > <property name="password" value="${jdbc.password}"/> > <property name="maxActive" value="100"/> > <property name="maxWait" value="1000"/> > <property name="poolPreparedStatements" value="true"/> > <property name="defaultAutoCommit" value="true"/> > </bean> > > </beans> > > > - In applicationContext-dao.xml: the sqlMapClient > > <?xml version="1.0" encoding="UTF-8"?> > > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"; > default-lazy-init="true"> > > <bean id="transactionManager" > class="org.springframework.jdbc.datasource.DataSourceTransactionManager"> > <property name="dataSource" ref="dataSource"/> > </bean> > > <bean id="sqlMapClient" > class="org.springframework.orm.ibatis.SqlMapClientFactoryBean"> > <property name="configLocation"> > <value>classpath:/sql-map-config.xml</value> > </property> > <property name="dataSource" ref="dataSource"/> > </bean> > </beans> > > > - In applicationContext-service.xml: This is where I tell Spring to > instanciate an iBatis based GPS device. > > <bean id="compass" class="org.compass.spring.LocalCompassBean"> > <property name="classMappings"> > <list> > <value>org.myorg.TestingPojo</value> > </list> > </property> > <property name="compassSettings"> > <props> > <prop key="compass.engine.connection">file://compass</prop> > <prop > key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncT > ransactionFactory</prop> > <prop key="compass.engine.analyzer.default.type">snowball</prop> > <prop key="compass.engine.analyzer.default.name">English</prop> > <!-- Mainly default values --> > <prop key="compass.engine.ramBufferSize">16</prop> > <prop key="compass.engine.maxBufferedDocs">10</prop> > <prop key="compass.engine.maxBufferedDeletedTerms">-1</prop> > <prop key="compass.engine.mergeFactor">10</prop> > <prop key="compass.engine.maxFieldLength">10000</prop> > <prop key="compass.engine.cacheIntervalInvalidation">5000</prop> > <prop key="compass.engine.indexManagerScheduleInterval">60</prop> > </props> > </property> > <property name="transactionManager" ref="transactionManager" /> > </bean> > > <bean id="iBatisGpsDevice" > class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice"> > <property name="name" value="iBatisDevice" /> > <property name="sqlMapClient" ref="sqlMapClient" /> > <property name="selectStatementsIds"> > <list> > <value>getAllTestingPojos</value> > </list> > </property> > <property name="pageSize" value="50" /> > </bean> > > <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps" > init-method="start" destroy-method="stop"> > <property name="compass"> > <ref bean="compass" /> > </property> > <property name="gpsDevices"> > <list> > <bean > class="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper"> > <property name="gpsDevice" ref="iBatisGpsDevice" /> > </bean> > </list> > </property> > </bean> > > > Regarding your previous mail: > - I emptied the handleRow method and ran the test - same result. The > exception occurs before handleRow is invoked. > - I'm not configuring any caches. Is there a default one? Should I config one? > - Setting fetchSize, different resultSetTypes, use of <select> or <statement> > elements didn't make any differences. > - Additionally, I tried truncating the contents of each Wikipedia article to > 500 characters, and I was able to index all the 650000 rows, so the number of > records being selected is not the only factor here (I just wanted to make > sure). > - I can imagine others (including you) have successfully made it with large > amounts of data (I'm new to iBatis and Compass/Lucene) so I'm sure there's a > solution. > > Lots of thanks. > > > > > > > ----- Original Message ---- > From: Clinton Begin <[EMAIL PROTECTED]> > To: user-java@ibatis.apache.org > > > > Sent: Monday, March 17, 2008 11:18:59 PM > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > Wow, I just read that other thread on the Compass site.... This is not > directly an iBATIS problem at all... > > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268) > at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843) > > This line tells me that the execution is never returning from the prepared > statement. iBATIS never has a chance to even see the ResultSet, let alone > get to the RowHandler. So swapping out the GpsDevice here won't make any > difference at all. This is odd behavior for a JDBC driver and tells me that > something is very wrong with the configuration. > > Which version of the JDBC driver are you using? What are the rest of the > iBATIS convfiguration details? > > Clinton > > > > On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <[EMAIL PROTECTED]> wrote: > > > Thanks for the summary. A few things: > > > > * PaginatedList was definitely the wrong solution for that, so I'm glad > > it's gone. > > > > * A RowHandler is possibly the right solution, but will only store as many > > records in memory as you tell it to. iBATIS does not just arbitrarily keep > > records around. Quite the opposite actually. Unless you keep a reference > > or configure a cache, iBATIS will not keep the object around at all. > > > > Looking at your implementation, there are a couple of things going on... > > > > session.create(o); > > if(currentItem== pageSize){ > > if (log.isDebugEnabled()) { > > log.debug("Indexing page number ["+ pageCount++ + "]"); > > } > > session.evictAll(); > > > > Things to Try: > > > > 1) These lines of code must be treated as guilty until proven innocent > > (especially the logging). The easiest way to do that is to delete them. > > Create an empty handler/GpsDevice and run the test to see what happens. > > > > 2) Look at the iBATIS configuration files. Is a cache configured for this > > query? What kind of cache? > > > > 3) You could try multiple queries of fixed sizes with > > queryForList(String,Object,int offset,int limit) ... but I can't see how > > that would be better than a RowHandler. > > > > 4) Next, just in case the driver's default fetch size is ridiculously high > > (or unlimited), try setting <select ... fetchSize="100"/>. You could also > > try a scrollable SesultSet with <statement ... resultSetType /> of > > SCROLL_INSENSITIVE or SCROLL_SENSITIVE. > > > > Finally, please know that I've used iBATIS to load tens of millions of rows > > for ETL and even analysis (NetFlix Prize) and was able to do so with a > > RowHandler or even simply multiple calls to queryForList(String,Object,int > > offset,int limit). I don't recall having to even think about the > > configuration, although I did manage to optimize it to achieve a > > read/insert performance of 15,000 records per second between two databases > > (on a single machine too). > > > > I'm sure there's something about the configuration here that is causing the > > problem, and it can be solved. > > > > I hope one of these helps. > > > > Clinton > > > > > > > > > > > > On Mon, Mar 17, 2008 at 1:53 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > Hi, there. :-) > > > > > > The thing is, I'm testing a search engine called Compass. One of the > > > tests consists of indexing a Wikipedia dump. In this case only the text > > > of the articles, so I downloaded it and imported it into a MySQL database > > > (about 650000 records - 1.5GB). > > > > > > You can configure Compass to access a database and index it's contents by > > > configuring what they call a GPS Device. You can do that through an ORM > > > such as iBatis, Hibernate or JPA. > > > > > > Compass provides an implementation of such a GPS Device called > > > SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the > > > results of a query and, so, index them. The query is just a "select * > > > from articles_table". > > > > > > So I wired everything up and ran several tests with different amounts of > > > data to be indexed and JVM stack space sizes. The result was an > > > "OutOfMemoryError: java heap space" error message whenever the size of > > > the selected rows was bigger than the amount of available heap space (I > > > can not index the 650000 records having 1GB of stack). > > > > > > After doing some profiling I concluded iBatis was loading the whole > > > amount of results into memory. So I decided to try Hibernate, which > > > worked just fine. > > > > > > Finally I found queryForPaginatedList was deprecated and that I should > > > use queryWithRowHandler instead, so I made my own implementation of the > > > SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I > > > had the same result (I profiled the application stack usage and I could > > > see a line growing steadily and finally crash against the stack ceiling > > > while performing the select statement). > > > > > > Shay Banon, Compass project leader, is planning to patch > > > SqlMapClientGpsDevice by replacing queryForPaginatedList with > > > queryWithRowHandler, but I'm doubtful it's gonna work. > > > > > > Finally you can, of course, devide the select stament into several > > > selects (and it works) but, in my opinion, that shouldn't be necessary. > > > You can read more on > > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > > > Thanks! > > > > > > > > > > > > > > > ----- Original Message ---- > > > From: Clinton Begin <[EMAIL PROTECTED]> > > > To: user-java@ibatis.apache.org > > > > > > > > > > > > Sent: Monday, March 17, 2008 2:22:39 PM > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > Wow, this thread is interesting. I suppose I could read the past > > > emails, but could someone summarize the problem and progress to date? I > > > may be able to shed some light on what is happening. > > > > > > Clinton > > > > > > > > > On Sun, Mar 16, 2008 at 11:49 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Another thing I forgot. This works perfectly well if we replace iBatis > > > > by Hibernate, so I don't think this is being caused by the MySQL driver. > > > > > > > > > > > > > > > > ----- Original Message ---- > > > > From: nch <[EMAIL PROTECTED]> > > > > To: user-java@ibatis.apache.org > > > > > > > > > > > > > > > > Sent: Sunday, March 16, 2008 10:18:42 PM > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > > > > > > > > > Well, I modified the mediumblob into a mediumtext and removed all other > > > > fields except the id one. > > > > > > > > > > > > ----- Original Message ---- > > > > From: nch <[EMAIL PROTECTED]> > > > > To: user-java@ibatis.apache.org > > > > Sent: Sunday, March 16, 2008 10:06:34 PM > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > > > > > > > > > You're probably right. I'm using the table called "text" which contains > > > > a field of type "mediumblob". > > > > CREATE TABLE /*$wgDBprefix*/text ( > > > > old_id int unsigned NOT NULL auto_increment, > > > > old_text mediumblob NOT NULL, > > > > old_flags tinyblob NOT NULL, > > > > PRIMARY KEY old_id (old_id) > > > > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240; > > > > > > > > > > > > > > > > > > > > You can find it's definition here: > > > > http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup > > > > > > > > I'll try to modify that field, first, and see what happens. > > > > I've already tried "select * from your_wikipedia_dump limit $start$, > > > > $size$" and it seems to work fine, but I see a possible issue with > > > > this, you can not remove any of the records you've already read during > > > > the whole process. This might not be an option. > > > > > > > > Thank you. > > > > > > > > > > > > ----- Original Message ---- > > > > From: Larry Meadors <[EMAIL PROTECTED]> > > > > To: user-java@ibatis.apache.org > > > > Sent: Sunday, March 16, 2008 4:19:43 PM > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > Yeah, I just noticed that - it looks like it's failing in the jdbc > > > > driver when it tries to read a blob, maybe? > > > > > > > > I know that some versions of the postgresql jdbc driver load the > > > > entire contents of a resultset into a massive byte[][] structure. I > > > > wonder if that is what is happening here? I think you can get the > > > > source for the MySQL driver and check that if you want to. > > > > > > > > If it is, you may need to structure your app to take smaller bites of > > > > this elephant. > > > > > > > > Try "select * from your_wikipedia_dump limit $start$, $size$" instead > > > > - that will limit the size of the results to $size$ rows, starting on > > > > row $start$ (the $start$ value is zero-based). If you experiment to > > > > see how big $size$ can get, you should be able to get pretty decent > > > > performance out of this. > > > > > > > > Psuedo-code would be like this... > > > > > > > > int start = 0; > > > > int size = 1000; // see how big you can make this to improve performance > > > > while(still_more_data){ > > > > still_more_data = index(start, size); > > > > start += size; > > > > } > > > > > > > > Your index method will return true if the query returned any data. If > > > > the query returns no data, it'll return false. > > > > > > > > I can't imagine what Hibernate is doing differently here to make this > > > > work if the JDBC driver is failing when executing the query. > > > > > > > > Larry > > > > > > > > > > > > On Sun, Mar 16, 2008 at 4:12 AM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > Sure. Please, see attached. > > > > > I don't think the problem is in the RowHandler, though, because the > > > > > OutOfMemoryError occurs before invoking RowHandle#handleRow. > > > > > > > > > > Cheers > > > > > > > > > > > > > > > ----- Original Message ---- > > > > > From: Larry Meadors <[EMAIL PROTECTED]> > > > > > To: user-java@ibatis.apache.org > > > > > > > > > > Sent: Sunday, March 16, 2008 3:13:27 AM > > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > Can you post the row handler you are using. > > > > > > > > > > Larry > > > > > > > > > > > > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > Hi, Nathan. > > > > > > I did so, but I'm still having the same issue. Perhaps I'm not > > > > > > using it > > > > > > correctly? > > > > > > See my last post to this forum entry: > > > > > > > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > > > > > > > > > See the stack trace: > > > > > > > > > > > > MemoryError: Java heap space: > > > > > > java.lang.OutOfMemoryError: Java heap space > > > > > > at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198) > > > > > > at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318) > > > > > > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366) > > > > > > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333) > > > > > > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435) > > > > > > at > > > > > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040) > > > > > > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443) > > > > > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777) > > > > > > at com.mysql.jdbc.Connection.execSQL(Connection.java:3249) > > > > > > at > > > > > > > > > > > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268) > > > > > > at > > > > > > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843) > > > > > > at > > > > > > > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169) > > > > > > at > > > > > > > > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649) > > > > > > at > > > > > > > > > > > com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156) > > > > > > at > > > > > > > > > > > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33) > > > > > > at > > > > > > > > > > > org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93) > > > > > > at > > > > > > > > > > > org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29) > > > > > > at > > > > > > org.compass.core.CompassTemplate.execute(CompassTemplate.java:132) > > > > > > at > > > > > > > > > > > org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161) > > > > > > at > > > > > > org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91) > > > > > > at > > > > > > > > > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98) > > > > > > at > > > > > > > > > > > org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33) > > > > > > at > > > > > > > > > > > org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128) > > > > > > at > > > > > > > > > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96) > > > > > > at > > > > > > > > > > > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133) > > > > > > at > > > > > > > > > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233) > > > > > > at > > > > > > > > > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ---- > > > > > > From: Nathan Maves <[EMAIL PROTECTED]> > > > > > > To: user-java@ibatis.apache.org > > > > > > > > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM > > > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > this is only my 2 cents but I would throw that class out. Write > > > > > > your own > > > > > > implementation which uses a row handler. This is the type of > > > > > > situation in > > > > > > which a row handler could really help. almost no memory would be > > > > > > used. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > Yes, I agree that dividing the query into several queries should > > > > > > > do as > > > > > > long as none of them uses more memory than available. Gracias, > > > > > > Carlos. > > > > > > > > > > > > > > Nevertheless, I looked into > > > > > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and, > > > > > > fundamentally, what it does is a queryForPaginatedList and iterates > > > > > through > > > > > > the resulting PaginatedList with nextPage() > > > > > > > > > > > > > > I guess that the reason why pages are kept in memory after > > > > > > > performing > > > > > > paginatedList.nextPage() is because a paginatedList.previousPage() > > > > > > can be > > > > > > issued but, should't it free pages if heap space is running out? > > > > > > > > > > > > > > > > > > > > > Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > Since we have a similar trouble around here i must say that you > > > > > > > have lot > > > > > > of options when going to compass.. > > > > > > > The first one is to index "object by object" by hand, the second > > > > > > > one is > > > > > to > > > > > > use the Gps wich will use the complete bunch of data and make > > > > > > subindexes > > > > > > (making parts of your "main index" with diferent "maps" for each > > > > > > subindex > > > > > in > > > > > > your gpsDriver. > > > > > > > if your indexing is as strong as ours i should look forward for > > > > > > > numer 1 > > > > > or > > > > > > number 3 depending of your needs. (More a Compass than iBatis issue) > > > > > > > Greetings > > > > > > > Carlos de Luna > > > > > > > > > > > > > > ----- Mensaje original ---- > > > > > > > De: nch > > > > > > > > > > > > > > Para: user-java@ibatis.apache.org; [EMAIL PROTECTED] > > > > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04 > > > > > > > Asunto: Re: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > > > > > > > > > > Sorry. Yes, I need all the data at one time, because the Compass > > > > > > > API > > > > > seems > > > > > > to be meant in that way. > > > > > > > > > > > > > > Larry Meadors wrote: OK, so the answer to the question is...[ > > > > > > > yes, i > > > > > need > > > > > > all the data at > > > > > > > > > > > > > > > > > > > > > > > > > > > > one time | no, i do not need all of the data at one time ]. > > > > > > > > > > > > > > Larry > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote: > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > I'm doing tests both under tomcat and jetty. I set 1024MB of > > > > > > > > heap > > > > > space > > > > > > in > > > > > > > > both cases. I am using a profiler to look into what's happening > > > > > > > > during > > > > > > > > execution of the indexing process and I can tell that's the > > > > > > > > exact > > > > > amount > > > > > > it > > > > > > > > displays for the VM. > > > > > > > > > > > > > > > > As I describe in the Compass user forum (see link bellow), I'm > > > > > indexing > > > > > > a > > > > > > > > Wikipedia dump of about 650000-675000 records. That's sort of > > > > > > > > 1-1.5GB > > > > > of > > > > > > > > data. > > > > > > > > > > > > > > > > Hibernate can handle all that data, I guess it's using some > > > > > > > > sort of > > > > > > > > pagination. iBatis seems to divide the select statement into > > > > > > > > several > > > > > > select > > > > > > > > statements, but it also seems each of such statements surpasses > > > > > > > > the > > > > > > limit of > > > > > > > > 1024MB when trying to index only 300000 of the database records. > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chris Lamey wrote: > > > > > > > > What is you JVM heap size set to and how much data is returned > > > > > > > > by > > > > > select > > > > > > * > > > > > > > > from table? > > > > > > > > > > > > > > > > If you're trying to pull back 1G worth of data into a JVM with > > > > > > > > a heap > > > > > > size > > > > > > > > set to 64M, you will hit the heap limit pretty quick. > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: nch [mailto:[EMAIL PROTECTED] > > > > > > > > Sent: Thu 3/13/2008 8:03 AM > > > > > > > > To: user-java@ibatis.apache.org > > > > > > > > Subject: select * causing " OutOfMemoryError: Java heap space" > > > > > > > > > > > > > > > > > > > > > > > > Hi, everybody. > > > > > > > > I'm testing the Compass search engine and, in particular, how > > > > > > > > to index > > > > > a > > > > > > big > > > > > > > > set of documents from a table in a MySQL database. In order to > > > > > > > > do this > > > > > I > > > > > > > > issue a "select * from table_name" using iBatis, but this seems > > > > > > > > to be > > > > > > > > causing the application to use all heap space available. > > > > > > > > I added a new entry into Compass user forum with details: > > > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278 > > > > > > > > > > > > > > > > Can you figure out why is this happening? > > > > > > > > > > > > > > > > Many thanks > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------- > > > > > > > > Be a better friend, newshound, and know-it-all with Yahoo! > > > > > > > > Mobile. Try > > > > > > it > > > > > > > > now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > > > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > > > > ¡Capacidad ilimitada de almacenamiento en tu correo! > > > > > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!: > > > > > > > http://correo.espanol.yahoo.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > Never miss a thing. Make Yahoo your homepage. > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. > > > > > Try it > > > > > now. > > > > > > > > > > > > ________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. > > > > > > > > ________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. > > > > > > > > > > > > > > > > > > > > ________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. > > > > > > > > > > > > > > > > > > > > > ________________________________ Never miss a thing. Make Yahoo your homepage. > > > > > > > > ________________________________ > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
