Looks like you can use the "userCursorFetch=true" param on the JDBC URL to get it to adhere to the fetch size:
http://bugs.mysql.com/bug.php?id=18148 Cheers, topher On 3/18/08 12:24 PM, "nch" <[EMAIL PROTECTED]> wrote: > > So-lu-tion. > But, yeah, WOW!! > > Clinton, you've been really helpful. > Lots of thanks to you and all people in this mailing list. > > ----- Original Message ---- > From: Clinton Begin <[EMAIL PROTECTED]> > To: user-java@ibatis.apache.org > Sent: Tuesday, March 18, 2008 6:45:28 PM > Subject: Re: select * causing " OutOfMemoryError: Java heap space" > > SOLUTION? > > A quick Google search yielded this thread from the MySQL boards which > details the exact problem your having. > > http://lists.mysql.com/java/9137 > > It's actually quite silly and definitely a MySQL driver issue. I > imagine that Hibernate's MySQL dialect configuration is aware of this > problem, which is why it works. Recall what I said earlier regarding > the fact that the driver throws the exception from the > PreparedStatement.execute() method... thus it's not even getting to > the point of returning a result set. This is why: > > // MySQL Driver code.... > protected boolean createStreamingResultSet() { > return ((this.resultSetType == java.sql.ResultSet.TYPE_FORWARD_ONLY) > && (this.resultSetConcurrency == java.sql.ResultSet.CONCUR_READ_ONLY) > && (this.fetchSize == Integer.MIN_VALUE)); // NOTICE THIS.... WTF!? > } > > Turns out that for some reason the MySQL driver will always load all > of the results in advance unless the fetchSize is set to > Integer.MIN_VALUE.... wow. > > Try setting fetchSize="-2147483648" > > Silly, but wow... > > Clinton > > On Tue, Mar 18, 2008 at 9:13 AM, nch <[EMAIL PROTECTED]> wrote: >> >> >> >> >> Hi! >> >> Clinton, answering to your last mail: >> - I tried MySQL driver versions 5.0.5 and 5.0.8 (the one I'm currently using) >> with MySQL 5.0.45 (Debian). Tried also on Windows with the same drivers, but >> can not remember the MySQL version. >> - Bellow are, I think I don't miss any, the different relevant configurations >> (some through Spring): >> >> - jdbc.properties: Database connection config >> >> jdbc.driverClassName=com.mysql.jdbc.Driver >> jdbc.url=jdbc:mysql://localhost/wikipedia_articles?createDatabaseIfNotExist=t >> rue&useUnicode=true&characterEncoding=utf-8 >> jdbc.username=x >> jdbc.password=y >> >> >> - sql-map-config.xml: iBatis config file >> >> <?xml version="1.0" encoding="UTF-8"?> >> <!DOCTYPE sqlMapConfig PUBLIC "-//ibatis.apache.org//DTD SQL Map Config >> 2.0//EN" >> "http://ibatis.apache.org/dtd/sql-map-config-2.dtd";> >> >> <sqlMapConfig> >> <settings enhancementEnabled="true" maxTransactions="20" maxRequests="32" >> maxSessions="10"/> >> >> <sqlMap resource="sqlmaps/TestingPojoSQL.xml"/> >> >> </sqlMapConfig> >> >> >> - TestingPojoSQL.xml: The operations needed for indexing: >> >> <?xml version="1.0" encoding="UTF-8"?> >> <!DOCTYPE sqlMap PUBLIC "-//iBATIS.com//DTD SQL Map 2.0//EN" >> "http://ibatis.apache.org/dtd/sql-map-2.dtd";> >> >> <sqlMap namespace="TestingPojoSQL"> >> <typeAlias alias="article" type="org.myorg.model.TestingPojo"/> >> >> <resultMap id="articleResult" class="article"> >> <result property="id" column="id"/> >> <result property="contents" column="contents"/> >> </resultMap> >> >> <select id="getAllTestingPojos" resultMap="articleResult" fetchSize="100" >> resultSetType="FORWARD_ONLY"> >> <![CDATA[ >> select * from articles >> ]]> >> </select> >> >> </sqlMap> >> >> >> - applicationContext-resources.xml: Here a DBCP data source is defined (I >> tried also C3P0. Made no difference): >> >> <?xml version="1.0" encoding="UTF-8"?> >> >> xmlns:jee="http://www.springframework.org/schema/jee"; >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> http://www.springframework.org/schema/jee >> http://www.springframework.org/schema/jee/spring-jee-2.0.xsd";> >> >> <bean id="propertyConfigurer" >> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer >> "> >> <property name="locations"> >> <list> >> <value>classpath:jdbc.properties</value> >> </list> >> </property> >> </bean> >> >> <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" >> destroy-method="close"> >> <property name="driverClassName" value="${jdbc.driverClassName}"/> >> <property name="url" value="${jdbc.url}"/> >> <property name="username" value="${jdbc.username}"/> >> <property name="password" value="${jdbc.password}"/> >> <property name="maxActive" value="100"/> >> <property name="maxWait" value="1000"/> >> <property name="poolPreparedStatements" value="true"/> >> <property name="defaultAutoCommit" value="true"/> >> </bean> >> >> </beans> >> >> >> - In applicationContext-dao.xml: the sqlMapClient >> >> <?xml version="1.0" encoding="UTF-8"?> >> >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd"; >> default-lazy-init="true"> >> >> <bean id="transactionManager" >> class="org.springframework.jdbc.datasource.DataSourceTransactionManager"> >> <property name="dataSource" ref="dataSource"/> >> </bean> >> >> <bean id="sqlMapClient" >> class="org.springframework.orm.ibatis.SqlMapClientFactoryBean"> >> <property name="configLocation"> >> <value>classpath:/sql-map-config.xml</value> >> </property> >> <property name="dataSource" ref="dataSource"/> >> </bean> >> </beans> >> >> >> - In applicationContext-service.xml: This is where I tell Spring to >> instanciate an iBatis based GPS device. >> >> <bean id="compass" class="org.compass.spring.LocalCompassBean"> >> <property name="classMappings"> >> <list> >> <value>org.myorg.TestingPojo</value> >> </list> >> </property> >> <property name="compassSettings"> >> <props> >> <prop key="compass.engine.connection">file://compass</prop> >> <prop >> key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncT >> ransactionFactory</prop> >> <prop key="compass.engine.analyzer.default.type">snowball</prop> >> <prop key="compass.engine.analyzer.default.name">English</prop> >> <!-- Mainly default values --> >> <prop key="compass.engine.ramBufferSize">16</prop> >> <prop key="compass.engine.maxBufferedDocs">10</prop> >> <prop key="compass.engine.maxBufferedDeletedTerms">-1</prop> >> <prop key="compass.engine.mergeFactor">10</prop> >> <prop key="compass.engine.maxFieldLength">10000</prop> >> <prop key="compass.engine.cacheIntervalInvalidation">5000</prop> >> <prop key="compass.engine.indexManagerScheduleInterval">60</prop> >> </props> >> </property> >> <property name="transactionManager" ref="transactionManager" /> >> </bean> >> >> <bean id="iBatisGpsDevice" >> class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice"> >> <property name="name" value="iBatisDevice" /> >> <property name="sqlMapClient" ref="sqlMapClient" /> >> <property name="selectStatementsIds"> >> <list> >> <value>getAllTestingPojos</value> >> </list> >> </property> >> <property name="pageSize" value="50" /> >> </bean> >> >> <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps" >> init-method="start" destroy-method="stop"> >> <property name="compass"> >> <ref bean="compass" /> >> </property> >> <property name="gpsDevices"> >> <list> >> <bean >> class="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper"> >> <property name="gpsDevice" ref="iBatisGpsDevice" /> >> </bean> >> </list> >> </property> >> </bean> >> >> >> Regarding your previous mail: >> - I emptied the handleRow method and ran the test - same result. The >> exception occurs before handleRow is invoked. >> - I'm not configuring any caches. Is there a default one? Should I config >> one? >> - Setting fetchSize, different resultSetTypes, use of <select> or <statement> >> elements didn't make any differences. >> - Additionally, I tried truncating the contents of each Wikipedia article to >> 500 characters, and I was able to index all the 650000 rows, so the number of >> records being selected is not the only factor here (I just wanted to make >> sure). >> - I can imagine others (including you) have successfully made it with large >> amounts of data (I'm new to iBatis and Compass/Lucene) so I'm sure there's a >> solution. >> >> Lots of thanks. >> >> >> >> >> >> >> ----- Original Message ---- >> From: Clinton Begin <[EMAIL PROTECTED]> >> To: user-java@ibatis.apache.org >> >> >> >> Sent: Monday, March 17, 2008 11:18:59 PM >> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >> >> Wow, I just read that other thread on the Compass site.... This is not >> directly an iBATIS problem at all... >> >> at >> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268) >> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843) >> >> This line tells me that the execution is never returning from the prepared >> statement. iBATIS never has a chance to even see the ResultSet, let alone >> get to the RowHandler. So swapping out the GpsDevice here won't make any >> difference at all. This is odd behavior for a JDBC driver and tells me that >> something is very wrong with the configuration. >> >> Which version of the JDBC driver are you using? What are the rest of the >> iBATIS convfiguration details? >> >> Clinton >> >> >> >> On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <[EMAIL PROTECTED]> >> wrote: >> >>> Thanks for the summary. A few things: >>> >>> * PaginatedList was definitely the wrong solution for that, so I'm glad it's >>> gone. >>> >>> * A RowHandler is possibly the right solution, but will only store as many >>> records in memory as you tell it to. iBATIS does not just arbitrarily keep >>> records around. Quite the opposite actually. Unless you keep a reference >>> or configure a cache, iBATIS will not keep the object around at all. >>> >>> Looking at your implementation, there are a couple of things going on... >>> >>> session.create(o); >>> if(currentItem== pageSize){ >>> if (log.isDebugEnabled()) { >>> log.debug("Indexing page number ["+ pageCount++ + "]"); >>> } >>> session.evictAll(); >>> >>> Things to Try: >>> >>> 1) These lines of code must be treated as guilty until proven innocent >>> (especially the logging). The easiest way to do that is to delete them. >>> Create an empty handler/GpsDevice and run the test to see what happens. >>> >>> 2) Look at the iBATIS configuration files. Is a cache configured for this >>> query? What kind of cache? >>> >>> 3) You could try multiple queries of fixed sizes with >>> queryForList(String,Object,int offset,int limit) ... but I can't see how >>> that would be better than a RowHandler. >>> >>> 4) Next, just in case the driver's default fetch size is ridiculously high >>> (or unlimited), try setting <select ... fetchSize="100"/>. You could also >>> try a scrollable SesultSet with <statement ... resultSetType /> of >>> SCROLL_INSENSITIVE or SCROLL_SENSITIVE. >>> >>> Finally, please know that I've used iBATIS to load tens of millions of rows >>> for ETL and even analysis (NetFlix Prize) and was able to do so with a >>> RowHandler or even simply multiple calls to queryForList(String,Object,int >>> offset,int limit). I don't recall having to even think about the >>> configuration, although I did manage to optimize it to achieve a read/insert >>> performance of 15,000 records per second between two databases (on a single >>> machine too). >>> >>> I'm sure there's something about the configuration here that is causing the >>> problem, and it can be solved. >>> >>> I hope one of these helps. >>> >>> Clinton >>> >>> >>> >>> >>> >>> On Mon, Mar 17, 2008 at 1:53 PM, nch <[EMAIL PROTECTED]> wrote: >>> >>>> >>>> >>>> >>>> >>>> Hi, there. :-) >>>> >>>> The thing is, I'm testing a search engine called Compass. One of the tests >>>> consists of indexing a Wikipedia dump. In this case only the text of the >>>> articles, so I downloaded it and imported it into a MySQL database (about >>>> 650000 records - 1.5GB). >>>> >>>> You can configure Compass to access a database and index it's contents by >>>> configuring what they call a GPS Device. You can do that through an ORM >>>> such as iBatis, Hibernate or JPA. >>>> >>>> Compass provides an implementation of such a GPS Device called >>>> SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the >>>> results of a query and, so, index them. The query is just a "select * from >>>> articles_table". >>>> >>>> So I wired everything up and ran several tests with different amounts of >>>> data to be indexed and JVM stack space sizes. The result was an >>>> "OutOfMemoryError: java heap space" error message whenever the size of the >>>> selected rows was bigger than the amount of available heap space (I can not >>>> index the 650000 records having 1GB of stack). >>>> >>>> After doing some profiling I concluded iBatis was loading the whole amount >>>> of results into memory. So I decided to try Hibernate, which worked just >>>> fine. >>>> >>>> Finally I found queryForPaginatedList was deprecated and that I should use >>>> queryWithRowHandler instead, so I made my own implementation of the >>>> SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I had >>>> the same result (I profiled the application stack usage and I could see a >>>> line growing steadily and finally crash against the stack ceiling while >>>> performing the select statement). >>>> >>>> Shay Banon, Compass project leader, is planning to patch >>>> SqlMapClientGpsDevice by replacing queryForPaginatedList with >>>> queryWithRowHandler, but I'm doubtful it's gonna work. >>>> >>>> Finally you can, of course, devide the select stament into several selects >>>> (and it works) but, in my opinion, that shouldn't be necessary. >>>> You can read more on >>>> http://forum.compass-project.org/thread.jspa?threadID=215278 >>>> >>>> Thanks! >>>> >>>> >>>> >>>> >>>> ----- Original Message ---- >>>> From: Clinton Begin <[EMAIL PROTECTED]> >>>> To: user-java@ibatis.apache.org >>>> >>>> >>>> >>>> Sent: Monday, March 17, 2008 2:22:39 PM >>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>> >>>> Wow, this thread is interesting. I suppose I could read the past emails, >>>> but could someone summarize the problem and progress to date? I may be >>>> able to shed some light on what is happening. >>>> >>>> Clinton >>>> >>>> >>>> On Sun, Mar 16, 2008 at 11:49 PM, nch <[EMAIL PROTECTED]> wrote: >>>> >>>>> >>>>> >>>>> >>>>> >>>>> Another thing I forgot. This works perfectly well if we replace iBatis by >>>>> Hibernate, so I don't think this is being caused by the MySQL driver. >>>>> >>>>> >>>>> >>>>> ----- Original Message ---- >>>>> From: nch <[EMAIL PROTECTED]> >>>>> To: user-java@ibatis.apache.org >>>>> >>>>> >>>>> >>>>> Sent: Sunday, March 16, 2008 10:18:42 PM >>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>>> >>>>> >>>>> >>>>> >>>>> Well, I modified the mediumblob into a mediumtext and removed all other >>>>> fields except the id one. >>>>> >>>>> >>>>> ----- Original Message ---- >>>>> From: nch <[EMAIL PROTECTED]> >>>>> To: user-java@ibatis.apache.org >>>>> Sent: Sunday, March 16, 2008 10:06:34 PM >>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>>> >>>>> >>>>> >>>>> >>>>> You're probably right. I'm using the table called "text" which contains a >>>>> field of type "mediumblob". >>>>> CREATE TABLE /*$wgDBprefix*/text ( >>>>> old_id int unsigned NOT NULL auto_increment, >>>>> old_text mediumblob NOT NULL, >>>>> old_flags tinyblob NOT NULL, >>>>> PRIMARY KEY old_id (old_id) >>>>> ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240; >>>>> >>>>> >>>>> >>>>> >>>>> You can find it's definition here: >>>>> http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables. >>>>> sql?view=markup >>>>> >>>>> I'll try to modify that field, first, and see what happens. >>>>> I've already tried "select * from your_wikipedia_dump limit $start$, >>>>> $size$" and it seems to work fine, but I see a possible issue with this, >>>>> you can not remove any of the records you've already read during the whole >>>>> process. This might not be an option. >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> ----- Original Message ---- >>>>> From: Larry Meadors <[EMAIL PROTECTED]> >>>>> To: user-java@ibatis.apache.org >>>>> Sent: Sunday, March 16, 2008 4:19:43 PM >>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>>> >>>>> Yeah, I just noticed that - it looks like it's failing in the jdbc >>>>> driver when it tries to read a blob, maybe? >>>>> >>>>> I know that some versions of the postgresql jdbc driver load the >>>>> entire contents of a resultset into a massive byte[][] structure. I >>>>> wonder if that is what is happening here? I think you can get the >>>>> source for the MySQL driver and check that if you want to. >>>>> >>>>> If it is, you may need to structure your app to take smaller bites of >>>>> this elephant. >>>>> >>>>> Try "select * from your_wikipedia_dump limit $start$, $size$" instead >>>>> - that will limit the size of the results to $size$ rows, starting on >>>>> row $start$ (the $start$ value is zero-based). If you experiment to >>>>> see how big $size$ can get, you should be able to get pretty decent >>>>> performance out of this. >>>>> >>>>> Psuedo-code would be like this... >>>>> >>>>> int start = 0; >>>>> int size = 1000; // see how big you can make this to improve performance >>>>> while(still_more_data){ >>>>> still_more_data = index(start, size); >>>>> start += size; >>>>> } >>>>> >>>>> Your index method will return true if the query returned any data. If >>>>> the query returns no data, it'll return false. >>>>> >>>>> I can't imagine what Hibernate is doing differently here to make this >>>>> work if the JDBC driver is failing when executing the query. >>>>> >>>>> Larry >>>>> >>>>> >>>>> On Sun, Mar 16, 2008 at 4:12 AM, nch <[EMAIL PROTECTED]> wrote: >>>>>> >>>>>> >>>>>> Sure. Please, see attached. >>>>>> I don't think the problem is in the RowHandler, though, because the >>>>>> OutOfMemoryError occurs before invoking RowHandle#handleRow. >>>>>> >>>>>> Cheers >>>>>> >>>>>> >>>>>> ----- Original Message ---- >>>>>> From: Larry Meadors <[EMAIL PROTECTED]> >>>>>> To: user-java@ibatis.apache.org >>>>>> >>>>>> Sent: Sunday, March 16, 2008 3:13:27 AM >>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>>>> >>>>>> Can you post the row handler you are using. >>>>>> >>>>>> Larry >>>>>> >>>>>> >>>>>> On Sat, Mar 15, 2008 at 1:14 PM, nch <[EMAIL PROTECTED]> wrote: >>>>>>> >>>>>>> >>>>>>> Hi, Nathan. >>>>>>> I did so, but I'm still having the same issue. Perhaps I'm not using it >>>>>>> correctly? >>>>>>> See my last post to this forum entry: >>>>>>> >>>>>>> http://forum.compass-project.org/thread.jspa?threadID=215278 >>>>>>> >>>>>>> See the stack trace: >>>>>>> >>>>>>> MemoryError: Java heap space: >>>>>>> java.lang.OutOfMemoryError: Java heap space >>>>>>> at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198) >>>>>>> at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318) >>>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366) >>>>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333) >>>>>>> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435) >>>>>>> at >>>>>>> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040) >>>>>>> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443) >>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777) >>>>>>> at com.mysql.jdbc.Connection.execSQL(Connection.java:3249) >>>>>>> at >>>>>>> >>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1 >>>>>> 268) >>>>>>> at >>>>>>> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843) >>>>>>> at >>>>>>> >>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre >>>>>> paredStatement.java:169) >>>>>>> at >>>>>>> >>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPre >>>>>> paredStatement.java:169) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.j >>>>>> ava:186) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQue >>>>>> ry(GeneralStatement.java:205) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW >>>>>> ithCallback(GeneralStatement.java:173) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryW >>>>>> ithRowHandler(GeneralStatement.java:133) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler( >>>>>> SqlMapExecutorDelegate.java:649) >>>>>>> at >>>>>>> >>>>>> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMa >>>>>> pSessionImpl.java:156) >>>>>>> at >>>>>>> >>>>>> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClien >>>>>> tGpsDeviceWithRowHandler.java:33) >>>>>>> at >>>>>>> >>>>>> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(Abstr >>>>>> actGpsDevice.java:93) >>>>>>> at >>>>>>> >>>>>> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallback >>>>>> WithoutResult.java:29) >>>>>>> at >>>>>>> org.compass.core.CompassTemplate.execute(CompassTemplate.java:132) >>>>>>> at >>>>>>> >>>>>> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.ja >>>>>> va:161) >>>>>>> at >>>>>>> org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91>>>>>>> ) >>>>>>> at >>>>>>> >>>>>> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTra >>>>>> nsactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98) >>>>>>> at >>>>>>> >>>>>> org.springframework.transaction.support.TransactionCallbackWithoutResult. >>>>>> doInTransaction(TransactionCallbackWithoutResult.java:33) >>>>>>> at >>>>>>> >>>>>> org.springframework.transaction.support.TransactionTemplate.execute(Trans >>>>>> actionTemplate.java:128) >>>>>>> at >>>>>>> >>>>>> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(Spr >>>>>> ingSyncTransactionGpsDeviceWrapper.java:96) >>>>>>> at >>>>>>> >>>>>> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassG >>>>>> ps.java:133) >>>>>>> at >>>>>>> >>>>>> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana >>>>>> ger$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233) >>>>>>> at >>>>>>> >>>>>> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexMana >>>>>> ger.doOperate(DefaultLuceneSearchEngineIndexManager.java:182) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ---- >>>>>>> From: Nathan Maves <[EMAIL PROTECTED]> >>>>>>> To: user-java@ibatis.apache.org >>>>>>> >>>>>>> Sent: Saturday, March 15, 2008 5:43:04 PM >>>>>>> Subject: Re: select * causing " OutOfMemoryError: Java heap space" >>>>>>> >>>>>>> this is only my 2 cents but I would throw that class out. Write your >>>>>>> own >>>>>>> implementation which uses a row handler. This is the type of situation >>>>>>> in >>>>>>> which a row handler could really help. almost no memory would be used. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote: >>>>>>>> >>>>>>>> Yes, I agree that dividing the query into several queries should do as >>>>>>> long as none of them uses more memory than available. Gracias, Carlos. >>>>>>>> >>>>>>>> Nevertheless, I looked into >>>>>>> org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and, >>>>>>> fundamentally, what it does is a queryForPaginatedList and iterates >>>>>> through >>>>>>> the resulting PaginatedList with nextPage() >>>>>>>> >>>>>>>> I guess that the reason why pages are kept in memory after performing >>>>>>> paginatedList.nextPage() is because a paginatedList.previousPage() can >>>>>>> be >>>>>>> issued but, should't it free pages if heap space is running out? >>>>>>>> >>>>>>>> >>>>>>>> Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote: >>>>>>>> >>>>>>>> Since we have a similar trouble around here i must say that you have >>>>>>>> lot >>>>>>> of options when going to compass.. >>>>>>>> The first one is to index "object by object" by hand, the second one is >>>>>> to >>>>>>> use the Gps wich will use the complete bunch of data and make subindexes >>>>>>> (making parts of your "main index" with diferent "maps" for each >>>>>>> subindex >>>>>> in >>>>>>> your gpsDriver. >>>>>>>> if your indexing is as strong as ours i should look forward for numer 1 >>>>>> or >>>>>>> number 3 depending of your needs. (More a Compass than iBatis issue) >>>>>>>> Greetings >>>>>>>> Carlos de Luna >>>>>>>> >>>>>>>> ----- Mensaje original ---- >>>>>>>> De: nch >>>>>>>> >>>>>>>> Para: user-java@ibatis.apache.org; [EMAIL PROTECTED] >>>>>>>> Enviado: jueves, 13 de marzo, 2008 9:37:04 >>>>>>>> Asunto: Re: select * causing " OutOfMemoryError: Java heap space" >>>>>>>> >>>>>>>> >>>>>>>> Sorry. Yes, I need all the data at one time, because the Compass API >>>>>> seems >>>>>>> to be meant in that way. >>>>>>>> >>>>>>>> Larry Meadors wrote: OK, so the answer to the question is...[ yes, i >>>>>> need >>>>>>> all the data at >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> one time | no, i do not need all of the data at one time ]. >>>>>>>> >>>>>>>> Larry >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 13, 2008 at 9:04 AM, nch wrote: >>>>>>>>> >>>>>>>>> Hi! >>>>>>>>> >>>>>>>>> I'm doing tests both under tomcat and jetty. I set 1024MB of heap >>>>>> space >>>>>>> in >>>>>>>>> both cases. I am using a profiler to look into what's happening during >>>>>>>>> execution of the indexing process and I can tell that's the exact >>>>>> amount >>>>>>> it >>>>>>>>> displays for the VM. >>>>>>>>> >>>>>>>>> As I describe in the Compass user forum (see link bellow), I'm >>>>>> indexing >>>>>>> a >>>>>>>>> Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB >>>>>> of >>>>>>>>> data. >>>>>>>>> >>>>>>>>> Hibernate can handle all that data, I guess it's using some sort of >>>>>>>>> pagination. iBatis seems to divide the select statement into several >>>>>>> select >>>>>>>>> statements, but it also seems each of such statements surpasses the >>>>>>> limit of >>>>>>>>> 1024MB when trying to index only 300000 of the database records. >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Chris Lamey wrote: >>>>>>>>> What is you JVM heap size set to and how much data is returned by >>>>>> select >>>>>>> * >>>>>>>>> from table? >>>>>>>>> >>>>>>>>> If you're trying to pull back 1G worth of data into a JVM with a heap >>>>>>> size >>>>>>>>> set to 64M, you will hit the heap limit pretty quick. >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: nch [mailto:[EMAIL PROTECTED] >>>>>>>>> Sent: Thu 3/13/2008 8:03 AM >>>>>>>>> To: user-java@ibatis.apache.org >>>>>>>>> Subject: select * causing " OutOfMemoryError: Java heap space" >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, everybody. >>>>>>>>> I'm testing the Compass search engine and, in particular, how to index >>>>>> a >>>>>>> big >>>>>>>>> set of documents from a table in a MySQL database. In order to do this >>>>>> I >>>>>>>>> issue a "select * from table_name" using iBatis, but this seems to be >>>>>>>>> causing the application to use all heap space available. >>>>>>>>> I added a new entry into Compass user forum with details: >>>>>>>>> http://forum.compass-project.org/thread.jspa?threadID=215278 >>>>>>>>> >>>>>>>>> Can you figure out why is this happening? >>>>>>>>> >>>>>>>>> Many thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------- >>>>>>>>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try >>>>>>> it >>>>>>>>> now. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________ >>>>>>>>> Never miss a thing. Make Yahoo your homepage. >>>>>>>> >>>>>>>> >>>>>>>> Never miss a thing. Make Yahoo your homepage. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> _________________________________________________________________________ >>>>>> ___________ >>>>>>>> ¡Capacidad ilimitada de almacenamiento en tu correo! >>>>>>>> No te preocupes más por el espacio de tu cuenta con Correo Yahoo!: >>>>>>>> http://correo.espanol.yahoo.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ________________________________ >>>>>>> Never miss a thing. Make Yahoo your homepage. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ________________________________ >>>>>>> Never miss a thing. Make Yahoo your homepage. >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> >>>>>> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it >>>>>> now. >>>>> >>>>> >>>>> ________________________________ > Looking for last minute shopping deals? Find them fast with Yahoo! Search. >>>>> >>>>> ________________________________ > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. >>>>> >>>>> >>>>> >>>>> >>>>> ________________________________ > Looking for last minute shopping deals? Find them fast with Yahoo! Search. >>>> >>>> >>>> >>>> >>>> >>>> >>>> ________________________________ > Never miss a thing. Make Yahoo your homepage. >>> >>> >> >> >> >> ________________________________ > >> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it >> now. > > > > > > > > ______________________________________________________________________________ > ______ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs
