Yes, I am using the same driver.
----- Original Message ----
From: Nathan Maves <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Tuesday, March 18, 2008 3:56:40 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
Are we absolutely positive that we are using the same driver with hibernate as
we are with ibatis?
On Mon, Mar 17, 2008 at 4:18 PM, Clinton Begin <[EMAIL PROTECTED]> wrote:
Wow, I just read that other thread on the Compass site.... This is not directly
an iBATIS problem at all...
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
This line tells me that the execution is never returning from the prepared
statement. iBATIS never has a chance to even see the ResultSet, let alone get
to the RowHandler. So swapping out the GpsDevice here won't make any difference
at all. This is odd behavior for a JDBC driver and tells me that something is
very wrong with the configuration.
Which version of the JDBC driver are you using? What are the rest of the
iBATIS convfiguration details?
Clinton
On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <[EMAIL PROTECTED]> wrote:
Thanks for the summary. A few things:
* PaginatedList was definitely the wrong solution for that, so I'm glad it's
gone.
* A RowHandler is possibly the right solution, but will only store as many
records in memory as you tell it to. iBATIS does not just arbitrarily keep
records around. Quite the opposite actually. Unless you keep a reference or
configure a cache, iBATIS will not keep the object around at all.
Looking at your implementation, there are a couple of things going on...
session.create(o);
if(currentItem== pageSize){
if (log.isDebugEnabled()) {
log.debug("Indexing page number ["+ pageCount++ + "]");
}
session.evictAll();
Things to Try:
1) These lines of code must be treated as guilty until proven innocent
(especially the logging). The easiest way to do that is to delete them.
Create an empty handler/GpsDevice and run the test to see what happens.
2) Look at the iBATIS configuration files. Is a cache configured for this
query? What kind of cache?
3) You could try multiple queries of fixed sizes with
queryForList(String,Object,int offset,int limit) ... but I can't see how that
would be better than a RowHandler.
4) Next, just in case the driver's default fetch size is ridiculously high (or
unlimited), try setting <select ... fetchSize="100"/>. You could also try a
scrollable SesultSet with <statement ... resultSetType /> of
SCROLL_INSENSITIVE or SCROLL_SENSITIVE.
Finally, please know that I've used iBATIS to load tens of millions of rows for
ETL and even analysis (NetFlix Prize) and was able to do so with a RowHandler
or even simply multiple calls to queryForList(String,Object,int offset,int
limit). I don't recall having to even think about the configuration, although
I did manage to optimize it to achieve a read/insert performance of 15,000
records per second between two databases (on a single machine too).
I'm sure there's something about the configuration here that is causing the
problem, and it can be solved.
I hope one of these helps.
Clinton
On Mon, Mar 17, 2008 at 1:53 PM, nch <[EMAIL PROTECTED]> wrote:
Hi, there. :-)
The thing is, I'm testing a search engine called Compass. One of the tests
consists of indexing a Wikipedia dump. In this case only the text of the
articles, so I downloaded it and imported it into a MySQL database (about
650000 records - 1.5GB).
You can configure Compass to access a database and index it's contents by
configuring what they call a GPS Device. You can do that through an ORM such as
iBatis, Hibernate or JPA.
Compass provides an implementation of such a GPS Device called
SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the
results of a query and, so, index them. The query is just a "select * from
articles_table".
So I wired everything up and ran several tests with different amounts of data
to be indexed and JVM stack space sizes. The result was an "OutOfMemoryError:
java heap space" error message whenever the size of the selected rows was
bigger than the amount of available heap space (I can not index the 650000
records having 1GB of stack).
After doing some profiling I concluded iBatis was loading the whole amount of
results into memory. So I decided to try Hibernate, which worked just fine.
Finally I found queryForPaginatedList was deprecated and that I should use
queryWithRowHandler instead, so I made my own implementation of the
SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I had the
same result (I profiled the application stack usage and I could see a line
growing steadily and finally crash against the stack ceiling while performing
the select statement).
Shay Banon, Compass project leader, is planning to patch SqlMapClientGpsDevice
by replacing queryForPaginatedList with queryWithRowHandler, but I'm doubtful
it's gonna work.
Finally you can, of course, devide the select stament into several selects (and
it works) but, in my opinion, that shouldn't be necessary.
You can read more on
http://forum.compass-project.org/thread.jspa?threadID=215278
Thanks!
----- Original Message ----
From: Clinton Begin <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Monday, March 17, 2008 2:22:39 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
Wow, this thread is interesting. I suppose I could read the past emails, but
could someone summarize the problem and progress to date? I may be able to
shed some light on what is happening.
Clinton
On Sun, Mar 16, 2008 at 11:49 PM, nch <[EMAIL PROTECTED]> wrote:
Another thing I forgot. This works perfectly well if we replace iBatis by
Hibernate, so I don't think this is being caused by the MySQL driver.
----- Original Message ----
From: nch <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 10:18:42 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
Well, I modified the mediumblob into a mediumtext and removed all other fields
except the id one.
----- Original Message ----
From: nch <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 10:06:34 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
You're probably right. I'm using the table called "text" which contains a field
of type "mediumblob".
CREATE TABLE /*$wgDBprefix*/text (
old_id int unsigned NOT NULL auto_increment,
old_text mediumblob NOT NULL,
old_flags tinyblob NOT NULL,
PRIMARY KEY old_id (old_id)
) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
You can find it's definition here:
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
I'll try to modify that field, first, and see what happens.
I've already tried "select * from your_wikipedia_dump limit $start$, $size$"
and it seems to work fine, but I see a possible issue with this, you can not
remove any of the records you've already read during the whole process. This
might not be an option.
Thank you.
----- Original Message ----
From: Larry Meadors <[EMAIL PROTECTED]>
To: user-java@ibatis.apache.org
Sent: Sunday, March 16, 2008 4:19:43 PM
Subject: Re: select * causing " OutOfMemoryError: Java heap space"
Yeah, I just noticed that - it looks like it's failing in the jdbc
driver when it tries to read a blob, maybe?
I know that some versions of the postgresql jdbc driver load the
entire contents of a resultset into a massive byte[][] structure. I
wonder if that is what is happening here? I think you can get the
source for the MySQL driver and check that if you want to.
If it is, you may need to structure your app to take smaller bites of
this elephant.
Try "select * from your_wikipedia_dump limit $start$, $size$" instead
- that will limit the size of the results to $size$ rows, starting on
row $start$ (the $start$ value is zero-based). If you experiment to
see how big $size$ can get, you should be able to get pretty decent
performance out of this.
Psuedo-code would be like this...
int start = 0;
int size = 1000; // see how big you can make this to improve performance
while(still_more_data){
still_more_data = index(start, size);
start += size;
}
Your index method will return true if the query returned any data. If
the query returns no data, it'll return false.
I can't imagine what Hibernate is doing differently here to make this
work if the JDBC driver is failing when executing the query.
Larry
On Sun, Mar 16, 2008 at 4:12 AM, nch <[EMAIL PROTECTED]> wrote:
>
>
> Sure. Please, see attached.
> I don't think the problem is in the RowHandler, though, because the
> OutOfMemoryError occurs before invoking RowHandle#handleRow.
>
> Cheers
>
>
> ----- Original Message ----
> From: Larry Meadors <[EMAIL PROTECTED]>
> To: user-java@ibatis.apache.org
>
> Sent: Sunday, March 16, 2008 3:13:27 AM
> Subject: Re: select * causing " OutOfMemoryError: Java heap space"
>
> Can you post the row handler you are using.
>
> Larry
>
>
> On Sat, Mar 15, 2008 at 1:14 PM, nch <[EMAIL PROTECTED]> wrote:
> >
> >
> > Hi, Nathan.
> > I did so, but I'm still having the same issue. Perhaps I'm not using it
> > correctly?
> > See my last post to this forum entry:
> >
> > http://forum.compass-project.org/thread.jspa?threadID=215278
> >
> > See the stack trace:
> >
> > MemoryError: Java heap space:
> > java.lang.OutOfMemoryError: Java heap space
> > at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
> > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> > at
> > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
> > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
> > at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> > at
> > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
> > at
> >
> com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
> > at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
> > at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
> > at
> >
> com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
> > at
> >
> com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
> > at
> >
> com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
> > at
> >
> org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(SqlMapClientGpsDeviceWithRowHandler.java:33)
> > at
> >
> org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(AbstractGpsDevice.java:93)
> > at
> >
> org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29)
> > at
> > org.compass.core.CompassTemplate.execute(CompassTemplate.java:132)
> > at
> >
> org.compass.gps.impl.SingleCompassGps.executeForIndex(SingleCompassGps.java:161)
> > at
> > org.compass.gps.device.AbstractGpsDevice.index(AbstractGpsDevice.java:91)
> > at
> >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult(SpringSyncTransactionGpsDeviceWrapper.java:98)
> > at
> >
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
> > at
> >
> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
> > at
> >
> org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index(SpringSyncTransactionGpsDeviceWrapper.java:96)
> > at
> >
> org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(SingleCompassGps.java:133)
> > at
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep(DefaultLuceneSearchEngineIndexManager.java:233)
> > at
> >
> org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate(DefaultLuceneSearchEngineIndexManager.java:182)
> >
> >
> >
> >
> >
> > ----- Original Message ----
> > From: Nathan Maves <[EMAIL PROTECTED]>
> > To: user-java@ibatis.apache.org
> >
> > Sent: Saturday, March 15, 2008 5:43:04 PM
> > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> >
> > this is only my 2 cents but I would throw that class out. Write your own
> > implementation which uses a row handler. This is the type of situation in
> > which a row handler could really help. almost no memory would be used.
> >
> >
> >
> > On Thu, Mar 13, 2008 at 4:57 PM, nch <[EMAIL PROTECTED]> wrote:
> > >
> > > Yes, I agree that dividing the query into several queries should do as
> > long as none of them uses more memory than available. Gracias, Carlos.
> > >
> > > Nevertheless, I looked into
> > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex and,
> > fundamentally, what it does is a queryForPaginatedList and iterates
> through
> > the resulting PaginatedList with nextPage()
> > >
> > > I guess that the reason why pages are kept in memory after performing
> > paginatedList.nextPage() is because a paginatedList.previousPage() can be
> > issued but, should't it free pages if heap space is running out?
> > >
> > >
> > > Carlos de Luna Saenz <[EMAIL PROTECTED]> wrote:
> > >
> > > Since we have a similar trouble around here i must say that you have lot
> > of options when going to compass..
> > > The first one is to index "object by object" by hand, the second one is
> to
> > use the Gps wich will use the complete bunch of data and make subindexes
> > (making parts of your "main index" with diferent "maps" for each subindex
> in
> > your gpsDriver.
> > > if your indexing is as strong as ours i should look forward for numer 1
> or
> > number 3 depending of your needs. (More a Compass than iBatis issue)
> > > Greetings
> > > Carlos de Luna
> > >
> > > ----- Mensaje original ----
> > > De: nch
> > >
> > > Para: user-java@ibatis.apache.org; [EMAIL PROTECTED]
> > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > Asunto: Re: select * causing " OutOfMemoryError: Java heap space"
> > >
> > >
> > > Sorry. Yes, I need all the data at one time, because the Compass API
> seems
> > to be meant in that way.
> > >
> > > Larry Meadors wrote: OK, so the answer to the question is...[ yes, i
> need
> > all the data at
> > >
> > >
> > >
> > > one time | no, i do not need all of the data at one time ].
> > >
> > > Larry
> > >
> > >
> > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > >
> > > > Hi!
> > > >
> > > > I'm doing tests both under tomcat and jetty. I set 1024MB of heap
> space
> > in
> > > > both cases. I am using a profiler to look into what's happening during
> > > > execution of the indexing process and I can tell that's the exact
> amount
> > it
> > > > displays for the VM.
> > > >
> > > > As I describe in the Compass user forum (see link bellow), I'm
> indexing
> > a
> > > > Wikipedia dump of about 650000-675000 records. That's sort of 1-1.5GB
> of
> > > > data.
> > > >
> > > > Hibernate can handle all that data, I guess it's using some sort of
> > > > pagination. iBatis seems to divide the select statement into several
> > select
> > > > statements, but it also seems each of such statements surpasses the
> > limit of
> > > > 1024MB when trying to index only 300000 of the database records.
> > > >
> > > > Thank you
> > > >
> > > >
> > > >
> > > > Chris Lamey wrote:
> > > > What is you JVM heap size set to and how much data is returned by
> select
> > *
> > > > from table?
> > > >
> > > > If you're trying to pull back 1G worth of data into a JVM with a heap
> > size
> > > > set to 64M, you will hit the heap limit pretty quick.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: nch [mailto:[EMAIL PROTECTED]
> > > > Sent: Thu 3/13/2008 8:03 AM
> > > > To: user-java@ibatis.apache.org
> > > > Subject: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > Hi, everybody.
> > > > I'm testing the Compass search engine and, in particular, how to index
> a
> > big
> > > > set of documents from a table in a MySQL database. In order to do this
> I
> > > > issue a "select * from table_name" using iBatis, but this seems to be
> > > > causing the application to use all heap space available.
> > > > I added a new entry into Compass user forum with details:
> > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > >
> > > > Can you figure out why is this happening?
> > > >
> > > > Many thanks
> > > >
> > > >
> > > > ---------------------------------
> > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> > it
> > > > now.
> > > >
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > > Never miss a thing. Make Yahoo your homepage.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> ____________________________________________________________________________________
> > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > http://correo.espanol.yahoo.com/
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ________________________________
> > Never miss a thing. Make Yahoo your homepage.
> >
> >
> >
> >
> > ________________________________
> > Never miss a thing. Make Yahoo your homepage.
>
>
> ________________________________
>
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.
Looking for last minute shopping deals?
Find them fast with Yahoo! Search.
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.
Looking for last minute shopping deals?
Find them fast with Yahoo! Search.
Never miss a thing. Make Yahoo your homepage.
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
