David Blasby wrote:
Okay, I've found a pretty nasty bug in the PostGIS driver.
My dataset is a table with about 250 rows in it. Each row has the
VMAP0 dataset polygon of a country in. Each row is large.
If you look closely at the JDBC datastore, you'll notice that it will
fetch 200 rows at a time - which means its pretty much loading in the
entire dataset for each query - thats about 250mb.
This doesnt fit into the memory allocated to geoserver (especially if
you're doing multiple requests at the same time), so an OutOfMemory
error gets thrown and the reader is closed. So far, so good.
Closing the reader will release the result set and the statement -
this frees up about 60mb of memory. It then tries to close the DB
connection. This fails - most likely because it was in the middle of
reading a tuple when it was rudely interupted by running out of memory.
It actually runs out of memory in PGStream.ReceiveTupleV3(), called
from QueryExecutorImpl.processResults().
Later, the connection is closed and the
PooledConnectionImpl$ConnectionHandler.invoke() is called, which
(because we're not AutoCommit) attempt to rollback the transaction
with a con.rollback(). This is where the problem occurs.
Inside QueryExecutorImpl.processResults(), it gets a "\" from the
server (highly likely from the bytea returned in the original query).
It, of course, doesnt understand this and throws an "An I/O error
occured while sending to the backend." error.
The end result is that the connection doesnt appear to be closed and
released back to the connectionpool. This means that connections to
the database are being leaked, and probably a fair amount of memory.
This, of course, causes the OutOfMemory error to happen more often.
Ug, is there any chance we could figure out the rows were big by looking
at the row metadata and then adjust the number of rows we request to
match? I was a bit surprised to see the large "page" size .. is there
any way we can configure that?
I'm probably going to have to talk to the postgresql JDBC people, but
I thought I would ask here first incase people know something about.
Are we doing anything "funny" with the connection pooling - I noticed
there was a "funny" proxy object there?
Not sure about the Proxy object, sometimes we clone a copy of the
Feature when doing a Feature Writer. And then we the user calls
"write()" we check to see if they
have changed anything against the original. To fix you could just make
a cusotm Feature implementation for PostGIS that records any changes and
maintains an isDirty method. Or make a lazy Feature that backs onto the
result set directly or ...
Needless to say nobody has optimized anything yet (okay except for
shapefile), I am going to do some optimizations for OracleDataStore as I go.
Jody
dave
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Geotools-devel mailing list
Geotools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Geotools-devel mailing list
Geotools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
