Postgis NG + large data set + non transactional read -> OOM
-----------------------------------------------------------
Key: GEOT-2235
URL: http://jira.codehaus.org/browse/GEOT-2235
Project: GeoTools
Issue Type: Bug
Components: data jdbc-ng
Affects Versions: 2.5.2
Reporter: Andrea Aime
Assignee: Justin Deoliveira
Fix For: 2.5.3
I was trying out the postgis NG datastore against the old one using the same
old Texas roads database
when I found out that the NG one was systematically throwing OOM.
A little investigation shows that PostgreSQL can do partial fetches only if
autocommit is false:
http://www.nabble.com/ResultSet---setFetchSize-fails-to-stop-heap-failures-td17519869.html#a17521489
http://jdbc.postgresql.org/documentation/83/query.html#query-with-cursor
Looking into the old jdbc datastores code one can see that auto-commit is not
set not even
for Transaction.AUTO_COMMIT. The upside is that partial fetches do work, the
downside
is that apparently one has to use transactions to ensure changes are properly
written?
Yet, the old postgis tests ensured that operations were working even on
auto_commit,
not sure if it was an unintended side effect or if the transactions are
committed on close unless
explicitly rolled back (I would find this odd thought)...
I made a few tests by adding setAutoCommit(false) to
PostgisDialect.initializeConnection...
of course test failed. I had to add two things to get tests working again:
- add an explicit con.commit() in JDBCTestSetup.run()
- add an explicit con.commit() in PostgisDialect.postCreateTable
After those two gross hacks all tests are passing again but... ugh...
On the other side having a datastore that cannot load huge amounts of data
by streaming them is a major problem... :(
Opinions?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
_______________________________________________
Geotools-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-devel
