I think this test also demonstrates that PostgreSQL Cursors aren't working with OpenJPA.
When I fetch 5 million entries using JDBC and iterate over them with cursors, heap memory consumption stays below 4 MB. When I fetch 2 million entries using OpenJPA with FetchBatchSize=2000, heap memory increases to 125MB. When I do it without FetchBatchSize set, memory usage is even higher, it hits 100% heap memory usage at 250MB. So, what I think might happen is that with FetchBatchSIze set, OpenJPA streams through the results of the JDBC driver, but the driver itself loads all result rows into memory at once and does not use cursors. With FetchBatchSize unset, OpenJPA tries to instantiate all entities at once and thus memory usage is even higher. So, something is probably wrong in the communication between the OpenJPA and the Postgre driver? Michael -----Ursprüngliche Nachricht----- Von: Michael Pflueger [mailto:michael.pflue...@sma.de] Gesendet: Dienstag, 13. September 2011 15:29 An: users@openjpa.apache.org Betreff: AW: Speed of fetching simple entities using OpenJPA Hi, attached is a zip containing both the OpenJPA and JDBC tests to read the entities, and a small prog to fill the DB. I'm doing entytymanager.clear() every 2000 records so memory does not fill up, and have tried to do it only every 20k records, but that seemed to make it a bit slower, not faster. Michael ___________________________________________________ SMA Solar Technology AG Aufsichtsrat: Guenther Cramer (Vorsitzender) Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner Handelsregister: Amtsgericht Kassel HRB 3972 Sitz der Gesellschaft: 34266 Niestetal USt-ID-Nr. DE 113 08 59 54 WEEE-Reg.-Nr. DE 95881150 ___________________________________________________
