Hi Michael, I understand your approach of using a flag to identify already processed objects. But introducing a flag or in my case another state just for processing my records, was something I wanted to avoid. I thought that Cayenne maybe has another way of fetching objects in a memory preserving manner. Maybe some Iterator which on creation fetches the primary keys only. And then while iterating, batches of data rows are fetched in the background.
Simon Am 17.12.2012 um 15:50 schrieb Michael Gentry: > Hi Simon, > > I don't know why your performIteratedQuery() would fail with a heap error. > Based upon your answer to #2, it sounds like you can do a fetch limit on > your query (call dataContext.setFetchLimit(limit) and do a normal > performQuery() and you'll get back real Cayenne objects) and only pull back > 100 or 1000 records, process them (setting them to a different state), then > commit. Do this in a new DataContext each time so the GC can reclaim the > memory. > > mrg > > > > On Mon, Dec 17, 2012 at 8:38 AM, Simon Schneider <[email protected]>wrote: > >> Hi Michael, >> >> the problem is, that I do not even get an iterator because executing a >> query like the following results in a Java Heap Space error: >> >> ResultIterator it = dataContext.performIteratedQuery(query); >> >> The answers to your questions are: >> >>> 1) How many records are you talking about? >> It's about half a million records >> >>> 2) Are you updating your object with a flag/etc you can query on again >> later (to exclude objects you've already processed)? >> I already do exclude objects by setting them to a different state. But it >> may happen that I have to process half a million records despite of this. >> >>> 3) What version of Cayenne are you using and what database? >> Cayenne 3.0.2, Postgres 9.1 >> >>> 4) When you convert your Map (from the iterated query) into a >> DataObject, are you creating a new DataContext or using the old one over >> and over again? >> At the moment I am using just one DataContext unregistering the processed >> objects. But as mentioned above execution does not even get to this point. >> >> Simon >> >>> Hi Simon, some questions: >>> >>> 1) How many records are you talking about? >>> 2) Are you updating your object with a flag/etc you can query on again >> later (to exclude objects you've already processed)? >>> 3) What version of Cayenne are you using and what database? >>> 4) When you convert your Map (from the iterated query) into a >> DataObject, are you creating a new DataContext or using the old one over >> and over again? >>> >>> For #4, if you are using the same DataContext repeatedly, try changing >> your logic to something more like: >>> >>> while (iterator.hasNextRow()) { >>> DataContext context = DataContext.createDataContext(); >>> Map row = (Map) iterator.nextRow(); >>> CayenneObject object = (CayenneObject) >> context.objectFromDataRow("CayenneObject", row); >>> ... >>> object.doStuff(); >>> ... >>> context.commitChanges(); >>> } >>> >>> This way you won't build up a ton of objects in a single DataContext and >> possibly run out of memory. >>> >>> mrg >>
