HI all,thanks for your answers! Actually i'm using Cayenne 2.0. Resultiterator works well! The memory doesn't grow, and the data are retrived in a resonable time (100.000 record, 60Mb in 50/60 seconds... this is the worst case). So i think that this is a good solution that also allow me to maintain the same logic i actually use for manage datacontext.
I was sure that Cayenne can solve my problem. Thanks for this great product! ;-) Davide On Wed, Mar 18, 2009 at 3:52 PM, Andrus Adamchik <[email protected]>wrote: > Actually using a ResultIterator should be entirely safe in 3.0 in respect > to memory management. If a user doesn't do something bad (like storing > obtained objects in a list himself), and only one object is processed at a > time, there won't be any memory issues. > > The servlet i'm implementig will be called by a different webapp >>> (backoffice.. in a server 2 server way ) with a low traffic but i'm >>> warried >>> about the warning reported in the article: >>> >>> "In web applications, programmers must ensure that no open >>> ResultIterators >>> are kept between HTTP requests. Failure to do so may result in too many >>> database connections being locked, thus quickly exhausting connection >>> pool. >>> >> > This one is still valid. Keep ResultIterator request-scoped, and close it > at the end. > > In general, an application with Web GUI is NOT a good candidate for >>> implementation using ResultIterators." >>> >> > Well, there are always exceptions... The general warning here is based on > the assumption that large results are usually taking lots of time to fetch > and process. So users may not see the response fast enough. If the response > time is reasonable in your case, you can use ResultIterator. > > Andrus > > > On Mar 18, 2009, at 4:17 PM, Robert Zeigler wrote: > > Problem is that the data context will cache the objects (up to the max >> object limit); this is alleviated somewhat in 3.0 with the use of weak >> references >> in the caching, but there's still potential for running out of memory. >> >> You can certainly use ResultIterator, with the caveat that you'll be >> getting DataRows instead of DataObjects. >> Alternatively, periodically discard the context that you're using and >> replace it with a fresh context, something like: >> >> Query query = createMyQuery();//SQLTemplate, or SelectQuery, or >> EJBQLQuery, or whatever. >> while(notDone) { >> ObjectContext context = ...;//create a new object context here. >> //do things like set the query fetch limit and fetch offset. >> List objs = context.performQuery(query); >> for(Obj o : objs) { >> //stream out the obj... >> } >> } >> >> Robert >> >> On Mar 18, 2009, at 3/186:14 AM , mr.abanjo wrote: >> >> Hi,i'm developing an application that must load a large ammount of data >>> (100.000 records) and stream them out from a servlet. >>> >>> I created a "for" cycle that for every step load 200 record from the >>> database (using datacontext.performquery method), and then flush the data >>> in >>> the serlvet output stream. >>> In this way i suppose that the max ammount of memory allocated by cayenne >>> is >>> : >>> >>> object size * 200 >>> >>> but seems that this is not true. When i try to get the data from the >>> servlet, after few seconds, i receive an "out of memory" error. (my 500Mb >>> allocated for the Heap are reached in a few time... :-(....) >>> >>> Which is the best practice in this case? >>> I can increase the heap, but i want to find a workaround that allow me to >>> have a small heap allocated during this operation. >>> >>> I found this article: >>> http://cayenne.apache.org/doc20/iterating-through-data-rows.html >>> Is this the right way? >>> >>> The servlet i'm implementig will be called by a different webapp >>> (backoffice.. in a server 2 server way ) with a low traffic but i'm >>> warried >>> about the warning reported in the article: >>> >>> "In web applications, programmers must ensure that no open >>> ResultIterators >>> are kept between HTTP requests. Failure to do so may result in too many >>> database connections being locked, thus quickly exhausting connection >>> pool. >>> In general, an application with Web GUI is NOT a good candidate for >>> implementation using ResultIterators." >>> >>> Thanks >>> Davide >>> >> >> >> >
