Doh!
Yes, of course BrokerClosingIterator should implement Iterator. That
was the whole point; my bad.
Also, re: failing to close. It's true that if an error occurs, or
even if the caller just abandons the Iterator, then the broker might
not get closed. A timeout would help here; I've also been known to
close the broker in a finalizer (which again is not guaranteed to run,
but it helps the odds).
--
Steve Clark
Technology Applications Team
Natural Resources Research Center/USGS
[EMAIL PROTECTED]
(970)226-9291
>>>>> Phil Whirlycott writes:
phil> That's a clever approach. Shouldn't you also implement
phil> Iterator on this? Also, if an exception occurs during the
phil> processing in the view layer, it's possible that this might
phil> not be properly closed. Not sure how you account for that?
phil> phil.
Steve Clark wrote:
>> Another alternative is for the database layer to wrap the
>> Iterator and to defer calling broker.close(). The wrapper then
>> either closes the broker in next() when the underlying Iterator
>> is exhausted, or sets a timeout and closes the broker when it
>> expires. I've used this technique to pass OJB's result
>> Iterators to my view layer. Here's pseudocode to illustrate:
>>
>> queryResults = doQuery(broker, ...);
>> return new BrokerClosingIterator(broker, queryResults);
>> }
>>
>> class BrokerClosingIterator {
>> public BrokerClosingIterator(PersistenceBroker broker, Iterator iter) {
>> // optionally set timeout
>> }
>> public boolean hasNext() {
>> boolean more = iter.hasNext();
>> if (!more && (broker != null)) {
>> broker.close();
>> broker = null;
>> }
>> return more;
>> }
>> public Object next() {
>> return iter.next();
>> }
>> }
>>
>>
>>>>>>> Phil Whirlycott writes:
>>
>>
phil> I was thinking about this some more on the way to work
phil> today, but before I offer up the idea I had, I will first
phil> re-iterate that this is probably a bad solution to whatever
phil> you are trying to do.
>>
phil> (If you can't fit all the data in memory, what do you expect
phil> a web browser to do with it? Concurrent requests?)
>>
phil> Anyway, you are right about the lack of clean separation
phil> between your MVC layers. Ideally, you would just pass the
phil> data in the request scope from your M to your C to your V.
phil> However, you've already pointed out that it's too big. To
phil> that, we all suggested an Iterator. And you've rightly
phil> poined out that you can't really pass an Iterator into your
phil> view, because the broker handling the underlying Connection
phil> will likely be closed at that point.
>>
phil> So, what you could do is make a servlet filter that does
phil> this:
>>
phil> public void doFilter(ServletRequest request, ServletResponse
phil> response, FilterChain chain) throws IOException,
phil> ServletException {
>>
phil> //Before...
phil> MyAppFacade facade = new MyAppFacade (); Iterator i =
phil> facade.getLotsAndLotsOfDataAsIterator();
>>
phil> //Put an Iterator in the request.
phil> HttpServletRequest req = (HttpServletRequest) request;
phil> req.setAttribute("mydata", i);
>>
phil> //Continue processing - let the view use the Iterator
phil> chain.doFilter(request, response);
>>
phil> //Close the facade and the underlying broker.
phil> facade.close();
>>
phil> }
>>
phil> Now, I don't necessarily recommend this, but it might do
phil> what you are asking for.
>>
phil> phil.
>>
>>
phil> [ Muliawan Sjarif ] wrote:
>>
>> >> I have already set maxActive=-1 in the first place.
>> >>
>> >> phil's right, Iterator is strongly recommended by OJB team
>> >> to be used if we're dealing with huge resultset..
>> >>
>> >> However, the problem now is the iterator object can't be
>> >> passed onto servlet or struts' view to be processed
>> >> there. The documentation suggests that for that kind of
>> >> purpose use Collection instead. How then?
>> >>
>> >> On Wed, 13 Oct 2004 13:46:31 +1000, Joe Latty
>> >> <[EMAIL PROTECTED]> wrote:
>> >>
>> >>> Without going into why you would want a query to return
>> >>> 350k records, you may have to change some parameters in the
>> >>> OJB.properties file.
>> >>>
>> >>> Look for the following and you may need to change maxActive
>> >>> to a non-positive number. This can give out of memory
>> >>> errors.
>> >>>
>> >>> # maximum number of brokers that can be borrowed from the
>> >>> # pool at one time. When non-positive, there is no limit.
>> >>> maxActive=100
>> >>>
>> >>>
>> >>>
>> >>> -----Original Message----- From: WHIRLYCOTT
>> >>> [mailto:[EMAIL PROTECTED] Sent: Wednesday, 13 October
>> >>> 2004 12:02 PM To: OJB Users List Subject: Re: Query problem
>> >>> for hundreds of thousands of records
>> >>>
>> >>> If you need the entire resultset in memory, you probably
>> >>> need to provide
>> >>>
>> >>> more memory to your jvm or add more memory physically to
>> >>> your hardware.
>> >>>
>> >>> It sounds like, in effect, that you are basically making a
>> >>> copy of your database table with ~350k records in memory!
>> >>>
>> >>> What happens when 2 users hit your website at the same
>> >>> time? What happens when 100 users hit your site?
>> >>>
>> >>> I don't know the specifics of what you are doing, but it
>> >>> doesn't sound like a particularily good solution. Why do
>> >>> you think you need to have the entire result set in memory?
>> >>>
>> >>> phil.
>> >>>
>> >>> [ Muliawan Sjarif ] wrote:
>> >>>
>> >>>
>> >>>> If I need the entire resultset in the memory what would be
>> >>>> the best approach? It seems that
>> >>>> broker.getCollectionByQuery(q) stuck with huge records.
>> >>>>
>> >>>>
>> >>>> On Tue, 12 Oct 2004 16:09:11 -0400, WHIRLYCOTT
>> >>>> <[EMAIL PROTECTED]>
>> >>>
>> >>> wrote:
>> >>>
>> >>>>> You probably are doing something like this:
>> >>>>>
>> >>>>> Collection c = broker.getCollectionByQuery(q);
>> >>>>>
>> >>>>> ... and ought to be doing something like this:
>> >>>>>
>> >>>>> Iterator i = broker.getIteratorByQuery(q);
>> >>>>>
>> >>>>> Unless you need the entire result set in memory, this
>> >>>>> should be your preferred approach. Just iterate through
>> >>>>> the results and do what you need to do to each of them.
>> >>>>>
>> >>>>> phil.
>> >>>>>
>> >>>>> [ Muliawan Sjarif ] wrote:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> I was doing a quick test for OJB particularly using
>> >>>>>> PersistenceBroker on one table that consists of 352544
>> >>>>>> records. No matter how many
>> >>>
>> >>> times
>> >>>
>> >>>>>> I tried, in the end all I got was the browser not
>> >>>>>> responding (no result displayed on the page) and tomcat
>> >>>>>> throwed
>> >>>
>> >>> java.lang.OutOfMemory
>> >>>
>> >>>>>> exception. For this test, dynamic proxy was utilized.
>> >>>>>>
>> >>>>>> Pertaining to this problem, I wonder if anyone has
>> >>>>>> encountered
>> >>>
>> >>> similar
>> >>>
>> >>>>>> problem. Or is there somewhere in the OJB's properties
>> >>>>>> that need to
>> >>>
>> >>> be
>> >>>
>> >>>>>> tweaked?
>> >>>>>>
>> >>>>>> Many thanks.
>> >>>>>>
>> >>>>>> Regards, Muliawan
>> >>>>>
>> >>>>> -- Whirlycott Philip Jacob [EMAIL PROTECTED]
>> >>>>> http://www.whirlycott.com/phil/
>> >>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail:
>> >>>>> [EMAIL PROTECTED] For additional
>> >>>>> commands, e-mail: [EMAIL PROTECTED]
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>> -- Whirlycott Philip Jacob [EMAIL PROTECTED]
>> >>> http://www.whirlycott.com/phil/
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> >>> For additional commands, e-mail:
>> >>> [EMAIL PROTECTED]
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> >>> For additional commands, e-mail:
>> >>> [EMAIL PROTECTED]
>> >>>
>> >>>
>> >>
>> >>
>> >>
>>
phil> -- Whirlycott Philip Jacob [EMAIL PROTECTED]
phil> http://www.whirlycott.com/phil/
>>
phil> ---------------------------------------------------------------------
phil> To unsubscribe, e-mail: [EMAIL PROTECTED]
phil> For additional commands, e-mail: [EMAIL PROTECTED]
>>
phil> --
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED] For
>> additional commands, e-mail: [EMAIL PROTECTED]
>>
phil> --
phil> Whirlycott Philip Jacob
phil> [EMAIL PROTECTED]
phil> http://www.whirlycott.com/phil/
phil> ---------------------------------------------------------------------
phil> To unsubscribe, e-mail: [EMAIL PROTECTED]
phil> For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
