Another alternative is for the database layer to wrap the Iterator and
to defer calling broker.close(). The wrapper then either closes the
broker in next() when the underlying Iterator is exhausted, or sets a
timeout and closes the broker when it expires. I've used this
technique to pass OJB's result Iterators to my view layer. Here's
pseudocode to illustrate:
queryResults = doQuery(broker, ...);
return new BrokerClosingIterator(broker, queryResults);
}
class BrokerClosingIterator {
public BrokerClosingIterator(PersistenceBroker broker, Iterator iter) {
// optionally set timeout
}
public boolean hasNext() {
boolean more = iter.hasNext();
if (!more && (broker != null)) {
broker.close();
broker = null;
}
return more;
}
public Object next() {
return iter.next();
}
}
Steve Clark
Technology Applications Team
Natural Resources Research Center/USGS
[EMAIL PROTECTED]
(970)226-9291
>>>>> Phil Whirlycott writes:
phil> I was thinking about this some more on the way to work
phil> today, but before I offer up the idea I had, I will
phil> first re-iterate that this is probably a bad solution
phil> to whatever you are trying to do.
phil> (If you can't fit all the data in memory, what do you
phil> expect a web browser to do with it? Concurrent
phil> requests?)
phil> Anyway, you are right about the lack of clean
phil> separation between your MVC layers. Ideally, you
phil> would just pass the data in the request scope from
phil> your M to your C to your V. However, you've already
phil> pointed out that it's too big. To that, we all
phil> suggested an Iterator. And you've rightly poined out
phil> that you can't really pass an Iterator into your view,
phil> because the broker handling the underlying Connection
phil> will likely be closed at that point.
phil> So, what you could do is make a servlet filter that
phil> does this:
phil> public void doFilter(ServletRequest request,
phil> ServletResponse response, FilterChain chain) throws
phil> IOException, ServletException {
phil> //Before...
phil> MyAppFacade facade = new MyAppFacade ();
phil> Iterator i =
phil> facade.getLotsAndLotsOfDataAsIterator();
phil> //Put an Iterator in the request.
phil> HttpServletRequest req = (HttpServletRequest)
phil> request; req.setAttribute("mydata", i);
phil> //Continue processing - let the view use the
phil> //Iterator
phil> chain.doFilter(request, response);
phil> //Close the facade and the underlying broker.
phil> facade.close();
phil> }
phil> Now, I don't necessarily recommend this, but it might
phil> do what you are asking for.
phil> phil.
phil> [ Muliawan Sjarif ] wrote:
>> I have already set maxActive=-1 in the first place.
>>
>> phil's right, Iterator is strongly recommended by OJB team to
>> be used if we're dealing with huge resultset..
>>
>> However, the problem now is the iterator object can't be passed
>> onto servlet or struts' view to be processed there. The
>> documentation suggests that for that kind of purpose use
>> Collection instead. How then?
>>
>> On Wed, 13 Oct 2004 13:46:31 +1000, Joe Latty
>> <[EMAIL PROTECTED]> wrote:
>>
>>> Without going into why you would want a query to return 350k
>>> records, you may have to change some parameters in the
>>> OJB.properties file.
>>>
>>> Look for the following and you may need to change maxActive to
>>> a non-positive number. This can give out of memory errors.
>>>
>>> # maximum number of brokers that can be borrowed from the pool
>>> # at one time. When non-positive, there is no limit.
>>> maxActive=100
>>>
>>>
>>>
>>> -----Original Message----- From: WHIRLYCOTT
>>> [mailto:[EMAIL PROTECTED] Sent: Wednesday, 13 October 2004
>>> 12:02 PM To: OJB Users List Subject: Re: Query problem for
>>> hundreds of thousands of records
>>>
>>> If you need the entire resultset in memory, you probably need
>>> to provide
>>>
>>> more memory to your jvm or add more memory physically to your
>>> hardware.
>>>
>>> It sounds like, in effect, that you are basically making a
>>> copy of your database table with ~350k records in memory!
>>>
>>> What happens when 2 users hit your website at the same time?
>>> What happens when 100 users hit your site?
>>>
>>> I don't know the specifics of what you are doing, but it
>>> doesn't sound like a particularily good solution. Why do you
>>> think you need to have the entire result set in memory?
>>>
>>> phil.
>>>
>>> [ Muliawan Sjarif ] wrote:
>>>
>>>
>>>> If I need the entire resultset in the memory what would be
>>>> the best approach? It seems that
>>>> broker.getCollectionByQuery(q) stuck with huge records.
>>>>
>>>>
>>>> On Tue, 12 Oct 2004 16:09:11 -0400, WHIRLYCOTT
>>>> <[EMAIL PROTECTED]>
>>>
>>> wrote:
>>>
>>>>> You probably are doing something like this:
>>>>>
>>>>> Collection c = broker.getCollectionByQuery(q);
>>>>>
>>>>> ... and ought to be doing something like this:
>>>>>
>>>>> Iterator i = broker.getIteratorByQuery(q);
>>>>>
>>>>> Unless you need the entire result set in memory, this should
>>>>> be your preferred approach. Just iterate through the
>>>>> results and do what you need to do to each of them.
>>>>>
>>>>> phil.
>>>>>
>>>>> [ Muliawan Sjarif ] wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I was doing a quick test for OJB particularly using
>>>>>> PersistenceBroker on one table that consists of 352544
>>>>>> records. No matter how many
>>>
>>> times
>>>
>>>>>> I tried, in the end all I got was the browser not
>>>>>> responding (no result displayed on the page) and tomcat
>>>>>> throwed
>>>
>>> java.lang.OutOfMemory
>>>
>>>>>> exception. For this test, dynamic proxy was utilized.
>>>>>>
>>>>>> Pertaining to this problem, I wonder if anyone has
>>>>>> encountered
>>>
>>> similar
>>>
>>>>>> problem. Or is there somewhere in the OJB's properties that
>>>>>> need to
>>>
>>> be
>>>
>>>>>> tweaked?
>>>>>>
>>>>>> Many thanks.
>>>>>>
>>>>>> Regards, Muliawan
>>>>>
>>>>> -- Whirlycott Philip Jacob [EMAIL PROTECTED]
>>>>> http://www.whirlycott.com/phil/
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>> -- Whirlycott Philip Jacob [EMAIL PROTECTED]
>>> http://www.whirlycott.com/phil/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [EMAIL PROTECTED] For
>>> additional commands, e-mail: [EMAIL PROTECTED]
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [EMAIL PROTECTED] For
>>> additional commands, e-mail: [EMAIL PROTECTED]
>>>
>>>
>>
>>
>>
phil> --
phil> Whirlycott Philip
phil> Jacob
phil> [EMAIL PROTECTED]
phil> http://www.whirlycott.com/phil/
phil> ---------------------------------------------------------------------
phil> To unsubscribe, e-mail:
phil> [EMAIL PROTECTED] For additional
phil> commands, e-mail: [EMAIL PROTECTED]
phil> --
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
