On 4 Mar 2004, at 17:49, Jerry Jalenak wrote:
Mark,
Let me give you some background first. On any given day I will be reporting
on about 80,000 accounts with over 300,000 detail records available for the
past 90 days. My clients are using this data to determine employment
elgibility, etc. The majority of these clients are individual employers
that might have between 1 and 100 accounts. These are no problem. The
'problem' clients are organizations that handle this type of work for
multiple employers on a contract basis. These clients can and do have
several thousand accounts that could *potentially* have data that needs to
be reported.
The approach I took was to write a separate servlet that wakes up every
hours, re-sweeps the account code table, re-sweeps the data table,
reconciles these into a set of nested maps, and places it into application
scope. Part of this process throws out any account that doesn't have data
to be reported, otherwise I'd be trying to handle 200,000 accounts. Based
on the account and type of data the user can 'see', they are presented a
list of accounts where they can pick one, many, or all. The application
then accesses the map structure in application scope, extracts the
appropriate data based on account, date range, etc., and returns a set of
<display /> tables to the user. These table are required to be
independantly sortable, pagable, etc. Again, for up to 100 or so accounts,
this is extremely fast. It just goes down the toilet once I start getting
over about 500 accounts.....
Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496
[EMAIL PROTECTED]
-----Original Message----- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results..
Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o)
Lets start at the beginning. How are you retrieving the records at the moment?
On 4 Mar 2004, at 17:20, Jerry Jalenak wrote:
In my application I often return large amounts of data -for instance,each accounta single user may have access to as many as 8500 accounts;I've solved mymay have several hundred detail records associated with it.to render.initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jspcompiled JSP?I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of afrom the DBI'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02....
Thanks!
Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496
[EMAIL PROTECTED]
-----Original Message----- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results..
We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020.
Some things I found to help with search results where we did do in memory sorting/paging were:
1. Create view-specific lightweight objects to be pulledthan lazy load2. Memory allowed, it's faster to pull them all at onceSadly, OpenLDAP4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once.
To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans.
Best Regards, Jacob
-----Original Message----- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results..
Hi,
I'm looking for the best way to handle big search results.
Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys).can browsedoesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the userSorting once andthrough these results.
What is the best way to handle these result object?every timestoring/caching it in the session or searching and sorting---------------------------------------------------------------------the user requests a new page of results?
Any thoughts/experiences?
Regards, Arne Brutschy
[EMAIL PROTECTED]To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail:---------------------------------------------------------------------
[EMAIL PROTECTED]To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail:to which it
This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entityadvised thatis addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be---------------------------------------------------------------------you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED]
To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
This transmission (and any information attached to it) may be confidential and
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient or the person responsible for
delivering the transmission to the intended recipient, be advised that you
have received this transmission in error and that any use, dissemination,
forwarding, printing, or copying of this information is strictly prohibited.
If you have received this transmission in error, please immediately notify
LabOne at the following email address: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
