Even if everything in your system handles 320,000 lines of html, your users never will. This isn't a performance issue, its a design one. You gotta go back to the drawing board and stop trying to get something to perform that your users will never accept.
Niall ----- Original Message ----- From: "Jerry Jalenak" <[EMAIL PROTECTED]> To: "'Struts Users Mailing List'" <[EMAIL PROTECTED]> Sent: Thursday, March 04, 2004 5:12 PM Subject: RE: Best way to handle big search results.. > But..... I'm already doing paging. Each independant account table only > returns the first 10 detail records; everything is available by paging > through the next 'n' pages. For example, one of my clients has access to > almost 8900 accounts. Of these accounts, there are about 3500 with reported > data in the past 90 days. In the past month alone, these 3500 accounts have > almost 48000 detail records. So I'm returning (probably not all 3500 > accounts, but a good majority of 'em) say 2000 tables, with potentially 10 > detail records per table. Each table has seven columns, so just for the > tables I'm returning about 320,000 lines of html. I'm really starting to > think this is a client side rendering issue - the browser simply can't > process the number of lines of html that I'm pushing it..... > > Jerry Jalenak > Development Manager, Web Publishing > LabOne, Inc. > 10101 Renner Blvd. > Lenexa, KS 66219 > (913) 577-1496 > > [EMAIL PROTECTED] > > > > -----Original Message----- > > From: Hookom, Jacob [mailto:[EMAIL PROTECTED] > > Sent: Thursday, March 04, 2004 11:03 AM > > To: Struts Users Mailing List > > Subject: RE: Best way to handle big search results.. > > > > > > Jerry, > > > > We ran into the same problems you are having, and then there > > reaches a point > > where you leave it according to what the requirements say, > > and when people > > complain, you tell them it's because it not only takes a > > while at the server > > to create the HTML for so many records, but it also takes > > just as long to > > render them in your browser. Then they say, "Really?" and > > you say, "Yes, > > this is why we should do paging," to which they reply, "then we should > > probably add it in there." > > > > -Jake > > > > -----Original Message----- > > From: Jerry Jalenak [mailto:[EMAIL PROTECTED] > > Sent: Thursday, March 04, 2004 10:58 AM > > To: 'Struts Users Mailing List' > > Subject: RE: Best way to handle big search results.. > > > > Sorry for the misunderstanding - are you asking how the > > request is made from > > the user to the app? or from the app to the database? > > > > Jerry Jalenak > > Development Manager, Web Publishing > > LabOne, Inc. > > 10101 Renner Blvd. > > Lenexa, KS 66219 > > (913) 577-1496 > > > > [EMAIL PROTECTED] > > > > > > > -----Original Message----- > > > From: Mark Lowe [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, March 04, 2004 10:55 AM > > > To: Struts Users Mailing List > > > Subject: Re: Best way to handle big search results.. > > > > > > > > > Sure but how do does the query get made? > > > > > > > > > > > > On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: > > > > > > > Mark, > > > > > > > > Let me give you some background first. On any given day > > I will be > > > > reporting > > > > on about 80,000 accounts with over 300,000 detail records > > available > > > > for the > > > > past 90 days. My clients are using this data to determine > > > employment > > > > elgibility, etc. The majority of these clients are individual > > > > employers > > > > that might have between 1 and 100 accounts. These are no > > > problem. The > > > > 'problem' clients are organizations that handle this type > > > of work for > > > > multiple employers on a contract basis. These clients can > > > and do have > > > > several thousand accounts that could *potentially* have data that > > > > needs to > > > > be reported. > > > > > > > > The approach I took was to write a separate servlet that > > > wakes up every > > > > hours, re-sweeps the account code table, re-sweeps the data table, > > > > reconciles these into a set of nested maps, and places it into > > > > application > > > > scope. Part of this process throws out any account that > > > doesn't have > > > > data > > > > to be reported, otherwise I'd be trying to handle 200,000 > > > accounts. > > > > Based > > > > on the account and type of data the user can 'see', they > > > are presented > > > > a > > > > list of accounts where they can pick one, many, or all. The > > > > application > > > > then accesses the map structure in application scope, extracts the > > > > appropriate data based on account, date range, etc., and > > > returns a set > > > > of > > > > <display /> tables to the user. These table are required to be > > > > independantly sortable, pagable, etc. Again, for up to 100 or so > > > > accounts, > > > > this is extremely fast. It just goes down the toilet > > once I start > > > > getting > > > > over about 500 accounts..... > > > > > > > > Jerry Jalenak > > > > Development Manager, Web Publishing > > > > LabOne, Inc. > > > > 10101 Renner Blvd. > > > > Lenexa, KS 66219 > > > > (913) 577-1496 > > > > > > > > [EMAIL PROTECTED] > > > > > > > > > > > >> -----Original Message----- > > > >> From: Mark Lowe [mailto:[EMAIL PROTECTED] > > > >> Sent: Thursday, March 04, 2004 10:39 AM > > > >> To: Struts Users Mailing List > > > >> Subject: Re: Best way to handle big search results.. > > > >> > > > >> > > > >> Sound's like you'll need some scrolling mechanism in > > > between this can > > > >> range from changing a query string for a jdbc type app > > or using an > > > >> object model like hibernate which supports result > > > scrolling.. I'd say > > > >> with reasonable confidence that returning 48,000 records in > > > >> one go is a > > > >> pretty unreasonable thing to expect. And who's gonna read > > > them all in > > > >> one go? :o) > > > >> > > > >> Lets start at the beginning. How are you retrieving the > > > >> records at the > > > >> moment? > > > >> > > > >> On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: > > > >> > > > >>> In my application I often return large amounts of data - > > > >> for instance, > > > >>> a > > > >>> single user may have access to as many as 8500 accounts; > > > >> each account > > > >>> may > > > >>> have several hundred detail records associated with it. > > > >> I've solved my > > > >>> initial problem of being able to return thousands of > > records (for > > > >>> instance, > > > >>> I can return almost 48,000 detail records in a little under 5 > > > >>> seconds), but > > > >>> the problem I have now is that it takes FOREVER for the jsp > > > >> to render. > > > >>> I > > > >>> mean, I've waited over an hour for this thing to complete. Does > > > >>> anyone have > > > >>> any tips on improving / optimizing the performance of a > > > >> compiled JSP? > > > >>> I'm > > > >>> using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02.... > > > >>> > > > >>> Thanks! > > > >>> > > > >>> Jerry Jalenak > > > >>> Development Manager, Web Publishing > > > >>> LabOne, Inc. > > > >>> 10101 Renner Blvd. > > > >>> Lenexa, KS 66219 > > > >>> (913) 577-1496 > > > >>> > > > >>> [EMAIL PROTECTED] > > > >>> > > > >>> > > > >>>> -----Original Message----- > > > >>>> From: Hookom, Jacob [mailto:[EMAIL PROTECTED] > > > >>>> Sent: Wednesday, March 03, 2004 1:39 PM > > > >>>> To: Struts Users Mailing List > > > >>>> Subject: RE: Best way to handle big search results.. > > > >>>> > > > >>>> > > > >>>> We have to work with search results that are sometimes 1000+ > > > >>>> items in size. > > > >>>> In addition, much of the information we have is not keyed so > > > >>>> we cannot say > > > >>>> give results between id 2000 and id 20020. > > > >>>> > > > >>>> Some things I found to help with search results where we did > > > >>>> do in memory > > > >>>> sorting/paging were: > > > >>>> > > > >>>> 1. Create view-specific lightweight objects to be pulled > > > >> from the DB > > > >>>> 2. Memory allowed, it's faster to pull them all at once > > > >> than lazy load > > > >>>> 4. The real cause of some of our performance problems were > > > >>>> the JSP's/HTML in > > > >>>> rendering a huge list to the client vs. only showing > > 20 at once. > > > >>>> > > > >>>> > > > >>>> To handle this, I created a SearchResultListAdaptor that > > > >>>> could sit in the > > > >>>> session and handle paging (if anyone wants to argue session > > > >>>> scope with me, > > > >>>> bring it on--). The SearchResultListAdaptor then contained > > > >>>> the behavior of > > > >>>> sort order/paging, etc. Sorting was done via bean > > > property using a > > > >>>> BeanUtilsComparator. So my SearchResultListAdaptor could > > > >>>> then work with an > > > >>>> list of beans. > > > >>>> > > > >>>> Best Regards, > > > >>>> Jacob > > > >>>> > > > >>>> -----Original Message----- > > > >>>> From: Arne Brutschy [mailto:[EMAIL PROTECTED] > > > >>>> Sent: Wednesday, March 03, 2004 12:57 PM > > > >>>> To: Struts Users Mailing List > > > >>>> Subject: Best way to handle big search results.. > > > >>>> > > > >>>> Hi, > > > >>>> > > > >>>> I'm looking for the best way to handle big search results. > > > >>>> > > > >>>> Setting: the user can search for other users in the ldap > > > >>>> directory. This > > > >>>> might return a long list of results (~40.000 entrys). > > > >> Sadly, OpenLDAP > > > >>>> doesn't support server-side sorting, so I have to sort > > > the results > > > >>>> myself. I want to display 20 items per page, and the user > > > >> can browse > > > >>>> through these results. > > > >>>> > > > >>>> What is the best way to handle these result object? > > > >> Sorting once and > > > >>>> storing/caching it in the session or searching and sorting > > > >> every time > > > >>>> the user requests a new page of results? > > > >>>> > > > >>>> Any thoughts/experiences? > > > >>>> > > > >>>> Regards, > > > >>>> Arne Brutschy > > > >>>> > > > >>>> > > > >>>> > > > >> > > > > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: > > > [EMAIL PROTECTED] > > > >>>> For additional commands, e-mail: > > > >> [EMAIL PROTECTED] > > > >>>> > > > >>>> > > > >> > > > > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: > > > [EMAIL PROTECTED] > > > >>>> For additional commands, e-mail: > > > >> [EMAIL PROTECTED] > > > >>>> > > > >>>> > > > >>> > > > >>> This transmission (and any information attached to it) may be > > > >>> confidential and > > > >>> is intended solely for the use of the individual or entity > > > >> to which it > > > >>> is > > > >>> addressed. If you are not the intended recipient or the person > > > >>> responsible for > > > >>> delivering the transmission to the intended recipient, be > > > >> advised that > > > >>> you > > > >>> have received this transmission in error and that any use, > > > >>> dissemination, > > > >>> forwarding, printing, or copying of this information is strictly > > > >>> prohibited. > > > >>> If you have received this transmission in error, please > > > immediately > > > >>> notify > > > >>> LabOne at the following email address: > > > >>> [EMAIL PROTECTED] > > > >>> > > > >>> > > > >>> > > > >> > > > > > --------------------------------------------------------------------- > > > >>> To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > > >>> For additional commands, e-mail: > > > [EMAIL PROTECTED] > > > >>> > > > >> > > > >> > > > >> > > > > > --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > > >> For additional commands, e-mail: > > > [EMAIL PROTECTED] > > > >> > > > >> > > > > > > > > This transmission (and any information attached to it) may be > > > > confidential and > > > > is intended solely for the use of the individual or entity > > > to which it > > > > is > > > > addressed. If you are not the intended recipient or the person > > > > responsible for > > > > delivering the transmission to the intended recipient, be > > > advised that > > > > you > > > > have received this transmission in error and that any use, > > > > dissemination, > > > > forwarding, printing, or copying of this information is strictly > > > > prohibited. > > > > If you have received this transmission in error, please > > immediately > > > > notify > > > > LabOne at the following email address: > > > > [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > This transmission (and any information attached to it) may be > > confidential > > and > > is intended solely for the use of the individual or entity to > > which it is > > addressed. If you are not the intended recipient or the > > person responsible > > for > > delivering the transmission to the intended recipient, be > > advised that you > > have received this transmission in error and that any use, > > dissemination, > > forwarding, printing, or copying of this information is > > strictly prohibited. > > If you have received this transmission in error, please > > immediately notify > > LabOne at the following email address: > > [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > This transmission (and any information attached to it) may be confidential and > is intended solely for the use of the individual or entity to which it is > addressed. If you are not the intended recipient or the person responsible for > delivering the transmission to the intended recipient, be advised that you > have received this transmission in error and that any use, dissemination, > forwarding, printing, or copying of this information is strictly prohibited. > If you have received this transmission in error, please immediately notify > LabOne at the following email address: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
