Unfortunately, the business gets to dictate this one, and they maintain that these large clients need to ability to view up to 90 days worth of data. I agree with you that this is logistically impossible, but since the business requirements state that it must be so, my QC group is testing it and saying that it fails. If it fails, it's got to be fixed.....
Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] > -----Original Message----- > From: Niall Pemberton [mailto:[EMAIL PROTECTED] > Sent: Thursday, March 04, 2004 11:17 AM > To: Struts Users Mailing List > Subject: Re: Best way to handle big search results.. > > > Even if everything in your system handles 320,000 lines of > html, your users > never will. This isn't a performance issue, its a design one. > You gotta go > back to the drawing board and stop trying to get something to > perform that > your users will never accept. > > Niall > > ----- Original Message ----- > From: "Jerry Jalenak" <[EMAIL PROTECTED]> > To: "'Struts Users Mailing List'" <[EMAIL PROTECTED]> > Sent: Thursday, March 04, 2004 5:12 PM > Subject: RE: Best way to handle big search results.. > > > > But..... I'm already doing paging. Each independant > account table only > > returns the first 10 detail records; everything is > available by paging > > through the next 'n' pages. For example, one of my clients > has access to > > almost 8900 accounts. Of these accounts, there are about 3500 with > reported > > data in the past 90 days. In the past month alone, these > 3500 accounts > have > > almost 48000 detail records. So I'm returning (probably > not all 3500 > > accounts, but a good majority of 'em) say 2000 tables, with > potentially 10 > > detail records per table. Each table has seven columns, so > just for the > > tables I'm returning about 320,000 lines of html. I'm > really starting to > > think this is a client side rendering issue - the browser > simply can't > > process the number of lines of html that I'm pushing it..... > > > > Jerry Jalenak > > Development Manager, Web Publishing > > LabOne, Inc. > > 10101 Renner Blvd. > > Lenexa, KS 66219 > > (913) 577-1496 > > > > [EMAIL PROTECTED] > > > > > > > -----Original Message----- > > > From: Hookom, Jacob [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, March 04, 2004 11:03 AM > > > To: Struts Users Mailing List > > > Subject: RE: Best way to handle big search results.. > > > > > > > > > Jerry, > > > > > > We ran into the same problems you are having, and then there > > > reaches a point > > > where you leave it according to what the requirements say, > > > and when people > > > complain, you tell them it's because it not only takes a > > > while at the server > > > to create the HTML for so many records, but it also takes > > > just as long to > > > render them in your browser. Then they say, "Really?" and > > > you say, "Yes, > > > this is why we should do paging," to which they reply, > "then we should > > > probably add it in there." > > > > > > -Jake > > > > > > -----Original Message----- > > > From: Jerry Jalenak [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, March 04, 2004 10:58 AM > > > To: 'Struts Users Mailing List' > > > Subject: RE: Best way to handle big search results.. > > > > > > Sorry for the misunderstanding - are you asking how the > > > request is made from > > > the user to the app? or from the app to the database? > > > > > > Jerry Jalenak > > > Development Manager, Web Publishing > > > LabOne, Inc. > > > 10101 Renner Blvd. > > > Lenexa, KS 66219 > > > (913) 577-1496 > > > > > > [EMAIL PROTECTED] > > > > > > > > > > -----Original Message----- > > > > From: Mark Lowe [mailto:[EMAIL PROTECTED] > > > > Sent: Thursday, March 04, 2004 10:55 AM > > > > To: Struts Users Mailing List > > > > Subject: Re: Best way to handle big search results.. > > > > > > > > > > > > Sure but how do does the query get made? > > > > > > > > > > > > > > > > On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: > > > > > > > > > Mark, > > > > > > > > > > Let me give you some background first. On any given day > > > I will be > > > > > reporting > > > > > on about 80,000 accounts with over 300,000 detail records > > > available > > > > > for the > > > > > past 90 days. My clients are using this data to determine > > > > employment > > > > > elgibility, etc. The majority of these clients are individual > > > > > employers > > > > > that might have between 1 and 100 accounts. These are no > > > > problem. The > > > > > 'problem' clients are organizations that handle this type > > > > of work for > > > > > multiple employers on a contract basis. These clients can > > > > and do have > > > > > several thousand accounts that could *potentially* > have data that > > > > > needs to > > > > > be reported. > > > > > > > > > > The approach I took was to write a separate servlet that > > > > wakes up every > > > > > hours, re-sweeps the account code table, re-sweeps > the data table, > > > > > reconciles these into a set of nested maps, and places it into > > > > > application > > > > > scope. Part of this process throws out any account that > > > > doesn't have > > > > > data > > > > > to be reported, otherwise I'd be trying to handle 200,000 > > > > accounts. > > > > > Based > > > > > on the account and type of data the user can 'see', they > > > > are presented > > > > > a > > > > > list of accounts where they can pick one, many, or all. The > > > > > application > > > > > then accesses the map structure in application scope, > extracts the > > > > > appropriate data based on account, date range, etc., and > > > > returns a set > > > > > of > > > > > <display /> tables to the user. These table are > required to be > > > > > independantly sortable, pagable, etc. Again, for up > to 100 or so > > > > > accounts, > > > > > this is extremely fast. It just goes down the toilet > > > once I start > > > > > getting > > > > > over about 500 accounts..... > > > > > > > > > > Jerry Jalenak > > > > > Development Manager, Web Publishing > > > > > LabOne, Inc. > > > > > 10101 Renner Blvd. > > > > > Lenexa, KS 66219 > > > > > (913) 577-1496 > > > > > > > > > > [EMAIL PROTECTED] > > > > > > > > > > > > > > >> -----Original Message----- > > > > >> From: Mark Lowe [mailto:[EMAIL PROTECTED] > > > > >> Sent: Thursday, March 04, 2004 10:39 AM > > > > >> To: Struts Users Mailing List > > > > >> Subject: Re: Best way to handle big search results.. > > > > >> > > > > >> > > > > >> Sound's like you'll need some scrolling mechanism in > > > > between this can > > > > >> range from changing a query string for a jdbc type app > > > or using an > > > > >> object model like hibernate which supports result > > > > scrolling.. I'd say > > > > >> with reasonable confidence that returning 48,000 records in > > > > >> one go is a > > > > >> pretty unreasonable thing to expect. And who's gonna read > > > > them all in > > > > >> one go? :o) > > > > >> > > > > >> Lets start at the beginning. How are you retrieving the > > > > >> records at the > > > > >> moment? > > > > >> > > > > >> On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: > > > > >> > > > > >>> In my application I often return large amounts of data - > > > > >> for instance, > > > > >>> a > > > > >>> single user may have access to as many as 8500 accounts; > > > > >> each account > > > > >>> may > > > > >>> have several hundred detail records associated with it. > > > > >> I've solved my > > > > >>> initial problem of being able to return thousands of > > > records (for > > > > >>> instance, > > > > >>> I can return almost 48,000 detail records in a > little under 5 > > > > >>> seconds), but > > > > >>> the problem I have now is that it takes FOREVER for the jsp > > > > >> to render. > > > > >>> I > > > > >>> mean, I've waited over an hour for this thing to > complete. Does > > > > >>> anyone have > > > > >>> any tips on improving / optimizing the performance of a > > > > >> compiled JSP? > > > > >>> I'm > > > > >>> using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02.... > > > > >>> > > > > >>> Thanks! > > > > >>> > > > > >>> Jerry Jalenak > > > > >>> Development Manager, Web Publishing > > > > >>> LabOne, Inc. > > > > >>> 10101 Renner Blvd. > > > > >>> Lenexa, KS 66219 > > > > >>> (913) 577-1496 > > > > >>> > > > > >>> [EMAIL PROTECTED] > > > > >>> > > > > >>> > > > > >>>> -----Original Message----- > > > > >>>> From: Hookom, Jacob > [mailto:[EMAIL PROTECTED] > > > > >>>> Sent: Wednesday, March 03, 2004 1:39 PM > > > > >>>> To: Struts Users Mailing List > > > > >>>> Subject: RE: Best way to handle big search results.. > > > > >>>> > > > > >>>> > > > > >>>> We have to work with search results that are > sometimes 1000+ > > > > >>>> items in size. > > > > >>>> In addition, much of the information we have is > not keyed so > > > > >>>> we cannot say > > > > >>>> give results between id 2000 and id 20020. > > > > >>>> > > > > >>>> Some things I found to help with search results > where we did > > > > >>>> do in memory > > > > >>>> sorting/paging were: > > > > >>>> > > > > >>>> 1. Create view-specific lightweight objects to be pulled > > > > >> from the DB > > > > >>>> 2. Memory allowed, it's faster to pull them all at once > > > > >> than lazy load > > > > >>>> 4. The real cause of some of our performance problems were > > > > >>>> the JSP's/HTML in > > > > >>>> rendering a huge list to the client vs. only showing > > > 20 at once. > > > > >>>> > > > > >>>> > > > > >>>> To handle this, I created a SearchResultListAdaptor that > > > > >>>> could sit in the > > > > >>>> session and handle paging (if anyone wants to argue session > > > > >>>> scope with me, > > > > >>>> bring it on--). The SearchResultListAdaptor then contained > > > > >>>> the behavior of > > > > >>>> sort order/paging, etc. Sorting was done via bean > > > > property using a > > > > >>>> BeanUtilsComparator. So my SearchResultListAdaptor could > > > > >>>> then work with an > > > > >>>> list of beans. > > > > >>>> > > > > >>>> Best Regards, > > > > >>>> Jacob > > > > >>>> > > > > >>>> -----Original Message----- > > > > >>>> From: Arne Brutschy > [mailto:[EMAIL PROTECTED] > > > > >>>> Sent: Wednesday, March 03, 2004 12:57 PM > > > > >>>> To: Struts Users Mailing List > > > > >>>> Subject: Best way to handle big search results.. > > > > >>>> > > > > >>>> Hi, > > > > >>>> > > > > >>>> I'm looking for the best way to handle big search results. > > > > >>>> > > > > >>>> Setting: the user can search for other users in the ldap > > > > >>>> directory. This > > > > >>>> might return a long list of results (~40.000 entrys). > > > > >> Sadly, OpenLDAP > > > > >>>> doesn't support server-side sorting, so I have to sort > > > > the results > > > > >>>> myself. I want to display 20 items per page, and the user > > > > >> can browse > > > > >>>> through these results. > > > > >>>> > > > > >>>> What is the best way to handle these result object? > > > > >> Sorting once and > > > > >>>> storing/caching it in the session or searching and sorting > > > > >> every time > > > > >>>> the user requests a new page of results? > > > > >>>> > > > > >>>> Any thoughts/experiences? > > > > >>>> > > > > >>>> Regards, > > > > >>>> Arne Brutschy > > > > >>>> > > > > >>>> > > > > >>>> > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > >>>> To unsubscribe, e-mail: > > > > [EMAIL PROTECTED] > > > > >>>> For additional commands, e-mail: > > > > >> [EMAIL PROTECTED] > > > > >>>> > > > > >>>> > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > >>>> To unsubscribe, e-mail: > > > > [EMAIL PROTECTED] > > > > >>>> For additional commands, e-mail: > > > > >> [EMAIL PROTECTED] > > > > >>>> > > > > >>>> > > > > >>> > > > > >>> This transmission (and any information attached to > it) may be > > > > >>> confidential and > > > > >>> is intended solely for the use of the individual or entity > > > > >> to which it > > > > >>> is > > > > >>> addressed. If you are not the intended recipient or > the person > > > > >>> responsible for > > > > >>> delivering the transmission to the intended recipient, be > > > > >> advised that > > > > >>> you > > > > >>> have received this transmission in error and that any use, > > > > >>> dissemination, > > > > >>> forwarding, printing, or copying of this > information is strictly > > > > >>> prohibited. > > > > >>> If you have received this transmission in error, please > > > > immediately > > > > >>> notify > > > > >>> LabOne at the following email address: > > > > >>> [EMAIL PROTECTED] > > > > >>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > >>> To unsubscribe, e-mail: > > > [EMAIL PROTECTED] > > > > >>> For additional commands, e-mail: > > > > [EMAIL PROTECTED] > > > > >>> > > > > >> > > > > >> > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > >> To unsubscribe, e-mail: > > > [EMAIL PROTECTED] > > > > >> For additional commands, e-mail: > > > > [EMAIL PROTECTED] > > > > >> > > > > >> > > > > > > > > > > This transmission (and any information attached to it) may be > > > > > confidential and > > > > > is intended solely for the use of the individual or entity > > > > to which it > > > > > is > > > > > addressed. If you are not the intended recipient or the person > > > > > responsible for > > > > > delivering the transmission to the intended recipient, be > > > > advised that > > > > > you > > > > > have received this transmission in error and that any use, > > > > > dissemination, > > > > > forwarding, printing, or copying of this information > is strictly > > > > > prohibited. > > > > > If you have received this transmission in error, please > > > immediately > > > > > notify > > > > > LabOne at the following email address: > > > > > [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: > [EMAIL PROTECTED] > > > > > For additional commands, e-mail: > > > [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: > [EMAIL PROTECTED] > > > > For additional commands, e-mail: > [EMAIL PROTECTED] > > > > > > > > > > > > > > This transmission (and any information attached to it) may be > > > confidential > > > and > > > is intended solely for the use of the individual or entity to > > > which it is > > > addressed. If you are not the intended recipient or the > > > person responsible > > > for > > > delivering the transmission to the intended recipient, be > > > advised that you > > > have received this transmission in error and that any use, > > > dissemination, > > > forwarding, printing, or copying of this information is > > > strictly prohibited. > > > If you have received this transmission in error, please > > > immediately notify > > > LabOne at the following email address: > > > [EMAIL PROTECTED] > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: > [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: > [EMAIL PROTECTED] > > > > > > > > > > This transmission (and any information attached to it) may > be confidential > and > > is intended solely for the use of the individual or entity > to which it is > > addressed. If you are not the intended recipient or the > person responsible > for > > delivering the transmission to the intended recipient, be > advised that you > > have received this transmission in error and that any use, > dissemination, > > forwarding, printing, or copying of this information is strictly > prohibited. > > If you have received this transmission in error, please > immediately notify > > LabOne at the following email address: > [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
