Re: Best way to handle big search results..

[Mark Lowe]

I think html rendering is the least of his worries :o)

I'm surprised you can get that many records into the application 
context in the first place.

On 4 Mar 2004, at 18:14, Hookom, Jacob wrote:

Yeah, stay away from tables within tables, they can be a big slow up.

-----Original Message-----
From: Jerry Jalenak [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 11:13 AM
To: 'Struts Users Mailing List'
Subject: RE: Best way to handle big search results..
But..... I'm already doing paging.  Each independant account table only
returns the first 10 detail records; everything is available by paging
through the next 'n' pages.  For example, one of my clients has access 
to
almost 8900 accounts.  Of these accounts, there are about 3500 with 
reported
data in the past 90 days.  In the past month alone, these 3500 
accounts have
almost 48000 detail records.  So I'm returning (probably not all 3500
accounts, but a good majority of 'em) say 2000 tables, with 
potentially 10
detail records per table.  Each table has seven columns, so just for 
the
tables I'm returning about 320,000 lines of html.  I'm really starting 
to
think this is a client side rendering issue - the browser simply can't
process the number of lines of html that I'm pushing it.....

Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Hookom, Jacob [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 11:03 AM
To: Struts Users Mailing List
Subject: RE: Best way to handle big search results..
Jerry,

We ran into the same problems you are having, and then there
reaches a point
where you leave it according to what the requirements say,
and when people
complain, you tell them it's because it not only takes a
while at the server
to create the HTML for so many records, but it also takes
just as long to
render them in your browser.  Then they say, "Really?" and
you say, "Yes,
this is why we should do paging," to which they reply, "then we should
probably add it in there."
-Jake

-----Original Message-----
From: Jerry Jalenak [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:58 AM
To: 'Struts Users Mailing List'
Subject: RE: Best way to handle big search results..
Sorry for the misunderstanding - are you asking how the
request is made from
the user to the app?  or from the app to the database?
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:55 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sure but how do does the query get made?



On 4 Mar 2004, at 17:49, Jerry Jalenak wrote:

Mark,

Let me give you some background first.  On any given day
I will be
reporting
on about 80,000 accounts with over 300,000 detail records
available
for the
past 90 days.  My clients are using this data to determine
employment
elgibility, etc.  The majority of these clients are individual
employers
that might have between 1 and 100 accounts.  These are no
problem.  The
'problem' clients are organizations that handle this type
of work for
multiple employers on a contract basis.  These clients can
and do have
several thousand accounts that could *potentially* have data that
needs to
be reported.
The approach I took was to write a separate servlet that
wakes up every
hours, re-sweeps the account code table, re-sweeps the data table,
reconciles these into a set of nested maps, and places it into
application
scope.  Part of this process throws out any account that
doesn't have
data
to be reported, otherwise I'd be trying to handle 200,000
accounts.
Based
on the account and type of data the user can 'see', they
are presented
a
list of accounts where they can pick one, many, or all.  The
application
then accesses the map structure in application scope, extracts the
appropriate data based on account, date range, etc., and
returns a set
of
<display /> tables to the user.  These table are required to be
independantly sortable, pagable, etc.  Again, for up to 100 or so
accounts,
this is extremely fast.  It just goes down the toilet
once I start
getting
over about 500 accounts.....
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:39 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sound's like you'll need some scrolling mechanism in
between this can
range from changing a query string for a jdbc type app
or using an
object model like hibernate which supports result
scrolling.. I'd say
with reasonable confidence that returning 48,000 records in
one go is a
pretty unreasonable thing to expect. And who's gonna read
them all in
one go? :o)

Lets start at the beginning. How are you retrieving the
records at the
moment?
On 4 Mar 2004, at 17:20, Jerry Jalenak wrote:

In my application I often return large amounts of data -
for instance,
a
single user may have access to as many as 8500 accounts;
each account
may
have several hundred detail records associated with it.
I've solved my
initial problem of being able to return thousands of
records (for
instance,
I can return almost 48,000 detail records in a little under 5
seconds), but
the problem I have now is that it takes FOREVER for the jsp
to render.
 I
mean, I've waited over an hour for this thing to complete.  Does
anyone have
any tips on improving / optimizing the performance of a
compiled JSP?
I'm
using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02....
Thanks!

Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Hookom, Jacob [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 03, 2004 1:39 PM
To: Struts Users Mailing List
Subject: RE: Best way to handle big search results..
We have to work with search results that are sometimes 1000+
items in size.
In addition, much of the information we have is not keyed so
we cannot say
give results between id 2000 and id 20020.
Some things I found to help with search results where we did
do in memory
sorting/paging were:
1. Create view-specific lightweight objects to be pulled
from the DB
2. Memory allowed, it's faster to pull them all at once
than lazy load
4. The real cause of some of our performance problems were
the JSP's/HTML in
rendering a huge list to the client vs. only showing
20 at once.


To handle this, I created a SearchResultListAdaptor that
could sit in the
session and handle paging (if anyone wants to argue session
scope with me,
bring it on--).  The SearchResultListAdaptor then contained
the behavior of
sort order/paging, etc.  Sorting was done via bean
property using a
BeanUtilsComparator.  So my SearchResultListAdaptor could
then work with an
list of beans.
Best Regards,
Jacob
-----Original Message-----
From: Arne Brutschy [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 03, 2004 12:57 PM
To: Struts Users Mailing List
Subject: Best way to handle big search results..
Hi,

I'm looking for the best way to handle big search results.

Setting: the user can search for other users in the ldap
directory. This
might return a long list of results (~40.000 entrys).
Sadly, OpenLDAP
doesn't support server-side sorting, so I have to sort
the results
myself. I want to display 20 items per page, and the user
can browse
through these results.

What is the best way to handle these result object?
Sorting once and
storing/caching it in the session or searching and sorting
every time
the user requests a new page of results?

Any thoughts/experiences?

Regards,
Arne Brutschy




---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]




---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]


This transmission (and any information attached to it) may be
confidential and
is intended solely for the use of the individual or entity
to which it
is
addressed. If you are not the intended recipient or the person
responsible for
delivering the transmission to the intended recipient, be
advised that
you
have received this transmission in error and that any use,
dissemination,
forwarding, printing, or copying of this information is strictly
prohibited.
If you have received this transmission in error, please
immediately
notify
LabOne at the following email address:
[EMAIL PROTECTED]




---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]





---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]


This transmission (and any information attached to it) may be
confidential and
is intended solely for the use of the individual or entity
to which it
is
addressed. If you are not the intended recipient or the person
responsible for
delivering the transmission to the intended recipient, be
advised that
you
have received this transmission in error and that any use,
dissemination,
forwarding, printing, or copying of this information is strictly
prohibited.
If you have received this transmission in error, please
immediately
notify
LabOne at the following email address:
[EMAIL PROTECTED]



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]




---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

This transmission (and any information attached to it) may be
confidential
and
is intended solely for the use of the individual or entity to
which it is
addressed. If you are not the intended recipient or the
person responsible
for
delivering the transmission to the intended recipient, be
advised that you
have received this transmission in error and that any use,
dissemination,
forwarding, printing, or copying of this information is
strictly prohibited.
If you have received this transmission in error, please
immediately notify
LabOne at the following email address:
[EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

This transmission (and any information attached to it) may be 
confidential
and
is intended solely for the use of the individual or entity to which it 
is
addressed. If you are not the intended recipient or the person 
responsible
for
delivering the transmission to the intended recipient, be advised that 
you
have received this transmission in error and that any use, 
dissemination,
forwarding, printing, or copying of this information is strictly 
prohibited.
If you have received this transmission in error, please immediately 
notify
LabOne at the following email address: 
[EMAIL PROTECTED]

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]