Re: Best way to handle big search results..

[Mark Lowe]

Okay ..

You said the results are in the application scope already. so you want 
to scoll through those.

so in you servlet theres something like

List peopleList = goAndGetShitLoadsOfRecordsFromDB();
context.setAttribute("people", peopleList.toArray());
..
<c:forEach var="person" items="${people}" begin="${param.index}" 
end="${param.index + 20">
<c:out value="${person.name}" />
</c:forEach>

<html-el:link page="/scroll.do?index=${param.index + 
20}">Next</html-el:link>

On 4 Mar 2004, at 17:58, Jerry Jalenak wrote:

Sorry for the misunderstanding - are you asking how the request is 
made from
the user to the app?  or from the app to the database?

Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:55 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sure but how do does the query get made?



On 4 Mar 2004, at 17:49, Jerry Jalenak wrote:

Mark,

Let me give you some background first.  On any given day I will be
reporting
on about 80,000 accounts with over 300,000 detail records available
for the
past 90 days.  My clients are using this data to determine
employment
elgibility, etc.  The majority of these clients are individual
employers
that might have between 1 and 100 accounts.  These are no
problem.  The
'problem' clients are organizations that handle this type
of work for
multiple employers on a contract basis.  These clients can
and do have
several thousand accounts that could *potentially* have data that
needs to
be reported.
The approach I took was to write a separate servlet that
wakes up every
hours, re-sweeps the account code table, re-sweeps the data table,
reconciles these into a set of nested maps, and places it into
application
scope.  Part of this process throws out any account that
doesn't have
data
to be reported, otherwise I'd be trying to handle 200,000
accounts.
Based
on the account and type of data the user can 'see', they
are presented
a
list of accounts where they can pick one, many, or all.  The
application
then accesses the map structure in application scope, extracts the
appropriate data based on account, date range, etc., and
returns a set
of
<display /> tables to the user.  These table are required to be
independantly sortable, pagable, etc.  Again, for up to 100 or so
accounts,
this is extremely fast.  It just goes down the toilet once I start
getting
over about 500 accounts.....
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:39 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sound's like you'll need some scrolling mechanism in
between this can
range from changing a query string for a jdbc type app or using an
object model like hibernate which supports result
scrolling.. I'd say
with reasonable confidence that returning 48,000 records in
one go is a
pretty unreasonable thing to expect. And who's gonna read
them all in
one go? :o)

Lets start at the beginning. How are you retrieving the
records at the
moment?
On 4 Mar 2004, at 17:20, Jerry Jalenak wrote:

In my application I often return large amounts of data -
for instance,
a
single user may have access to as many as 8500 accounts;
each account
may
have several hundred detail records associated with it.
I've solved my
initial problem of being able to return thousands of records (for
instance,
I can return almost 48,000 detail records in a little under 5
seconds), but
the problem I have now is that it takes FOREVER for the jsp
to render.
 I
mean, I've waited over an hour for this thing to complete.  Does
anyone have
any tips on improving / optimizing the performance of a
compiled JSP?
I'm
using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02....
Thanks!

Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS  66219
(913) 577-1496
[EMAIL PROTECTED]


-----Original Message-----
From: Hookom, Jacob [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 03, 2004 1:39 PM
To: Struts Users Mailing List
Subject: RE: Best way to handle big search results..
We have to work with search results that are sometimes 1000+
items in size.
In addition, much of the information we have is not keyed so
we cannot say
give results between id 2000 and id 20020.
Some things I found to help with search results where we did
do in memory
sorting/paging were:
1. Create view-specific lightweight objects to be pulled
from the DB
2. Memory allowed, it's faster to pull them all at once
than lazy load
4. The real cause of some of our performance problems were
the JSP's/HTML in
rendering a huge list to the client vs. only showing 20 at once.
To handle this, I created a SearchResultListAdaptor that
could sit in the
session and handle paging (if anyone wants to argue session
scope with me,
bring it on--).  The SearchResultListAdaptor then contained
the behavior of
sort order/paging, etc.  Sorting was done via bean
property using a
BeanUtilsComparator.  So my SearchResultListAdaptor could
then work with an
list of beans.
Best Regards,
Jacob
-----Original Message-----
From: Arne Brutschy [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 03, 2004 12:57 PM
To: Struts Users Mailing List
Subject: Best way to handle big search results..
Hi,

I'm looking for the best way to handle big search results.

Setting: the user can search for other users in the ldap
directory. This
might return a long list of results (~40.000 entrys).
Sadly, OpenLDAP
doesn't support server-side sorting, so I have to sort
the results
myself. I want to display 20 items per page, and the user
can browse
through these results.

What is the best way to handle these result object?
Sorting once and
storing/caching it in the session or searching and sorting
every time
the user requests a new page of results?

Any thoughts/experiences?

Regards,
Arne Brutschy



---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]



---------------------------------------------------------------------
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]


This transmission (and any information attached to it) may be
confidential and
is intended solely for the use of the individual or entity
to which it
is
addressed. If you are not the intended recipient or the person
responsible for
delivering the transmission to the intended recipient, be
advised that
you
have received this transmission in error and that any use,
dissemination,
forwarding, printing, or copying of this information is strictly
prohibited.
If you have received this transmission in error, please
immediately
notify
LabOne at the following email address:
[EMAIL PROTECTED]



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]




---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]


This transmission (and any information attached to it) may be
confidential and
is intended solely for the use of the individual or entity
to which it
is
addressed. If you are not the intended recipient or the person
responsible for
delivering the transmission to the intended recipient, be
advised that
you
have received this transmission in error and that any use,
dissemination,
forwarding, printing, or copying of this information is strictly
prohibited.
If you have received this transmission in error, please immediately
notify
LabOne at the following email address:
[EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

This transmission (and any information attached to it) may be 
confidential and
is intended solely for the use of the individual or entity to which it 
is
addressed. If you are not the intended recipient or the person 
responsible for
delivering the transmission to the intended recipient, be advised that 
you
have received this transmission in error and that any use, 
dissemination,
forwarding, printing, or copying of this information is strictly 
prohibited.
If you have received this transmission in error, please immediately 
notify
LabOne at the following email address: 
[EMAIL PROTECTED]

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]