Re: Best way to handle big search results..
Google for the Value List Handler pattern. It should also be on the java.sun.com J2EE blueprints page. I use a pager object which encapsulates a list of the sorted results. If the result set size is over a certain threshhold, the pager instead wraps a value list handler which only fetches subsets of the results. What you can do depends on what functionality is available in OpenLDAP. I definitely wouldn't reload with every request. If each result item is large, perhaps you could sort and then store only a list of IDs in the session, and with each request make a query to OpenLDAP to return user information for the set of IDs on the next page. Hopefully you can batch this, otherwise you'll be making 20 LDAP calls per user request and that would be a bad thing. Rick DeBay On Wed, 03 Mar 2004 19:57 , Arne Brutschy [EMAIL PROTECTED] sent: Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Best way to handle big search results..
In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Best way to handle big search results..
One thing that can help for remote users (dialup users), is to put a compression filter on your app, it will take care of extra whitespaces, etc while allowing you to keep your jsp formatting clean. Along the lines of JSP speed, I've found Tomcat 5 to be a huge speed improvement over Tomcat 4. Things to look at is refactoring your forEach and choose loops to see what works best for you. The other thing is to use CSS as much as possible with div's etc to remove tables within tables which do take quite a while to render. We had a consultant come in who strictly does HTML to take our pages and convert them to CSS standards, I couldn't believe what all the page rendered in so little code. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:20 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail
Re: Best way to handle big search results..
Jerry: Jacob's note (see below) already says this would be a problem, but he has also mentioned a way to solve your problem; check if that works for you. Store the records in a session variable (that's right, session, instead of request makes sense here!!) and display just 20 or so records at a time. You will have to see if this is feasible.. it will depend on how strong-big your server is, etc..:) Regards, Geeta Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Best way to handle big search results..
Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Best way to handle big search results..
Jacob / Geeta - Thanks for the responses. The business requires that each account / detail record set be placed in an independantly pagable/sortable table, so I'm using the display / taglib to handle this. For relatively small numbers (say up to a 100 accounts or so) the performance is really pretty good. I'd like to be able to restrict the date range that a user can specify, but again, business requirement states that 'the user must be able to browse upto 90 days worth of data'. I've tried to explain that, for some of these large clients this simply isn't going to work. I was hoping (8-) ) that there was a setting in Tomcat that I may have missed (I've already set up Tomcat as a 'production' environment). I don't know if it makes a difference, but I have yet to move to the EL tags - I'm still using the old Struts taglibs for things like logic /, etc. Has anyone benchmarked the performance between these two approaches? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Geeta Ramani [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:33 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Jerry: Jacob's note (see below) already says this would be a problem, but he has also mentioned a way to solve your problem; check if that works for you. Store the records in a session variable (that's right, session, instead of request makes sense here!!) and display just 20 or so records at a time. You will have to see if this is feasible.. it will depend on how strong-big your server is, etc..:) Regards, Geeta Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED
RE: Best way to handle big search results..
Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per
Re: Best way to handle big search results..
Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting
RE: Best way to handle big search results..
Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting
RE: Best way to handle big search results..
Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: In my application I often return large amounts of data - for instance, a single user may have access to as many as 8500 accounts; each account may have several hundred detail records associated with it. I've solved my initial problem of being able to return thousands of records (for instance, I can return almost 48,000 detail records in a little under 5 seconds), but the problem I have now is that it takes FOREVER for the jsp to render. I mean, I've waited over an hour for this thing to complete. Does anyone have any tips on improving / optimizing the performance of a compiled JSP? I'm using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02 Thanks! Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 1:39 PM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id
RE: Best way to handle big search results..
But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them all in one go? :o) Lets start at the beginning. How are you retrieving the records at the moment? On 4 Mar 2004, at 17:20, Jerry
Re: Best way to handle big search results..
Okay ..
You said the results are in the application scope already. so you want
to scoll through those.
so in you servlet theres something like
List peopleList = goAndGetShitLoadsOfRecordsFromDB();
context.setAttribute(people, peopleList.toArray());
..
c:forEach var=person items=${people} begin=${param.index}
end=${param.index + 20
c:out value=${person.name} /
/c:forEach
html-el:link page=/scroll.do?index=${param.index +
20}Next/html-el:link
On 4 Mar 2004, at 17:58, Jerry Jalenak wrote:
Sorry for the misunderstanding - are you asking how the request is
made from
the user to the app? or from the app to the database?
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS 66219
(913) 577-1496
[EMAIL PROTECTED]
-Original Message-
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:55 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sure but how do does the query get made?
On 4 Mar 2004, at 17:49, Jerry Jalenak wrote:
Mark,
Let me give you some background first. On any given day I will be
reporting
on about 80,000 accounts with over 300,000 detail records available
for the
past 90 days. My clients are using this data to determine
employment
elgibility, etc. The majority of these clients are individual
employers
that might have between 1 and 100 accounts. These are no
problem. The
'problem' clients are organizations that handle this type
of work for
multiple employers on a contract basis. These clients can
and do have
several thousand accounts that could *potentially* have data that
needs to
be reported.
The approach I took was to write a separate servlet that
wakes up every
hours, re-sweeps the account code table, re-sweeps the data table,
reconciles these into a set of nested maps, and places it into
application
scope. Part of this process throws out any account that
doesn't have
data
to be reported, otherwise I'd be trying to handle 200,000
accounts.
Based
on the account and type of data the user can 'see', they
are presented
a
list of accounts where they can pick one, many, or all. The
application
then accesses the map structure in application scope, extracts the
appropriate data based on account, date range, etc., and
returns a set
of
display / tables to the user. These table are required to be
independantly sortable, pagable, etc. Again, for up to 100 or so
accounts,
this is extremely fast. It just goes down the toilet once I start
getting
over about 500 accounts.
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS 66219
(913) 577-1496
[EMAIL PROTECTED]
-Original Message-
From: Mark Lowe [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 04, 2004 10:39 AM
To: Struts Users Mailing List
Subject: Re: Best way to handle big search results..
Sound's like you'll need some scrolling mechanism in
between this can
range from changing a query string for a jdbc type app or using an
object model like hibernate which supports result
scrolling.. I'd say
with reasonable confidence that returning 48,000 records in
one go is a
pretty unreasonable thing to expect. And who's gonna read
them all in
one go? :o)
Lets start at the beginning. How are you retrieving the
records at the
moment?
On 4 Mar 2004, at 17:20, Jerry Jalenak wrote:
In my application I often return large amounts of data -
for instance,
a
single user may have access to as many as 8500 accounts;
each account
may
have several hundred detail records associated with it.
I've solved my
initial problem of being able to return thousands of records (for
instance,
I can return almost 48,000 detail records in a little under 5
seconds), but
the problem I have now is that it takes FOREVER for the jsp
to render.
I
mean, I've waited over an hour for this thing to complete. Does
anyone have
any tips on improving / optimizing the performance of a
compiled JSP?
I'm
using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02
Thanks!
Jerry Jalenak
Development Manager, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS 66219
(913) 577-1496
[EMAIL PROTECTED]
-Original Message-
From: Hookom, Jacob [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 03, 2004 1:39 PM
To: Struts Users Mailing List
Subject: RE: Best way to handle big search results..
We have to work with search results that are sometimes 1000+
items in size.
In addition, much of the information we have is not keyed so
we cannot say
give results between id 2000 and id 20020.
Some things I found to help with search results where we did
do in memory
sorting/paging were:
1. Create view-specific lightweight objects to be pulled
from the DB
2. Memory allowed, it's faster to pull them all at once
than lazy load
4. The real cause of some of our performance problems were
the JSP's/HTML in
rendering a huge list to the client vs. only showing 20 at once.
To handle this, I created a SearchResultListAdaptor that
could sit
RE: Best way to handle big search results..
Yeah, stay away from tables within tables, they can be a big slow up. -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:13 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable
Re: Best way to handle big search results..
Even if everything in your system handles 320,000 lines of html, your users never will. This isn't a performance issue, its a design one. You gotta go back to the drawing board and stop trying to get something to perform that your users will never accept. Niall - Original Message - From: Jerry Jalenak [EMAIL PROTECTED] To: 'Struts Users Mailing List' [EMAIL PROTECTED] Sent: Thursday, March 04, 2004 5:12 PM Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re
Re: Best way to handle big search results..
I think html rendering is the least of his worries :o) I'm surprised you can get that many records into the application context in the first place. On 4 Mar 2004, at 18:14, Hookom, Jacob wrote: Yeah, stay away from tables within tables, they can be a big slow up. -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:13 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sound's like you'll need some scrolling mechanism in between this can range from changing a query string for a jdbc type app or using an object model like hibernate which supports result scrolling.. I'd say with reasonable confidence that returning 48,000 records in one go is a pretty unreasonable thing to expect. And who's gonna read them
RE: Best way to handle big search results..
Unfortunately, the business gets to dictate this one, and they maintain that these large clients need to ability to view up to 90 days worth of data. I agree with you that this is logistically impossible, but since the business requirements state that it must be so, my QC group is testing it and saying that it fails. If it fails, it's got to be fixed. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Niall Pemberton [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:17 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Even if everything in your system handles 320,000 lines of html, your users never will. This isn't a performance issue, its a design one. You gotta go back to the drawing board and stop trying to get something to perform that your users will never accept. Niall - Original Message - From: Jerry Jalenak [EMAIL PROTECTED] To: 'Struts Users Mailing List' [EMAIL PROTECTED] Sent: Thursday, March 04, 2004 5:12 PM Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see
RE: Best way to handle big search results..
Hasn't been a problem so far 8-) When i first designed this system, I went with an approach to do my lookup directly from the db. Turned out that, with these large account clients, the SQL call was coming back fairly quickly, but it was taking significant time to run through the records and apply various filters that are available to the user (date range, previously viewed, sub-set, etc.) So I went back to the drawing board, wrote this second servlet to maintain some in-memory data structures (basically maps-in-maps), and dramatically sped up the whole app. The only remaining issue is this one - where I am returning a couple of thousand accounts and several thousand detail records I may have to end up re-negogitaing this one business requirement. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:21 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. I think html rendering is the least of his worries :o) I'm surprised you can get that many records into the application context in the first place. On 4 Mar 2004, at 18:14, Hookom, Jacob wrote: Yeah, stay away from tables within tables, they can be a big slow up. -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:13 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places
Re: Best way to handle big search results..
!= null) {
if(listIterator.hasNext()) {
object = listIterator.next();
list.add(object);
}
} else {
throw new ChainedException(ListHandler: getNextElements(int) failed
-- listIterator null);
}
return list;
}
public void resetIndex()
throws ChainedException{
if(listIterator != null){
listIterator = list.listIterator();
} else {
throw new ChainedException(ListHandler: resetIndex() failed --
listIterator null.);
}
}
} ///;-)
At 07:46 AM 3/4/2004, you wrote:
Google for the Value List Handler pattern. It should also be on the
java.sun.com
J2EE blueprints page.
I use a pager object which encapsulates a list of the sorted results. If the
result set size is over a certain threshhold, the pager instead wraps a value
list handler which only fetches subsets of the results. What you can do
depends
on what functionality is available in OpenLDAP. I definitely wouldn't reload
with every request. If each result item is large, perhaps you could sort and
then store only a list of IDs in the session, and with each request make a
query
to OpenLDAP to return user information for the set of IDs on the next page.
Hopefully you can batch this, otherwise you'll be making 20 LDAP calls per
user
request and that would be a bad thing.
Rick DeBay
On Wed, 03 Mar 2004 19:57 , Arne Brutschy
[EMAIL PROTECTED] sent:
Hi,
I'm looking for the best way to handle big search results.
Setting: the user can search for other users in the ldap directory. This
might return a long list of results (~40.000 entrys). Sadly, OpenLDAP
doesn't support server-side sorting, so I have to sort the results
myself. I want to display 20 items per page, and the user can browse
through these results.
What is the best way to handle these result object? Sorting once and
storing/caching it in the session or searching and sorting every time
the user requests a new page of results?
Any thoughts/experiences?
Regards,
Arne Brutschy
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Best way to handle big search results..
++1 At 09:17 AM 3/4/2004, you wrote: Even if everything in your system handles 320,000 lines of html, your users never will. This isn't a performance issue, its a design one. You gotta go back to the drawing board and stop trying to get something to perform that your users will never accept. Niall - Original Message - From: Jerry Jalenak [EMAIL PROTECTED] To: 'Struts Users Mailing List' [EMAIL PROTECTED] Sent: Thursday, March 04, 2004 5:12 PM Subject: RE: Best way to handle big search results.. But. I'm already doing paging. Each independant account table only returns the first 10 detail records; everything is available by paging through the next 'n' pages. For example, one of my clients has access to almost 8900 accounts. Of these accounts, there are about 3500 with reported data in the past 90 days. In the past month alone, these 3500 accounts have almost 48000 detail records. So I'm returning (probably not all 3500 accounts, but a good majority of 'em) say 2000 tables, with potentially 10 detail records per table. Each table has seven columns, so just for the tables I'm returning about 320,000 lines of html. I'm really starting to think this is a client side rendering issue - the browser simply can't process the number of lines of html that I'm pushing it. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Hookom, Jacob [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 11:03 AM To: Struts Users Mailing List Subject: RE: Best way to handle big search results.. Jerry, We ran into the same problems you are having, and then there reaches a point where you leave it according to what the requirements say, and when people complain, you tell them it's because it not only takes a while at the server to create the HTML for so many records, but it also takes just as long to render them in your browser. Then they say, Really? and you say, Yes, this is why we should do paging, to which they reply, then we should probably add it in there. -Jake -Original Message- From: Jerry Jalenak [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:58 AM To: 'Struts Users Mailing List' Subject: RE: Best way to handle big search results.. Sorry for the misunderstanding - are you asking how the request is made from the user to the app? or from the app to the database? Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:55 AM To: Struts Users Mailing List Subject: Re: Best way to handle big search results.. Sure but how do does the query get made? On 4 Mar 2004, at 17:49, Jerry Jalenak wrote: Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported. The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of display / tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts. Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Mark Lowe [mailto:[EMAIL PROTECTED] Sent: Thursday, March 04, 2004 10:39 AM To: Struts
Best way to handle big search results..
Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Best way to handle big search results..
We have to work with search results that are sometimes 1000+ items in size. In addition, much of the information we have is not keyed so we cannot say give results between id 2000 and id 20020. Some things I found to help with search results where we did do in memory sorting/paging were: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in rendering a huge list to the client vs. only showing 20 at once. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob -Original Message- From: Arne Brutschy [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 12:57 PM To: Struts Users Mailing List Subject: Best way to handle big search results.. Hi, I'm looking for the best way to handle big search results. Setting: the user can search for other users in the ldap directory. This might return a long list of results (~40.000 entrys). Sadly, OpenLDAP doesn't support server-side sorting, so I have to sort the results myself. I want to display 20 items per page, and the user can browse through these results. What is the best way to handle these result object? Sorting once and storing/caching it in the session or searching and sorting every time the user requests a new page of results? Any thoughts/experiences? Regards, Arne Brutschy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Best way to handle big search results..
Jacob Ame: Hookom, Jacob wrote: 1. Create view-specific lightweight objects to be pulled from the DB 2. Memory allowed, it's faster to pull them all at once than lazy load 4. The real cause of some of our performance problems were the JSP's/HTML in Excatly what we found too.. and our results were only in the hundreds rather than in the thousands! Many rows of data coupled with tables were a real killer.. To handle this, I created a SearchResultListAdaptor that could sit in the session and handle paging (if anyone wants to argue session scope with me, bring it on--). In fact this is one place where a session variable comes in most handy..:) The SearchResultListAdaptor then contained the behavior of sort order/paging, etc. Sorting was done via bean property using a BeanUtilsComparator. Almost exactly what we did too..:) So my SearchResultListAdaptor could then work with an list of beans. Best Regards, Jacob Regards, Geeta - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
