Martin, See my reply to Anmol with the subject "Out of Memory Problem" - again, this is what LargeSelect is designed to do.
A few pertinent notes: 1. When dealing with such large quantities of data you should ensure that query you provide does not include a non-index sort or any other criteria that will require all rows to be retrieved before a subset of rows can be returned by the database. If you do not follow this advice, each query that is executed to retrieve a subset of the results will actually retrieve all of the records resulting in a far from optimal solution. 2. Using a database that provides offset and limit mechanisms that are supported by torque will produce the best results - i.e. MySql, PostgreSQL currently and Oracle when someone gets around to completing the support. When this support is not provided, Village will execute the entire query and then discard the rows outside the memory limit subset - acceptable in many situations, but probably not when you have 1M+ rows. 3. If you have millions of rows you should provide your user with the ability to filter the records - nobody will be interested in viewing 1,000,000 records, just the few hundred that meet some criteria they are interested in. Look at how Scarab handles its user searches for an excellent example (but combine this solution with LargeSelect). HTH, Scott -- Scott Eade Backstage Technologies Pty. Ltd. Web: http://www.backstagetech.com.au > From: "Martin Jericho" <[EMAIL PROTECTED]> > Reply-To: "Turbine Torque Users List" <[EMAIL PROTECTED]> > Date: Sun, 28 Jul 2002 16:55:35 +1000 > To: <[EMAIL PROTECTED]> > Subject: Iterating through a large list > > I am using torque in an application which deals with thousands of records, > maybe eventually millions. As far as I can tell the only way to iterate > through them is using the standard ArrayList from the doSelect method. This > of course doesn't get me very far before I run out of memory. > > Why doesn't torque implement it's own List subclass instead of using an > ArrayList when selecting multiple records? This way it could internally avoid > having to keep all of the rows in memory at the same time. The ArrayList > implementation looks very Micky-Mouse. > > Another alternative is to have methods that return an iterator rather than a > list, in the vast majority of cases where you only need sequential access. > > How are other people dealing with these limitations? > > -- To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
