Thanks Scott, I did see your reply to Anmol, but I'm afraid this isn't the answer I'm looking for.
Having even 500 records in memory at one time is still a very bad thing to do, and retrieving smaller chunks using LargeSelect is low level complexity I really don't want to have to think about. It also impacts performance because it would require a new query with every chunk instead of a single query for the whole list. I'm not showing the million records to the user, I'm doing batch processing on the server. I am amazed that no-one else has had this problem before. I'm even more amazed that the developers designed it with such a fundamental flaw in the first place. I'll give them more credit than to suggest that they don't understand the issue, but the only alternative is that it was never intended to be used in an industrial strength application. I am willing to implement some sort of workaround if there a plans for a fix in the very near future, but if not I will be forced to use something else, which of course is something I would really like to avoid. Considering that a fix would be very straight-forward to implement, I don't think it's unreasonable to expect it before the next beta. > ----- Original Message ----- > From: "Scott Eade" <[EMAIL PROTECTED]> > To: "Turbine Torque Users List" <[EMAIL PROTECTED]> > Sent: Monday, July 29, 2002 9:06 AM > Subject: Re: Iterating through a large list > > > > Martin, > > > > See my reply to Anmol with the subject "Out of Memory Problem" - again, > this > > is what LargeSelect is designed to do. > > > > A few pertinent notes: > > > > 1. When dealing with such large quantities of data you should ensure that > > query you provide does not include a non-index sort or any other criteria > > that will require all rows to be retrieved before a subset of rows can be > > returned by the database. If you do not follow this advice, each query > that > > is executed to retrieve a subset of the results will actually retrieve all > > of the records resulting in a far from optimal solution. > > > > 2. Using a database that provides offset and limit mechanisms that are > > supported by torque will produce the best results - i.e. MySql, PostgreSQL > > currently and Oracle when someone gets around to completing the support. > > When this support is not provided, Village will execute the entire query > and > > then discard the rows outside the memory limit subset - acceptable in many > > situations, but probably not when you have 1M+ rows. > > > > 3. If you have millions of rows you should provide your user with the > > ability to filter the records - nobody will be interested in viewing > > 1,000,000 records, just the few hundred that meet some criteria they are > > interested in. Look at how Scarab handles its user searches for an > > excellent example (but combine this solution with LargeSelect). > > > > HTH, > > > > Scott > > -- > > Scott Eade > > Backstage Technologies Pty. Ltd. > > Web: http://www.backstagetech.com.au > > > > > > > From: "Martin Jericho" <[EMAIL PROTECTED]> > > > Reply-To: "Turbine Torque Users List" > <[EMAIL PROTECTED]> > > > Date: Sun, 28 Jul 2002 16:55:35 +1000 > > > To: <[EMAIL PROTECTED]> > > > Subject: Iterating through a large list > > > > > > I am using torque in an application which deals with thousands of > records, > > > maybe eventually millions. As far as I can tell the only way to iterate > > > through them is using the standard ArrayList from the doSelect method. > This > > > of course doesn't get me very far before I run out of memory. > > > > > > Why doesn't torque implement it's own List subclass instead of using an > > > ArrayList when selecting multiple records? This way it could internally > avoid > > > having to keep all of the rows in memory at the same time. The > ArrayList > > > implementation looks very Micky-Mouse. > > > > > > Another alternative is to have methods that return an iterator rather > than a > > > list, in the vast majority of cases where you only need sequential > access. > > > > > > How are other people dealing with these limitations? > > > -- To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
