You cannot even consider loading even a thousand records directly.
Get a set of ID numbers. Load each record as it's needed. This is very basic stuff, and not even that hard to implement. I am just saying for the record that this is not hard to do, hopefully no one else will be scared away from the concept. As a general rule of coding, do it right the first time. One thread reading a list of integers while another is on the GUI is not complex. Isaac On 10/28/06, Da Martian <[EMAIL PROTECTED]> wrote:
Hi Thanks for the reposnse. The main reason is my record count could be from a few thousands to a million. But even at the lowly numbers of around 10000 the interface can seem slugish if you read every record before displaying anything. As you mention, and has been disucssed above, doing stuff in the background is good way to go, but more complex. As a generla rule of coding I put as few unneccessary threads into a "phase 1" program as I can, because the complexity goes up hugly, threads can be complex to use, co-ordinate, interrupt etc... and chance of bugs goes up drmatically. So I tend to do that for a "Phase 2" - Bells and whistles phase and only when there isnt a simpler way. This thread has covered just about all approaches I can think of :-) thanks for the reponses. S On 10/27/06, Isaac Raway <[EMAIL PROTECTED]> wrote: > > Why don't you design the table with a unique row ID, stored in an > integer field, then fetch a list of those ID numbers? > > For 5000 rows, assuming you store them in you application as 4 byte > longs, that's about 19 k of memory. > > Counting that result as you receive it isn't that difficult. If it > takes a long time (it probably won't) you can do it in another thread > and update the interface as appropriate. > > I'm not seeing a downside here. > > Isaac > > On 10/26/06, Da Martian <[EMAIL PROTECTED]> wrote: > > No there isnt, but RDBM systems are a generalised data retrieval > mechanism. > > As such they suffer from that generality. > > > > Dont get me wrong, RDBM systems are appropriate for 95% of all data > > requirements I have had to deal with and I would never dream of trying > to > > write one from scratch, nor can I imagine a world without them. > > > > However certain applications (Weather data, Gnome data, Large indices > (like > > google)) require using somethng designed specifically for that purpose. > If > > you customise data retrieval (and particluar your sorting/indcies/access > > path) you can leave rdbms in the dust in terms of performance. All I > have > > read about google, suggests they do exactly this. Although I must point > out, > > I dont actually know anything about google with any certainty. Just what > has > > "leaked" out over the years on the rumour mill. But designiing my own > > "google" like indices (on a smaller scale of coure) and some > specialisted > > weather stuff, it neccessary to throw away the rdbms and do it yourself. > For > > a goole query for instance, they know they will get a list of 1 or more > > words. They also know they will only ever search through the index of > words. > > They dont have other data types, records or tables. Why go through all > the > > hassles of compiling SQLs, and that generic overhead when your > application > > will only ever do one thing? You can just make an API like this > > "search(wordlist): Resultset. " > > > > You immediatly save yourself complexity and processing time. Then for > large > > indices you will know your data set, so instead of using a std BTree you > > would use a more appropraite DS possible with skip lists etc.. > > . > > As for performing a database search twice, this whole thread has shown, > that > > sometimes the you have to :-) > > > > S > > > > On 10/25/06, John Stanton <[EMAIL PROTECTED]> wrote: > > > > > > There is no magic in data retrieval. Google use the same physical > laws > > > as us ordinary mortals. > > > > > > I see no reason to ever perform a dataabase search twice. > > > > > > > > > > -- > Isaac Raway > Entia non sunt multiplicanda praeter necessitatem. > > http://blueapples.org - blog > http://stonenotes.com - personal knowledge management > > > ----------------------------------------------------------------------------- > To unsubscribe, send email to [EMAIL PROTECTED] > > ----------------------------------------------------------------------------- > >
-- Isaac Raway Entia non sunt multiplicanda praeter necessitatem. http://blueapples.org - blog http://stonenotes.com - personal knowledge management ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
