Hi Kaj,
----- Original Message -----
From: "Kaj Mikkelsen" <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Monday, June 13, 2005 4:02 PM
Subject: [list] RE: [delphi-en] Large database and efficiency?
> Last time I tried the Interbase and the MySQL as well via ODBC.
> Tried to insert with no keys, and the wanted define keys later on.
Keys and indexes are critical in RDBMS performance... you cannot really do
without them.
> What I saw was that the first 50.000 records went OK, but then everything
> slowed down, and became slower and slower until it was useless.
> The apllication ran for 3 days, and wasn't finished.
>
> The data is a directory structure and the amounty of diskspace it fills.
> Lines like:
> 987000 /usr/kmi/test/this/that/and/so/on
Are these like directory stats, a bit like that produced by "du" under
Unix/Linux?
> I want to store the whole kline and the number, and then perhaps create a
> transaction table holding the individual directories.
>
> What I want to do is:
> sort entries according to size, showing top level, and then breakdown by
> clicking.
> select entries like all entries containing /kmi/
Are you ever going to want to search for entries spanning more than one
directory ('foo/bar') or partial directories ('oo')?
> Could be done by running through the textfile, but that is possible going
> to
> be to slow.
You say the file contains 1,500,000 lines, correct? And you have to read
this once and process it?
Is this file going to grow significantly over time or not?
What sort of machine is going to process the file? Supposing an average
line length of 80 characters, you'll roughly be looking at 120Mb or so of
data to read - This can probably be accomodated in RAM on a sufficiently
specced machine, and would be much quicker/more efficient than populating a
database (If you represent the data in memory as a tree (supposing it's
actually a directory tree), then you might actually somewhat less than this
amount of RAM.) How permanent does the processed data need to be?
Cheers
Walter
-----------------------------------------------------
Home page: http://groups.yahoo.com/group/delphi-en/
To unsubscribe: [EMAIL PROTECTED]
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/delphi-en/
<*> To unsubscribe from this group, send an email to:
[EMAIL PROTECTED]
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
