Tommy Pham wrote:
Many of my customers have coming up on 20 years of data available. There has been a debate on transferring historic data to a separate database, but having it available is not causing a problem, except for some counts and larger search actions, and being able to see how long a client has been visiting is often useful. Statistical analysis is always done on a separate machine, with a replicated copy of the data, so as not to affect the active users ... What kind of counts/filters? What kind of RAID subsystem is the storage? What's the total size of the DB? Up to 20 years of data should be in the peta range. In that peta range, if you're not having performance issue and not using either RAID 0, 0+1, 10, 50, or 60, I'd love to hear about the application and database design in details. :)
We are still only in hundreds on Mb and historic data is has less detail than the current 'transactions'. The current postcode table is 500Mb, and while the LLPG data would increase that by the order of 100, it's currently only restricted to a councils immediate working area, so we keep the problem contained. Dropping back to postcode for out of area enquiries. Users complain if an enquiry takes more than a few seconds, and Firebird is giving me more than adequate performance, and allows shadow data to be created via triggers to reduce the need for 'counting'.
I have a new 'application' which is using the same search criteria but the data volume is growing a lot faster, 10Gb on the test system here, but I am still seeing the same search speeds once the correct indexes have been generated. But it will take a few more years before that starts reaching the 100Gb level :)
-- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk// Firebird - http://www.firebirdsql.org/index.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php