The file size limits are imposed by the operating system, so one thing you might consider is some of the linux filesystems that do not have that limitation, (IIRC, XFS doesn't have that limit), or the use of a solaris machine, which (again, IIRC :)) doesn't have that limitation, either.
Others on this list can probably help you more than I can on that front. Seth Northrop wrote: > Some background... > > We have Objects > r1, r2 ... r1600 Which each have (~1600) regions > d1, d2 ... d500 Which each have (~500) datapoints > > So, roughly we have around 786,000 datapoints on a given object. > > Typically, we might take 15 or so regions and take data from each of their > datapoints (~7500) at ~1300 "steps" or environmental variations. So, each > test will generate about 9.75M floats. A curve can be extracted from the > 1300 steps (floats) for each datapoint. > > We then run these tests hundreds of times a day - sometimes on different > objects, sometimes on the same ojbects (so you can analyse the curves > through time) > > We have to date stored this raw data within compressed text files indexed > by MySQL. However, as you can imagine, querying this data is a pain. We > have to ask MySQL for all of the tests of a given object, then analyse the > files to extract the appropriate curves - often opening up 100s of > datafiles. > > The question is whether anyone has any more intelligent ways of storing > this data within MySQL without busting MySQLs file size limits, or > reasonable CPU/RAM contraints. > > Storing the data raw in rows doesn't seem like the most sane technique. > Nor does throwing it into a char for each datapoint. What I'm looking for > is a _lossless_ technique to store the curves within the database for each > datapoint (we need to be able to extract and analyse data for each > datapoint). My guess is that someone out there is doing scientific > analysis similar to this using MySQL as their backend. > > The environment is R&D so queries will be fairly limited. We obviously > don't want queries to take down the DB server, but, at the same time this > isn't the backend data to Yahoo! serving millions of requests every > minute. > > Thanks! > Seth > > > --------------------------------------------------------------------- > Before posting, please check: > http://www.mysql.com/manual.php (the manual) > http://lists.mysql.com/ (the list archive) > > To request this thread, e-mail <[EMAIL PROTECTED]> > To unsubscribe, e-mail <[EMAIL PROTECTED]> > Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php > > > -- Gabriel Cain Unix Systems Administrator [EMAIL PROTECTED] Dialup USA, Inc. 888-460-2286 ext 208 --------------------------------------------------------------------- Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
