On Nov 23, 2005, at 6:49 AM, Michael Segel wrote:
I'm a little skeptical whenever I see a question like this.
There are always limits.
And while you could get up to 100 GB, I wouldn't recommend it performance
wise as Derby is currently written.
Well, if I understand it correctly, IBM have the resources to actually set up and test databases of that size, and I'm happy that they have, and will do so. It should be a learning experience for them, and I expect that they will share with the community whatever they learn about Derby, warts and all.
I think the important point isn't to find a theoretical limit, but to choose
the right database for the right job.
I can't disagree!
...
Going over a certain point, you're going to want to choose a database that
has a better storage method than using the file system.
(And I had better clarify...)
I'm talking about having the data stored in either raw or cooked chunks. Not
As separate files within a directory. (More akin to Informix's Standard
Engine.)
Well, my experience does not lead me to believe that a database engine will necessarily be better at managing persistent storage via more primitive i/o, compared to letting the operating system manage the data. There are so many parameters that affect disk performance that "better storage method than... the file system" trivializes the issue.
Just off the top, how memory is used to store the user's record (row) data, how that memory is mapped to the operating system's i/o buffers, whether logs or data pages hold "the truth", how the data to be written is organized for multiple users, what operating system calls are used to write and/or verify the data, whether the writes are ordered by physical location on disk, whether the disk controller is buffered, the size of the buffers, and what form of verification that the write has succeeded the engine requests.
Few of these issues have anything to do with whether the disk space is "raw", "cooked", or in a file system.
...
Its not like I haven't been playing with databases since the 1980s?
(eeew! Now I know I'm getting old.) ;-)
The raw/cooked/files discussion reminds me a lot of the assembler/C/C++ arguments over the years. A tiny percentage of programmers these days have time to understand the architecture of a modern CPU chip and how to optimize a program based on it. We leave that to the language runtime implementers.
And few of us are interested enough and need to learn the ins and outs of writing raw data to partitions. We leave these details to the operating system gurus. Give me a nice file abstraction and high performance, direct i/o buffers and let the experts play in their own domains.
