Thanks for the input. All excellent stuff...
One point of clarification:
Give yourself lots of memory and lots of file descriptors. Use the
most
modern JVM you have access to. And, if you're going to put thousands
of
multi-gigabyte individual databases onto a single filesystem, you'll
need a multi-terabyte file system, so make sure you get a solid server
operating system that can handle that.
The original XML reports are into the multi-gigabyte size on occasion,
but XML is inherently inefficient and these files are de-normalized.
Once we get them into derby, 1GB of XML boils down to a 20 to 30MB
derby database. So we won't have a major space problem with reasonable
hardware.
Thanks so much for the feedback.
Geoff
