None of the examples you gave require that the data reside in a single
file. They don't even need to reside on a single system. Distributed
databases make nearly unlimited database sizes possible by joining several
physical data sources together into logical objects, much as AFS joins
together many distinct physical file systems into a single logical file
system.
As the original query implied, the only time you really need to have your
data together in a single file is when you need to perform a physical
operation on a chunk of data, and you wish it to happen in an atomic way
>From the file system's perspective.
Always keep in mind that even your traditional physical file system breaks
file objects down into separate physical blocks of data. The illusion of
larger logical objects is provided by software, and the same principle can
easily be extended to more complex objects.
Raster images are even easier to break up across files, since they usually
follow a regular file format which can be easily reconstructed from
separated chunks.
It sounds as if you need to investigate software applications which scale
better. The file system doesn't need to handle large files. The software
needs to be able to work with large data objects using regular files.
That should be a much easier problem to solve.
-- Lee
