If you choose to store binary data such as JPEGs in files you have to
have a limit on their number or some form of directory tree so that you
keep each directory from overflowing, so that alone is a good reason for
using Sqlite. The ease of backing up one file instead of tens of
thousands of individual files is another win.
Some time ago I developed an archive system in some ways like an Sqlite
without the SQL and lighter. It just holds BLOBs in a B-Tree structure
in a single file and has the advantage of overcoming directory size
problems plus it holds small files packed into its pages so that it
beats the filing systems which allocate a cluster as a minimum size of
each file. Its purpose was to permit legacy ISAM applications to be
upgraded to use large objects. Today Sqlite would do that job.
As for your final comment, I recollect hearing Dave (?) Britten from
Britten-Lee saying the same things many years ago as he sold the concept
of their database hardware. Perhaps the logic of his ideas is finally
catching on.
JS
Teg wrote:
Hello John,
I'm not just storing JPG files, I'm storing them plus MD5 hashes,
topics and other metadata. I agree, it's not as efficient as disk
storage but, being able to easily search and manage the images as well
as just keeping them in one place makes it worth the performance hit.
The hash gives me automatic duplicate detection/rejection on
insertions too.
I was archiving Manga for instance and I wanted to read it chapter by
chapter by using a drop list for manga title and another one for
chapter lists. Using pure disk storage, I found it difficult to keep
track of things. I have far fewer files to back up now too. Lately, I
have been experimenting with changing the default page sizes since for
my usage, the default 1K page size seems inefficient.
It's not a problem in Linux but, in windows there's a 8-12K file per
folder limit when using FAT32 file systems. That got in the way some
times too.
You know, there's been a lot of talk lately about "Database as file
system/file system with embedded meta-data". I'm essentially doing
that with SQlite.
C
Sunday, December 11, 2005, 2:18:20 PM, you wrote:
JS> Reads like good common sense to me :-). I mentioned Base64 and Hex as a
JS> possibility if TEXT type were to be used. As you so clearly point out
JS> they would be pointless if a BLOB is used.
JS> I also appreciate your comments about "horses for courses". If you are
JS> just storing a JPEG image why do it inefficiently in a database unless
JS> you have need for the facilities supplied by the database? Simple is
JS> always better. The idea of storing simple non-variant computational
JS> data structures in XML is bizarre.
JS> Teg wrote:
Hello John,
You guys are both going off in you own directions and arguing
something that's really an unimportant implementation detail.
Serialization's a wonderful thing, if you need it. It's nothing more
than a protocol for storing, saving, transmitting something in a
format that can be re-constituted on the the other side by a reader.
It's useful, if you need it and solves problems some implementations
have.
As for using base64 or hex encodings, why bother? SQLite is
perfectly happy with binary insertions using the newer parameterized
SQL compiling. I insert naked JPG files all day long with no encoding.
Inserting binary though doesn't solve the basic problem with
alignment and struct packing (which is only an issue in some cases),
serialization does solve this problem but, it may be a non-issue
depending on your implementation/CPU.
When your only tool is a hammer, all of your problems look like nails.
To me, a more fundamental question is why someone would want to save
structs to a database instead of to a flat disk file. I use a database
when I need to find something, not just to store things. For storage,
I just use the much faster naked disk I/O.
C
Sunday, December 11, 2005, 1:14:03 PM, you wrote:
JS> I was describing a free union used to transform big endian to little
JS> endian and vice versa. There is no encoding involved.
JS> My alternatives, BASE64 and ASCII HEX are encodings of the binary integers.
JS> You let the buzzwords drown out the fundamentals.
JS> pps wrote:
John Stanton wrote:
http://www.google.ca/search?q=boost+serialization
