Paul McCullagh wrote:
On Nov 9, 2008, at 11:41 PM, Brian Aker wrote:
On Nov 10, 2008, at 7:27 AM, Jim Starkey wrote:
Philosophically, I'm not sure than inconvenience should be a major
consideration when considering architectures. If drizzle could
delay blob materialization to the last possible nanosecond in hope
it would need to, drizzle would be a great deal more application
friendly.
This is something I find more interesting because if we can delay the
last moment (or just pass in a fd/offset/length) we can do sendfile
tricks to just map from disk to card bypassing memory entirely. This
would mean though that an engine laid out a blob on disk as a single
object.
Part of he logic in the original protocol was:
SELECT a, b, c
Is that if I asked for the values in that order, I don't want to have
to make a second request for a particular member of that set. I have
already said "this is what I want, give it to me". The blob fetching
has always implied a sort of "but do you really want it?". The blob
fetching in Oracle was an original limitations of their design, it
just happened to be in MySQL that it didn't have that limitation
(though the current twice copy of blob data is ridiculous).
[ Little plug for one of our projects, but I think it is quite
relevant to this discussion. ]
The functionality of the BLOB streaming engine (www.blobstreaming.org)
already covers quite a bit of this stuff.
The engine:
- Maintains a BLOB repository where BLOBs are stored sequentially on
disk (sendfile() possible).
- Issues BLOB locators which can be stored in the rows instead of the
data.
- Allows BLOBs to be streamed in and out of the database using HTTP
PUT and GET.
- Automatically delays deletion of BLOBs so that they can be "grabbed
later".
Replication of the BLOB data is possible in a similar manor described
by Mats. However, the BLOB data need not be written to the binary log
because it can be retrieved from the repository.
Engines call the BLOB streaming engine's "server-side" streaming API
in order to support this functionality. Consistency between databases
tables and the repository is maintained using reference counting calls
in the API.
A dormant piece of Falcon is probably worth mentioning -- blob
repositories. A blob repository is a named database object with a file
name pattern for "volumes", an optional rollover period, an optional max
size, and the name of a sequence to assign blob ids. Any blob field can
have the name of a repository (a repository can contain blobs from any
number of fields). Blobs are, not surprisingly, stored in the current
volume. When a volume reaches it max size or rollover date, a new
volume is created.
Repository blobs have stable blob ids that can be referenced without
materializing the blob. If a repository blob is copied from one field
or table to another, the blob id is reused. This means that by
defaults, repository blobs are not deleted, though there is an explicit
mechanism to "expunge" repository blobs. A reference to an expunged
blob is a simple runtime error -- no biggie.
Blob repositories are designed with the replication facilities so
repository volumes are fungible, i.e. volumes from all replicants are
interchangeable. There are two implications for backup:
1. Old volumes are stable, can be shared among replicants, and only
need to be backed up once.
2. Replication volumes are interchangeable among replicants.
Given the typical nature of blob data, it makes all the sense in the
world to store blobs outside of primary database in volumes that become
read-only when closed.
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
