Matthew T. O'Connor wrote:
Agreed, the only reason I propose the header_list table is for
performance reasons. This allows the header_values table (which will be
much bigger) to be searched based on an int comparison rather than text
search. I think this is a serious performance boost, but if it's proven
that it's not, then we don't need it.
I'm still doubtfull.
header_list: ( Contains an exhaustive list of all headers from all
messages in the database. )
[...]
header_values: ( Contains the values from all the headers in all the
messages in database )
[...]
This structure will make it very easy to query all the headers from a
given message or find all the messages with a given header, or a
given header value. It also leaves our current structure intact
which will make it easier to phase in.
It will work, but you still require a lookup on an extra table to do header searches. There has to be hidden
cost there. Also the added db-interaction at insertion time is very significant.
What I like about the yukatan approach is that it tries to make searches on very common headers as cheap and
fast as possible. It does this by going one step beyond separate header storage: preparse certain headers for
common attributes: the in-reply-to and references headers will be used for threading. These headers contain
one or more message-id header values, which are stored separately. With this approach building message-threads
can be done with fully indexed, single-table queries! Of course a union with the msgids from the opened
mailbox is still required, but you can't beat such a setup wrt threading.
What do you think? I don't think we need to special case any headers
not even sendername or subject.
The more I think about this, the more convinced I am that we do need to treat common headers differently in
the end. And your suggestion to preseed the header_names table and hardcode their ids tells me we're agreed here.
The other major thing they have on us is that they manage each MIME
entity inside an email separately. This has nice advantages for
searching headers of attachments etc.. That would be a large change
from our datamodel, but might be nice to think about someday.
We can *easily* switch to storing attachments separately. The whole messageblk approach will support this
flawlessly. In fact, I can switch to storing attachments in separate blks today, and noone will notice because
it will be transparant from the point of view of message retrieval.
But I don't think searching for certain attachment headers is a very common use-case at the moment. IIRC, imap
doesn't support this. It could probably help though in building BODYSTRUCTURE response, so lets consider this
for a moment.
If I want to store mime-part headers the same as message-headers, first thing I have to do is change the
physmessage_id references in the header tables to messageblk_idnr references. Next I would have to store
mime-part headers same as message-headers. Finally I would have to change the messageblks model
from:
block[0]: message-header
block[1]: message-body[slice1]
block[N]: message-body[sliceN]
to
block[0]: raw message-headers
block[1]: mime-part[0]
block[N]: mime-part[N]
Where the only really tricky part is converting an existing dbmail storage to this modified approach. Still,
way cool stuff.
But that's me getting ahead of myself. Small steps.
Their model makes a lot of sense in many ways, but I still don't like
some of it. They special case a handful of headers for each MIME entity
not only by having a separate copy in the entity table but also by
having a separate table for many of these headers. Perhaps someday we
can add this for performance reasons, but I don't think we need to.
Yukatan also has a headers table much like the one I described above,
they don't have the header_list table broken out the way I do, but I
think that is why then need to special case alot of the headers.
So in summary, yes the Yukatan model is nice, and has a lot of
advantages over ours, but I still think we are best served by starting
with the two table design I described earlier. This should be
sufficiently fast and flexible that we can go a long way before we have
to special case anything.
I'll keep your idea in mind as I start building test-cases for the insertion phase. But I still think that
having to choose between a convoluted 'if not select header_name then insert header_name; insert
header_value;' and a simple 'insert header_name, header_value' for each header of each message inserted, the
latter seems to make more sense for now.
But I'm always open to arguments and willing to change my mind (as-if my wife
would say :-)
Thanks for helping me think this through a little.
--
________________________________________________________________
Paul Stevens mailto:[EMAIL PROTECTED]
NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED]
The Netherlands________________________________http://www.nfg.nl
