Well, as mentioned, the first thing to do is setup the database
structure so that it's ready for the three-state message UIDS, to recap:
state=0 - no UID yet (haven't seen provider)
state=1 - UID, but possibly invalid
state=2 - allowed to send to client.
This should be done, and tested first. Distributed systems (in practice)
tend to appear completely broken until they're completely working, so
getting the surprises that can show up here completely out of the way is
(IMO) a good first step.
After that, we'll need a provider-client API similar to the following:
int uidp_init(int fd, char *addr, int addrlen);
sets up the backend. addr/addrlen is a binary representation of
the address bound to at fd and will be added to the token.
int uidp_discover(int fd);
fd is a socket bound to the address of a peer. discovery might
add additional peers automatically.
int uidp_select();
get the file descriptor for a select() loop. this fd can be
selected() on for read-activity. if is DOES show read-activity,
uidp_wait() should be called as soon as possible - EVEN if
messages are not needed to be sent.
int uidp_wait(unsigned int r, unsigned int *uu);
waits until we have the token including regeneration step. "r"
is the number if msec we want to wait before attempting
regeneration.
any new peers that have seen the token will automatically be
added to the list. additionally, if our host address (set with
uidp_init) is not on the token, we'll add it.
this routine will not return until the token is available so
it MUST NOT BE CALLED inside a database transaction.
*uu will contain the new UID. if uu is NULL, the token is
passed immediately.
the return value will be the number of times it has seen the
token it has just received.
void uidp_pass(unsigned int n);
reconstructs the token and passes it on.
I think this is sufficient, but I [as always] welcome discussion on
this. Note that uidp_wait() is designed such that if it returns 0, then
it had to regenerate the token, if it returns 1 then the token it just
generated was valid. DBMAIL will keep track of this value and make sure
it always increases.
The on-wire protocol isn't too complicated; but the packets (using UDP)
are limited to 500 bytes. If we reserve 128 bytes for the token string
itself, that leaves us 372 bytes for the UID and discovery list. UID is
4 bytes, so that's 368 bytes for discovery or room for 92 hosts on the
cluster.
Anyone want to run a dbmail cluster with more than 92 hosts? If so, than
uidp_wait() needs to evict the peer that's furthest away.
Now, the token ring is formed from the address. Each host looks at its
discovery list and picks the next highest "network address" to be it's
immediate peer. We consider the network address to be sequence-space so
the lowest network address is "just above" the highest address.
I think that covers all the important stuff... Am I missing anything?
On Tue, 2005-05-31 at 19:38 +0000, Aaron Stone wrote:
> Yes, that fills it out. I'll start massaging the thread into an action
> plan.
>
> On Tue, May 31, 2005, Geo Carncross <[EMAIL PROTECTED]>
> said:
>
> > On Tue, 2005-05-31 at 17:15 +0000, Aaron Stone wrote:
> >> I've added a page to the wiki with your last three emails. I think we're
> >> at the point where we should hammer out a plan for the implementation.
> >>
> >> http://dbmail.org/dokuwiki/doku.php?id=unique_token
> >>
> >> There's just one piece left to properly refute, and I think that we need
> >> an actual analysis (Geo, the one you provided regarding UIDVALIDITY,
> >> inodes and file offsets would be ideal)
> >
> > I'm not completely sure what you want here.
> >
> > ``
> > Originally, UW/IMAP could make an optimization of making the UID the
> > binary offset within the mbox file and UIDVALIDITY the inode number.
> > Even if it were never implemented as such because, it can be represented
> > as such, it's going to be the same kind of problem.
> >
> > That means any solution to one problem will work for the other.
> >
> > So, think of it this way: Can you build a file system that distributes
> > across multiple machines, and IF A MACHINE IS DOWN still guarantee that
> > they increase the length of the file at the same time?
> > ''
> >
> > Is this what you were looking for?
> >
> > Are you looking for a rigorous proof that the problems are identical? Or
> > that they are generally unsolvable (without some kind of arbiter)?
> >
> >
> >> http://dbmail.org/dokuwiki/doku.php?id=unique_id_guid
> >>
> >> Because on that page it is currently stated:
> >>
> >> Contrary to earlier thoughts this would be consistent with the RFC. This
> >> is due to the fact that the RFC does not require ´increment-only¡ idÿs as
> >> defined in the current DBMail implemenation. The unique_idÿs detailed in
> >> the below ´RFC Overview¡ can be guidÿs, they must be unique and ascending,
> >> but do not have to be sequential.
> >
> >
> > I don't think I said sequential, but I did say "strictly ascending".
> > Note even in my similar-problem example, I point out that UID numbers
> > could be file offsets in an mbox file- this doesn't suggest that they be
> > sequential, does it?
> >
> >
> > In any event, since the UID numbers will be generated from a token ring,
> > they might as well be sequential.
> >
> > That said, dbmail has it's OWN NEED to maintain a certain amount of
> > referential integrity and I think it's perfectly sane and wise to use
> > GUIDs for that.
> >
> > However. Users need to see UIDs, which means at some level, a map is
> > required. One might as well use UID numbers as that reference, but if
> > tokens are going to be lost frequently, this would mean more overhead.
> >
> > --
> > Internet Connection High Quality Web Hosting
> > http://www.internetconnection.net/
> >
> > _______________________________________________
> > Dbmail-dev mailing list
> > Dbmail-dev@dbmail.org
> > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
> >
--
Internet Connection High Quality Web Hosting
http://www.internetconnection.net/
