On Jan 25, 2009, at 11:40 PM, Kiran Ayyagari wrote:
hi Emmanuel,
Emmanuel Lecharny wrote:
Hi guys,
it's essential that we have a decent Changelog system in order to
be able to build a reliable server, plus use it for replication
eventually.
The currengt changelog system has many advantages :
- it basically works
- it's damn fast
- it can be extended easily
But it also has some drawbacks :
- it currently stores changes info in memory
- it stores forward and revert changes (we don't need revert usually)
- it stores changes in LdifEntry which uses Attributes instead of
the new Entry API
We need to extend the ChangeLog API to get able to handle :
- on disk storage
- Entry instead of Attrinutes
didn't understand this. Atm, the LdifEntry used in changelog holds a
ClientEntry
and adds all the attributes to it. You mean to use a ServerEntry here?
- only forward operations
- fast lookup for an entry by its DN, or entryCSN/UUID attributes.
We need to allow duplicates in the DN index, there will be collisions.
How do we access CSN/UUID I think, this JournalInterceptor will be
before the
ReplicationInterceptor then we don't have access to CSN. One way
would be to
create these values in JournalInterceptor and pass on to
ReplicationInterceptor
wdyt?
We may also add a new interceptor for that purpose, namely a
JournalInterceptor.
Nothing complicated though. My idea is to use a simple file, being
rotated when we put some tag, plus some JDBM index around to be
able to lookup for DN, UUID or entryCSN.
have started experimenting with a simple RandomAccessFile based
Journal
( but the work is quite slow cause of me being lazy ;) )
I'm not sure exactly what the requirements here are but a couple of
existing projects that might provide a usable journal/log:
objectweb howl is a transaction log, basically its designed to write
"prepare" entries that have some data and "commit" entries that say
the data is no longer relevant. I imagine you could write the ldif
entry as the "prepare" record and some indication of all replicas
being up-to-date as the "commit" entry.
activemq has quite a few journal/log schemes. I think that they are
mostly using kaha. I'm don't understand all the features but think
that it would do everything you need. IIUC typically the messages
themselves are written to one "database" and for each subscription an
index entry is written to a related reference database. As the acks
come in for the subscriptions the index entries are removed and after
all deliveries have happened the main entry is removed.
I imagine you could implement the whole replication scheme using
activemq embedded brokers on each server but this might not be the
most performant implementation.
thanks
david jencks
thanks
david jencks
wdyt ?
thanks for putting the thoughts in ink, have a better understanding
now
--
Kiran Ayyagari
