Emmanuel Lecharny wrote:
Hi,
IMO this marking of journal should happen in the DirectoryService's
sync() operation.
As the sync can occur on a quite big interval of time (let say 15 secs
by default), if we depend on it to add a checkpoint, that mean we may
lose 15 secs of modifications. It may be seen as accepteblae, but
IMHO, the idea is to store logs as fast as possible, and when a
modification is considered as done, then add a checkpoint. This will
limitate the potential loss of information.
sorry if my initial description gave a different perspective. The data *will*
be written to journal as soon as an
operation(add/del/mod) is done, the 'mark' operation is just to put a
checkpoint in the journal to tell us that the DIT
data present till this checkpoint is flushed to jdbm files on disk. So a
journal *always* contains the data irrespective
of the state of jdbm file(s).
But I want to put the 'mark' only when jdbm files are synced so that we can
figure out which records weren't flushed to
the jdbm file(s).
A change to the partition's sync() method to call DirectoryService's
sync() which intern calls (each) partition's
commit() (a new method) would help. A special flag like 'isDirty' in the
partition will allow us to avoid calling
commit() on each partition.
Any other ideas about how best we can maintain a checkpoint/mark after
*any* sync operation on DIT data?.
when the mod is considered done (even if not written into the
backend), we should consider the operation valid, and then write it on
the log, and when done, adding a checkpoint. That should be done for
every modification, whatever it costs.
yep this is the case
IMO, JDBM should be used to help access to modifications operation,
but the master table should not contain the real data, but just a
pointer to another file in which modification are written in a
sequential way. We just keep an offset into the Mastertable. If
something goes really wrong, we can rebuild the master table and all
the indexes from this sequential file.
yeah, it would be great if we can avoid having the complete data in a
sequential file
and btw, the howl journal file holds some data in clear text format which might
be a security concern
(yes, we do have cryptography but thats a different thing altogether ;) )
This can be discussed, I'm just speaking my mind here, not imposing
any solution.
thanks Emmanuel
Kiran Ayyagari
