* openchangemapidump use of LDB has been improved. It now uses a
single transaction for record commit rather than 1 for each
record attribute. This change decreases the time needed for the
tool to backup the database with a rate up to 30x.
* attachments and mail content:
I've noticed I had forgotten one important case. When the mail
content or the size of the attachments is too large to be stored
in the GetProps/GetPropsAll replies, we need to open a stream to
fetch the data. This means:
1. openchangemapidump preliminary draft is not completed
2. This is a very good opportunity to write the backend
code needed to store this streaming content in SQL like
database.
On Tue, 2007-09-11 at 18:07 +0200, Julien Kerihuel wrote:
> Hi List,
>
> I finally decide to push the openchangemapidump code at the current
> stage of development, prior I start bringing improvements. This way I
> will at least have a working copy prior I start the modifications listed
> below (-;
>
> * Too much LDB transactions:
> For the moment there is one transaction for each of the
> MAPI properties. It drastically consumes time and can be
> easily improved with a single common transaction: init
> ldb msg, fill it with all its attributes and parameters,
> then call ldb_add.
>
> * No SQL backend introduced yet:
> All MAPI data are stored in LDB and attachments are
> base64 encoded. I guess we all agree this is ugly.
> Anyway that's part of the improvement I'll work on:
> separate content from the object. Maybe add storage
> backend such as sqlite or fs.
>
> Anyway this piece of code should provide preliminary material to
> discuss/test the checksum algorithm + proposal for storage backend.
>
> How to use it?
>
> openchangemapidump -b test.ldb
> ldbedit -H test.ldb
>
> Normally data are already stored in such way we can restore them easily.
> If new elements arrives in your Mailbox store and you run
> openchangemapidump again, the database will only add new elements. Moved
> object should fail anyway => checksum and global uniqueness issue.
>
>
> I've also duplicated the mapidump_write_{container,content,attachment}
> code, so we can easily work on a single kind of mapi object rather than
> all of them.
>
> Cheers,
>
> Julien.
>
> On Tue, 2007-09-11 at 01:17 +0200, Julien Kerihuel wrote:
> > On Mon, 2007-09-10 at 11:35 +0200, Julien Kerihuel wrote:
> > > Before I push the code on the SVN, I need:
> > > - clean-up the code and fix numerous memory leaks
> > > - Add storage support for multi-valued properties and
> > > generates a LDIF schema file to handle isSingleValued.
> > > - Add some skeleton files for further use (checksum
> > > algorithm etc.)
> > > - add sample sqlite backend implementation for content
> > > storage.
> >
> > Small update on openchangemapidump:
> > * It now supports multi-value properties
> > * All properties are now dumped in a way we can easily restore
> > them.
> > * The code now store attachments (unique ID fetched from
> > PR_RECORD_KEY)
> > * LDIF schema files shouldn't be needed after all
> >
> > We are anyway still storing big data blobs in LDB (base64 encoded) and
> > we now need to move them to a SQL like database (not done yet) so the
> > tool can be reliable. I'll check tomorrow how I can add a preliminary
> > skeleton for this.
> >
> > I'll push the code on svn as soon as the skeleton of the SQL backend is
> > written.
> > -- Julien Kerihuel [EMAIL PROTECTED] OpenChange Project Manager GPG Fingerprint: 0B55 783D A781 6329 108A B609 7EF6 FE11 A35F 1F79
signature.asc
Description: This is a digitally signed message part
_______________________________________________ devel mailing list [email protected] http://mailman.openchange.org/listinfo/devel
