Second draft of logging patches
On Sat, 13 Nov 2010 06:25:57 +0100, Michal Sojka wrote: > On Wed, 10 Nov 2010, David Bremner wrote: > > I had a thought of a possibly interesting application of the (yet to be > > written) log playback code. It could be use to implement a simple > > queuing system where commands are only logged but not actually run on > > the database. > Hi, I think this could be very interesting and that it could make the > tagging operation asynchronous and thus faster as was suggested by > Sebastian in id:"87k4l5whwe.fsf at SSpaeth.de". > > In this case, however, the queuing should happen in the client, not in > the library. Agreed. The current implementation of automagically logging things in the library would be undesirable here. That specific thing shouldn't be too hard to change. Also worth thinking about is (optionally) using the Maildir to log most tag changes; Maildirs are designed (imperfectly, but still) to support simulataneous writes. The database could then catch up by reading the Maildir. d
Second draft of logging patches
On Wed, 10 Nov 2010, David Bremner wrote: > On Sun, 24 Oct 2010 18:01:02 -0300, david at tethera.net wrote: > > Here is my second try at logging, taking into account the feedback I > > got from Rob and Michal. There is definitely some tidying to do; in > > particular I know the protoypes in public headers need > > documentation. Also, I should add a configuration option to > > enable configuration by command or something like that. > > I had a thought of a possibly interesting application of the (yet to be > written) log playback code. It could be use to implement a simple > queuing system where commands are only logged but not actually run on > the database. I'm not sure about the performance implications, but it > could be interesting because it eliminates the need to have a server > running in order to eliminate write contention for the tag database. > The "queue runner" could be as simple as a cron job, or it could be > something spawned by one of the queue operations; the point would be > that queueing could continue while the snapshot of the queue was run. Hi, I think this could be very interesting and that it could make the tagging operation asynchronous and thus faster as was suggested by Sebastian in id:"87k4l5whwe.fsf at SSpaeth.de". In this case, however, the queuing should happen in the client, not in the library. -Michal
Re: Second draft of logging patches
On Sat, 13 Nov 2010 06:25:57 +0100, Michal Sojka wrote: > On Wed, 10 Nov 2010, David Bremner wrote: > > I had a thought of a possibly interesting application of the (yet to be > > written) log playback code. It could be use to implement a simple > > queuing system where commands are only logged but not actually run on > > the database. > Hi, I think this could be very interesting and that it could make the > tagging operation asynchronous and thus faster as was suggested by > Sebastian in id:"87k4l5whwe@sspaeth.de". > > In this case, however, the queuing should happen in the client, not in > the library. Agreed. The current implementation of automagically logging things in the library would be undesirable here. That specific thing shouldn't be too hard to change. Also worth thinking about is (optionally) using the Maildir to log most tag changes; Maildirs are designed (imperfectly, but still) to support simulataneous writes. The database could then catch up by reading the Maildir. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Second draft of logging patches
On Wed, 10 Nov 2010, David Bremner wrote: > On Sun, 24 Oct 2010 18:01:02 -0300, da...@tethera.net wrote: > > Here is my second try at logging, taking into account the feedback I > > got from Rob and Michal. There is definitely some tidying to do; in > > particular I know the protoypes in public headers need > > documentation. Also, I should add a configuration option to > > enable configuration by command or something like that. > > I had a thought of a possibly interesting application of the (yet to be > written) log playback code. It could be use to implement a simple > queuing system where commands are only logged but not actually run on > the database. I'm not sure about the performance implications, but it > could be interesting because it eliminates the need to have a server > running in order to eliminate write contention for the tag database. > The "queue runner" could be as simple as a cron job, or it could be > something spawned by one of the queue operations; the point would be > that queueing could continue while the snapshot of the queue was run. Hi, I think this could be very interesting and that it could make the tagging operation asynchronous and thus faster as was suggested by Sebastian in id:"87k4l5whwe@sspaeth.de". In this case, however, the queuing should happen in the client, not in the library. -Michal ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Second draft of logging patches
On Sun, 24 Oct 2010 18:01:02 -0300, david at tethera.net wrote: > Here is my second try at logging, taking into account the feedback I > got from Rob and Michal. There is definitely some tidying to do; in > particular I know the protoypes in public headers need > documentation. Also, I should add a configuration option to > enable configuration by command or something like that. I had a thought of a possibly interesting application of the (yet to be written) log playback code. It could be use to implement a simple queuing system where commands are only logged but not actually run on the database. I'm not sure about the performance implications, but it could be interesting because it eliminates the need to have a server running in order to eliminate write contention for the tag database. The "queue runner" could be as simple as a cron job, or it could be something spawned by one of the queue operations; the point would be that queueing could continue while the snapshot of the queue was run. d
Re: Second draft of logging patches
On Sun, 24 Oct 2010 18:01:02 -0300, da...@tethera.net wrote: > Here is my second try at logging, taking into account the feedback I > got from Rob and Michal. There is definitely some tidying to do; in > particular I know the protoypes in public headers need > documentation. Also, I should add a configuration option to > enable configuration by command or something like that. I had a thought of a possibly interesting application of the (yet to be written) log playback code. It could be use to implement a simple queuing system where commands are only logged but not actually run on the database. I'm not sure about the performance implications, but it could be interesting because it eliminates the need to have a server running in order to eliminate write contention for the tag database. The "queue runner" could be as simple as a cron job, or it could be something spawned by one of the queue operations; the point would be that queueing could continue while the snapshot of the queue was run. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Second draft of logging patches
Here is my second try at logging, taking into account the feedback I got from Rob and Michal. There is definitely some tidying to do; in particular I know the protoypes in public headers need documentation. Also, I should add a configuration option to enable configuration by command or something like that. It does do write-ahead logging for tag changes, based on calls to notmuch_message_(freeze|thaw). It is more or less hardcoded to output to .notmuch/log At the moment the buffering is line by line, but in principle this just needs to be changed in one place (the call to notmuch_log_open). Currently logging is disabled by default, unless a call to notmuch_database_open_log is made. In the long run, if we keep this code, then maybe the API of notmuch_database_open should be modified to optionally enable logging.
Second draft of logging patches
Here is my second try at logging, taking into account the feedback I got from Rob and Michal. There is definitely some tidying to do; in particular I know the protoypes in public headers need documentation. Also, I should add a configuration option to enable configuration by command or something like that. It does do write-ahead logging for tag changes, based on calls to notmuch_message_(freeze|thaw). It is more or less hardcoded to output to .notmuch/log At the moment the buffering is line by line, but in principle this just needs to be changed in one place (the call to notmuch_log_open). Currently logging is disabled by default, unless a call to notmuch_database_open_log is made. In the long run, if we keep this code, then maybe the API of notmuch_database_open should be modified to optionally enable logging. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
