Re: How do you synchronize your notmuch tags across multiple machines?
David Mazieres writes: > Dan Čermák writes: > >> Could you maybe elaborate in more detail, as I don't know how it would >> help me exactly? Are you running a server for syncing your tags? >> >> My problem is the following: I have 3 machines E, B and C of which none >> is always up and running. I would like to be able to sync my email on >> any of the machines with offlineimap (but always only with one syncing) >> and to transfer what's missing to the others, once I switch the >> currently used machine. > > The problem is that muchsync doesn't know about imap or offlineimap. > Muchsync will happily ensure that all of your notmuch maildirs and tags > are synchronized. But if you run offlineimap on E and then sync E to B, > B will already have a copy of all new messages. If you run offlineimap > on B then, it will download additional copies of the messages already > synchronized over muchsync. > > One solution would be to archive all the messages downloaded with > offlineimap as you download them. But then you have to remember to run > muchsync to upload them to other machines, or you risk losing track of > messages if you download them to a machine that you then immediately > turn off. > > I'm open to other ideas if they require some kind of new feature from > muchsync, but I'm worried that this would require delving into the guts > of offlineimap (and then wouldn't work with other solutions like isync). > I have thought of an alternative: couldn't muchsync be taught to not synchronize the mail from certain directories but only the tags? I think this would help some of the users of notmuch: they could synchronize their mailboxes via offlineimap/isync/$other_tool and the tags (and non IMAP based messages) via muchsync. If this is possible and viable, I'd definitely help out in implementing this. > The ideal solution would be to implement an imap server on top of > libnotmuch. If we had that, then you could just use offlineimap and > isync through the imap (as opposed to file system) interface, and > everything will just work. I have experimented with something like this a little bit, but instead of using an imap server on top of libnotmuch, I simply used a local dovecot and deliver all my mail via offlineimap to subfolders of the local IMAP server. The issue is: it does not work with muchsync + offlineimap. The problem is exactly the same as when synchronizing via the file system: muchsync will create new messages in dovecot's maildir, but offlineimap does not know about these. Consequently it will start duplicating messages, because they have different IMAP IDs. This could be maybe made to work when both endpoints were libnotmuch based IMAP servers, but I don't have and probably will never have that. I really need to synchronize email from arbitrary IMAP servers, as I can't put my work email on a private email server. > That would also have the benefit of making > notmuch work really well with phones--you could use the gmail app on > your phone and the emacs/vim front-end on your desktop. It's even > conceivable that such an imap server could use notmuch's indexing to > support the gmail imap extensions: > > https://developers.google.com/gmail/imap/imap-extensions > > Unfortunately, implementing an imap server is a bit beyond the scope of > muchsync and not something I have time for right now. > > David signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
So --
the client runs something like this:
//---
#!/bin/bash
REMOTE_HOME=/home/myself
LOCAL_HOME=/home/myself
if [ "$#" -ne 1 ]
then
echo "Please give the maildir name!"
exit 1
fi
declare -A local_config
local_config=(
["personal"]="$LOCAL_HOME/.notmuch-config.personal"
["work"]="$LOCAL_HOME/.notmuch-config.work" )
declare -A remote_program
remote_program=(
["personal"]="$REMOTE_HOME/config/email/syncmail-wrapper-personal.sh"
["work"]="$REMOTE_HOME/config/email/syncmail-wrapper-work.sh" )
arg=$1
muchsync -C ${local_config[$arg]} -r ${remote_program[$arg]} -- myserver.org
//---
And in the server, ~/config/email/syncmail-wrapper-work.sh for example
is:
//---
#!/bin/sh
CONFIG=/home/myself/.notmuch-config.work
MAILDIR=/var/mail/vmail/my_mailbox@work
LOCK_WAIT_MAX=5
MUCHSYNC=/usr/bin/muchsync
PID=`/usr/lib/dovecot/maildirlock ${MAILDIR} ${LOCK_WAIT_MAX}`
if [ -z "$PID" ]
then
echo "Maildir was locked for more than ${LOCK_WAIT_MAX} seconds"
else
$MUCHSYNC --config=$CONFIG $@
kill $PID
fi
//---
J.
On Fri, Jan 18, 2019 at 09:02:11PM -0700, jpellegrini wrote:
> Hello,
>
> Sorry, it seems that the code in me previous email was removed by nabble.
>
> I'll post again from a real email client :-)
>
>
> Regarding your first question: yes, it's my primary mail server. I use a
> virtual machine
> at Linode.
>
> J.
>
>
>
> --
> Sent from: http://notmuch.198994.n3.nabble.com/
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Hello, Sorry, it seems that the code in me previous email was removed by nabble. I'll post again from a real email client :-) Regarding your first question: yes, it's my primary mail server. I use a virtual machine at Linode. J. -- Sent from: http://notmuch.198994.n3.nabble.com/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
My solution to this problem is to have only one notmuch instance run on a remote VPS and connect to it via "thin" notmuch-emacs front-ends, very similar to [1]. It has been working very well for the last few years. [1] https://notmuchmail.org/remoteusage/ On Mon, 17 Dec 2018, Dan Čermák wrote: Hi list, first and foremost: thanks to everyone who contributed to notmuch, it's a truly awesome piece of software. Unfortunately, I am facing the problem (which I have been postponing to face for a while) that I'll need to keep my notmuch database in sync over multiple machines. Thus my question: how are you achieving that? How well does your solution work? (I have found muchsync, but unfortunately very little reports about how well it works, which isn't necessarily a bad thing. Beside that only some scripts around notmuch dump.) Thanks in advance, Dan ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Perhaps a bit late to reply to this, but I believe this can be useful: I managed to sync one machine which runs postfix and receives my email, and other three (home desktop, work desktop and notebook). - the server runs Postfix and Dovecot - muchsync is configured to synchronize local maildirs in the clients with the same maildir which is used by postfix and dovecot - the only synchronization I saw as necessary was to not let 2 different muchsync clients, or the dovecot server, to access the maildir at the same time (postfix never removes or edits messages, it only drops them in the maildir). since dovecot /does/ provide a maildir locking program, this was actually easy to do: the clients run this: in the server, the program that muchsync runs (the $remote_program above) is - also, since both myuser, dovecote and postfix will write to the same directory, I had to configure dovecot and postfix to use my uid and gid (I actually have these in a postfixadmin setup, so they're in adatabase). So far it works fine -- because I don't need to use IMAP all the time. I mostly use notmuch-aware clientes in the desktops and notebook, and only use IMAP from the phone. Hope this is useful. J. -- Sent from: http://notmuch.198994.n3.nabble.com/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Hi, I too, am using muchsync to synchronize between two machine, I too have profited from David's quick responses, and I too deal with the problem of only being able to "fetch mail" on one machine. I followed the discussion with interest. Looking at the isync (mbsync) docs (I can't speak for offlineimap) it might be easier: the state of the sync seems to be kept in little files per Maildir subdirectory, called .mbsyncstate and .uivalidity. If muchsync were to sync these files as well, then I think it would work. See the explanation of the .mbsyncstate file in this thread: https://sourceforge.net/p/isync/mailman/message/34682807/ quote: "mbsync maintains a mapping of remote (col 1) to local (col 2) uids. when you migrate, you can just duplicate the columns." He's talking about migrating a mail store from offlineimap to mbsync, I guess the issues would be the same. HTH, JT ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Dan Čermák writes: >> The ideal solution would be to implement an imap server on top of >> libnotmuch. If we had that, then you could just use offlineimap and >> isync through the imap (as opposed to file system) interface, and >> everything will just work. > > Stupid question: how would the tags be synced with your local machines > in this case? Via muchsync or would one keep the tags on the > notmuch-imap server? However you want. If you run a local imap server and a remote imap server, then you can just sync between the two imap servers with any imap synching tool. But if the two hosts happen to be running imap servers implemented on top of libnotmuch, then you could use muchsync which would be faster. > How about a hacky and not ideal solution: "teach" notmuch to not only > synchronize the read and deleted tags with the maildir, but all tags? Unfortunately, there's no standard for how to encode these. Also, there's a pretty fundamental design decision in notmuch that it doesn't edit the mail messages. Now some imap servers, like dovecot, define other flags on a per-maildir basis, but it only works up to a certain number of flags, and you'd have to parse an extra file that tells you what the mappings are. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
David Mazieres writes: > Dan Čermák writes: > >> Could you maybe elaborate in more detail, as I don't know how it would >> help me exactly? Are you running a server for syncing your tags? >> >> My problem is the following: I have 3 machines E, B and C of which none >> is always up and running. I would like to be able to sync my email on >> any of the machines with offlineimap (but always only with one syncing) >> and to transfer what's missing to the others, once I switch the >> currently used machine. > > The problem is that muchsync doesn't know about imap or offlineimap. > Muchsync will happily ensure that all of your notmuch maildirs and tags > are synchronized. But if you run offlineimap on E and then sync E to B, > B will already have a copy of all new messages. If you run offlineimap > on B then, it will download additional copies of the messages already > synchronized over muchsync. > > One solution would be to archive all the messages downloaded with > offlineimap as you download them. But then you have to remember to run > muchsync to upload them to other machines, or you risk losing track of > messages if you download them to a machine that you then immediately > turn off. > > I'm open to other ideas if they require some kind of new feature from > muchsync, but I'm worried that this would require delving into the guts > of offlineimap (and then wouldn't work with other solutions like isync). > I am afraid that you are right in terms of delving into offlineimap/isync/etc. I have experimented a little with offlineimap + muchsync, but I couldn't get offlineimap to recognize messages that muchsync downloaded. My guess is that offlineimap keeps a separate UID database somewhere, that muchsync would have to write to. Messing with offlineimap's internal state sounds wrong to me. > The ideal solution would be to implement an imap server on top of > libnotmuch. If we had that, then you could just use offlineimap and > isync through the imap (as opposed to file system) interface, and > everything will just work. Stupid question: how would the tags be synced with your local machines in this case? Via muchsync or would one keep the tags on the notmuch-imap server? > That would also have the benefit of making > notmuch work really well with phones--you could use the gmail app on > your phone and the emacs/vim front-end on your desktop. It's even > conceivable that such an imap server could use notmuch's indexing to > support the gmail imap extensions: > > https://developers.google.com/gmail/imap/imap-extensions > > Unfortunately, implementing an imap server is a bit beyond the scope of > muchsync and not something I have time for right now. > > David How about a hacky and not ideal solution: "teach" notmuch to not only synchronize the read and deleted tags with the maildir, but all tags? E.g. by injecting a header into the email? I would guess that this is simpler to implement then an IMAP server, but it carries its own set of problems (iirc there were discussions about such an idea on this list but were discarded for reasons that I don't remember). For instance I don't know whether one could even push tags to the server, as one would have push the whole message back. signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Could you maybe elaborate in more detail, as I don't know how it would help me exactly? Are you running a server for syncing your tags? My problem is the following: I have 3 machines E, B and C of which none is always up and running. I would like to be able to sync my email on any of the machines with offlineimap (but always only with one syncing) and to transfer what's missing to the others, once I switch the currently used machine. Cheers, Dan Brian Sniffen writes: > I had the same problem. I found a solution: dispense with imap. This can be > in two ways: > > For work, I use getmail (and imap) to read messages from an Exchange server > and drop them in a maildir. Muchsync then coordinates those. > > For personal mail, I just use muchsync to directly manipulate Dovecot’s > maildir. > > Both work fine. In the “work” case, changes are not pushed back up to the > server. > > -- > Brian Sniffen > >> On Jan 3, 2019, at 5:27 PM, Dan Čermák wrote: >> >> I have just given muchsync a try and it synchronizes email and tags very >> quickly. I am quite impressed by it! >> >> Unfortunately, I have hit exactly the same problem that you describe: I >> have to have a single machine that pulls in my email via offlineimap and >> then sync to the others via muchsync. That is a little inconvenient, as >> I was hoping that I could switch the "master" depending on which machine >> I am currently using and not having it up and online at all times. >> >> I am afraid that that's not going to be easy to accomplish, as it >> requires muchsync and a maildir synchronization program (in my case >> offlineimap) to play together. The problem with offlineimap seems to be >> that it expects the maildir filenames to have a different form and does >> not recognize those that muchsync created (and it instead tries to clone >> all my inboxes again). In case I'll manage to get muchsync to work >> without a dedicated master, I'll let the list know. >> >> >> Cheers, >> >> Dan >> >> Tom Hirschowitz writes: >> >>> I also use muchsync. Had a few issues with it in the beginning but the >>> author, David Mazières, was quite efficient in fixing them, and patient >>> with me being awkwardly incompetent. It's been working like a charm >>> since then. >>> >>> It took me some time to figure out a working set up though. My mail is >>> now fetched first on the same machine everytime, and then synchronised >>> from other machines through muchsync. This is a bit annoying, and I'd be >>> curious if anyone had a better way. >>> >>> Tom >>> * Dan Čermák: > I have found muchsync, but unfortunately very little reports about how > well it works, which isn't necessarily a bad thing. Muchsync works well for me, although I only need to sync between two machines. It is quite fast after the initial synchronisation, and I did not have any problems yet. I reported a small error in the documentation, but that should be fixed by now. -Ralph ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch >>> ___ >>> notmuch mailing list >>> notmuch@notmuchmail.org >>> https://notmuchmail.org/mailman/listinfo/notmuch >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
I had the same problem. I found a solution: dispense with imap. This can be in two ways: For work, I use getmail (and imap) to read messages from an Exchange server and drop them in a maildir. Muchsync then coordinates those. For personal mail, I just use muchsync to directly manipulate Dovecot’s maildir. Both work fine. In the “work” case, changes are not pushed back up to the server. -- Brian Sniffen > On Jan 3, 2019, at 5:27 PM, Dan Čermák wrote: > > I have just given muchsync a try and it synchronizes email and tags very > quickly. I am quite impressed by it! > > Unfortunately, I have hit exactly the same problem that you describe: I > have to have a single machine that pulls in my email via offlineimap and > then sync to the others via muchsync. That is a little inconvenient, as > I was hoping that I could switch the "master" depending on which machine > I am currently using and not having it up and online at all times. > > I am afraid that that's not going to be easy to accomplish, as it > requires muchsync and a maildir synchronization program (in my case > offlineimap) to play together. The problem with offlineimap seems to be > that it expects the maildir filenames to have a different form and does > not recognize those that muchsync created (and it instead tries to clone > all my inboxes again). In case I'll manage to get muchsync to work > without a dedicated master, I'll let the list know. > > > Cheers, > > Dan > > Tom Hirschowitz writes: > >> I also use muchsync. Had a few issues with it in the beginning but the >> author, David Mazières, was quite efficient in fixing them, and patient >> with me being awkwardly incompetent. It's been working like a charm >> since then. >> >> It took me some time to figure out a working set up though. My mail is >> now fetched first on the same machine everytime, and then synchronised >> from other machines through muchsync. This is a bit annoying, and I'd be >> curious if anyone had a better way. >> >> Tom >> >>> * Dan Čermák: >>> I have found muchsync, but unfortunately very little reports about how well it works, which isn't necessarily a bad thing. >>> >>> Muchsync works well for me, although I only need to sync between two >>> machines. It is quite fast after the initial synchronisation, and I >>> did not have any problems yet. I reported a small error in the >>> documentation, but that should be fixed by now. >>> >>> -Ralph >>> ___ >>> notmuch mailing list >>> notmuch@notmuchmail.org >>> https://notmuchmail.org/mailman/listinfo/notmuch >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch > ___ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
I have just given muchsync a try and it synchronizes email and tags very quickly. I am quite impressed by it! Unfortunately, I have hit exactly the same problem that you describe: I have to have a single machine that pulls in my email via offlineimap and then sync to the others via muchsync. That is a little inconvenient, as I was hoping that I could switch the "master" depending on which machine I am currently using and not having it up and online at all times. I am afraid that that's not going to be easy to accomplish, as it requires muchsync and a maildir synchronization program (in my case offlineimap) to play together. The problem with offlineimap seems to be that it expects the maildir filenames to have a different form and does not recognize those that muchsync created (and it instead tries to clone all my inboxes again). In case I'll manage to get muchsync to work without a dedicated master, I'll let the list know. Cheers, Dan Tom Hirschowitz writes: > I also use muchsync. Had a few issues with it in the beginning but the > author, David Mazières, was quite efficient in fixing them, and patient > with me being awkwardly incompetent. It's been working like a charm > since then. > > It took me some time to figure out a working set up though. My mail is > now fetched first on the same machine everytime, and then synchronised > from other machines through muchsync. This is a bit annoying, and I'd be > curious if anyone had a better way. > > Tom > >> * Dan Čermák: >> >>> I have found muchsync, but unfortunately very little reports about how >>> well it works, which isn't necessarily a bad thing. >> >> Muchsync works well for me, although I only need to sync between two >> machines. It is quite fast after the initial synchronisation, and I >> did not have any problems yet. I reported a small error in the >> documentation, but that should be fixed by now. >> >> -Ralph >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch > ___ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
I also use muchsync. Had a few issues with it in the beginning but the author, David Mazières, was quite efficient in fixing them, and patient with me being awkwardly incompetent. It's been working like a charm since then. It took me some time to figure out a working set up though. My mail is now fetched first on the same machine everytime, and then synchronised from other machines through muchsync. This is a bit annoying, and I'd be curious if anyone had a better way. Tom > * Dan Čermák: > >> I have found muchsync, but unfortunately very little reports about how >> well it works, which isn't necessarily a bad thing. > > Muchsync works well for me, although I only need to sync between two > machines. It is quite fast after the initial synchronisation, and I > did not have any problems yet. I reported a small error in the > documentation, but that should be fixed by now. > > -Ralph > ___ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
* Dan Čermák: > I have found muchsync, but unfortunately very little reports about how > well it works, which isn't necessarily a bad thing. Muchsync works well for me, although I only need to sync between two machines. It is quite fast after the initial synchronisation, and I did not have any problems yet. I reported a small error in the documentation, but that should be fixed by now. -Ralph ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
