Remote usage script updated

2011-03-04 Thread Jesse Rosenthal
I have mine remotely, just because I don't have a local database to
query. I made a shell wrapper, pretty much just like remote script, but
that only passes the query along, and then set that script as my
address-lookup program. 

Hmm... I guess the two could be combined, but configuring for your own
address script and so on seems like it might be more trouble than just
making the script from scratch.

On Thu, 03 Mar 2011 17:38:51 +0100, Kristoffer Str?m  wrote:
> 
> Hey, one open question on the remote script, is anyone using address 
> completion with this?
> 
> ATM i'm using the vala script for this, but i don't know if integrating this 
> remotely would be the most elegant solution.
> 
> How do you running notmuch-remote make this work?
> 
> Best, Kristoffer


Re: Remote usage script updated

2011-03-04 Thread Jesse Rosenthal
I have mine remotely, just because I don't have a local database to
query. I made a shell wrapper, pretty much just like remote script, but
that only passes the query along, and then set that script as my
address-lookup program. 

Hmm... I guess the two could be combined, but configuring for your own
address script and so on seems like it might be more trouble than just
making the script from scratch.

On Thu, 03 Mar 2011 17:38:51 +0100, Kristoffer Ström kristof...@rymdkoloni.se 
wrote:
 
 Hey, one open question on the remote script, is anyone using address 
 completion with this?
 
 ATM i'm using the vala script for this, but i don't know if integrating this 
 remotely would be the most elegant solution.
 
 How do you running notmuch-remote make this work?
 
 Best, Kristoffer
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-03-03 Thread Kristoffer Ström

Hey, one open question on the remote script, is anyone using address completion 
with this?

ATM i'm using the vala script for this, but i don't know if integrating this 
remotely would be the most elegant solution.

How do you running notmuch-remote make this work?

Best, Kristoffer


Re: Remote usage script updated

2011-03-03 Thread Kristoffer Ström

Hey, one open question on the remote script, is anyone using address completion 
with this?

ATM i'm using the vala script for this, but i don't know if integrating this 
remotely would be the most elegant solution.

How do you running notmuch-remote make this work?

Best, Kristoffer
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-02-15 Thread Michal Sojka
On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
> Michal,
> 
> On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka  
> wrote:
> > Hmm, this code worked well with dropbear ssh server but it seems that
> > with openssh server the result is not that good. Namely, if the master
> > connection is dead, the command running true blocked for a long time.
> 
> Seemed to work okay for me when I played around with it a bit (in
> different circumstances, and with a confused laptop waking up from
> hibernation). But I'll hold off on updating it till I can figure out the
> most reliable way.

Hi Jesse,

I've just found that there is a SSH option called ControlPersist and
this is the missing piece to reliable connection sharing for notmuch
remote script. This option is available since openssh 5.6
(http://www.openssh.org/txt/release-5.6) with a fix of race conditions
in 5.7 (http://www.openssh.org/txt/release-5.7).

Now I have in my script the following command:
  ssh -x -a -oControlMaster=auto -oControlPersist=600 -S $SOCKET 
$USER@$SSH_HOST $NOTMUCH_REMOTE_BIN ${CMD} ${args}

where SOCKET="${CACHE}/.ssh-socket-${USER}@${SSH_HOST}" and it works
perfectly.

Unfortunately, this option is not supported in recently released Debian
Squeeze but from http://bugs.debian.org/594295 seems that it might be
supported later, perhaps in a point release.

-Michal


Re: Remote usage script updated

2011-02-15 Thread Michal Sojka
On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
 Michal,
 
 On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka sojk...@fel.cvut.cz wrote:
  Hmm, this code worked well with dropbear ssh server but it seems that
  with openssh server the result is not that good. Namely, if the master
  connection is dead, the command running true blocked for a long time.
 
 Seemed to work okay for me when I played around with it a bit (in
 different circumstances, and with a confused laptop waking up from
 hibernation). But I'll hold off on updating it till I can figure out the
 most reliable way.

Hi Jesse,

I've just found that there is a SSH option called ControlPersist and
this is the missing piece to reliable connection sharing for notmuch
remote script. This option is available since openssh 5.6
(http://www.openssh.org/txt/release-5.6) with a fix of race conditions
in 5.7 (http://www.openssh.org/txt/release-5.7).

Now I have in my script the following command:
  ssh -x -a -oControlMaster=auto -oControlPersist=600 -S $SOCKET 
$USER@$SSH_HOST $NOTMUCH_REMOTE_BIN ${CMD} ${args}

where SOCKET=${CACHE}/.ssh-socket-${USER}@${SSH_HOST} and it works
perfectly.

Unfortunately, this option is not supported in recently released Debian
Squeeze but from http://bugs.debian.org/594295 seems that it might be
supported later, perhaps in a point release.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-02-11 Thread Michal Sojka
On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
> By the way, I've also realized that most attachment downloading is done
> not by "--format=raw" but by "notmuch part". It's possible to do caching
> there as well. There are a few options there:
> 
>   * One option would be to just cache by the attachment number -- but
> this is very fragile if you delete an attachment through mutt or
> some other client that allows it.

This is obviously the simplest possibility. I do not know how often do
you (and others) delete attachments, but I do it rarely and usually I
delete all the attachments of the messages older than something. So this
would work for me.

Additionally, I have added the following command to the attachment
fetching code: 

find ${CACHE} -ctime +$CACHE_MAX_DAYS -and -mtime +$CACHE_MAX_DAYS -print0 
| xargs -r -0 rm -f

It deletes attachments older than one week (in my case) so it there is
some inconsistency, it would not last more than one week.

>   * cache by the hash of the attachment. The idea is that asking the
> server to fetch it, hash it, and send the hash would still save
> time over sending the whole attachment. Probably -- though most
> attachments are small enough and most connections are fast enough
> that this might not actually matter.

If people do more weird things with their attachments, this is probably
the best solution.

>   * Actually stick the attachment hash in the json output in the first
> place. But this would be a lot of trouble for a small gain for
> very few.

Agreed.

-Michal


Re: Remote usage script updated

2011-02-10 Thread Michal Sojka
On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
 By the way, I've also realized that most attachment downloading is done
 not by --format=raw but by notmuch part. It's possible to do caching
 there as well. There are a few options there:
 
   * One option would be to just cache by the attachment number -- but
 this is very fragile if you delete an attachment through mutt or
 some other client that allows it.

This is obviously the simplest possibility. I do not know how often do
you (and others) delete attachments, but I do it rarely and usually I
delete all the attachments of the messages older than something. So this
would work for me.

Additionally, I have added the following command to the attachment
fetching code: 

find ${CACHE} -ctime +$CACHE_MAX_DAYS -and -mtime +$CACHE_MAX_DAYS -print0 
| xargs -r -0 rm -f

It deletes attachments older than one week (in my case) so it there is
some inconsistency, it would not last more than one week.

   * cache by the hash of the attachment. The idea is that asking the
 server to fetch it, hash it, and send the hash would still save
 time over sending the whole attachment. Probably -- though most
 attachments are small enough and most connections are fast enough
 that this might not actually matter.

If people do more weird things with their attachments, this is probably
the best solution.

   * Actually stick the attachment hash in the json output in the first
 place. But this would be a lot of trouble for a small gain for
 very few.

Agreed.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-02-09 Thread Jesse Rosenthal
Michal,

On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka  wrote:
> Hmm, this code worked well with dropbear ssh server but it seems that
> with openssh server the result is not that good. Namely, if the master
> connection is dead, the command running true blocked for a long time.

Seemed to work okay for me when I played around with it a bit (in
different circumstances, and with a confused laptop waking up from
hibernation). But I'll hold off on updating it till I can figure out the
most reliable way.

I'll definitely take your suggestion to use printf instead of manually
escaping characters. 

Thanks, in any event, for your suggestions!

By the way, I've also realized that most attachment downloading is done
not by "--format=raw" but by "notmuch part". It's possible to do caching
there as well. There are a few options there:

  * One option would be to just cache by the attachment number -- but
this is very fragile if you delete an attachment through mutt or
some other client that allows it.

  * cache by the hash of the attachment. The idea is that asking the
server to fetch it, hash it, and send the hash would still save
time over sending the whole attachment. Probably -- though most
attachments are small enough and most connections are fast enough
that this might not actually matter.

  * Actually stick the attachment hash in the json output in the first
place. But this would be a lot of trouble for a small gain for
very few.

It would come in handy when I'm trying to get a biggish pdf over a bus's
slow wireless connection. But it's certainly not a priority.

Best,
Jesse


Re: Remote usage script updated

2011-02-09 Thread Jesse Rosenthal
Michal,

On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka sojk...@fel.cvut.cz wrote:
 Hmm, this code worked well with dropbear ssh server but it seems that
 with openssh server the result is not that good. Namely, if the master
 connection is dead, the command running true blocked for a long time.

Seemed to work okay for me when I played around with it a bit (in
different circumstances, and with a confused laptop waking up from
hibernation). But I'll hold off on updating it till I can figure out the
most reliable way.

I'll definitely take your suggestion to use printf instead of manually
escaping characters. 

Thanks, in any event, for your suggestions!

By the way, I've also realized that most attachment downloading is done
not by --format=raw but by notmuch part. It's possible to do caching
there as well. There are a few options there:

  * One option would be to just cache by the attachment number -- but
this is very fragile if you delete an attachment through mutt or
some other client that allows it.

  * cache by the hash of the attachment. The idea is that asking the
server to fetch it, hash it, and send the hash would still save
time over sending the whole attachment. Probably -- though most
attachments are small enough and most connections are fast enough
that this might not actually matter.

  * Actually stick the attachment hash in the json output in the first
place. But this would be a lot of trouble for a small gain for
very few.

It would come in handy when I'm trying to get a biggish pdf over a bus's
slow wireless connection. But it's certainly not a priority.

Best,
Jesse
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-02-06 Thread Michal Sojka
On Sat, 05 Feb 2011, Michal Sojka wrote:
> On Thu, 27 Jan 2011, Jesse Rosenthal wrote:
> > A future feature might be to integrate the ControlMaster feature of
> > openssh into the script, instead of having to open a connection
> > manually, but there are some complications there (dead sockets still
> > around if you go offline, etc.).
> 
> In another project I worked around the dead sockets this way:
> 
> sshgw() {
> local socket="$HOME/.ssh/cangw-connection"
> 
> if [[ ! -S $socket ]] || ! ssh -x -a -S $socket root at 192.168.2.3 true; 
> then
>   # Create master connection to speed up subsequent commands.
>   ssh -N -f -M -S $socket root at 192.168.2.3 >/dev/null 2>&1
> fi
> ssh -x -a -S $socket root at 192.168.2.3 "$@"
> }

Hmm, this code worked well with dropbear ssh server but it seems that
with openssh server the result is not that good. Namely, if the master
connection is dead, the command running true blocked for a long time. I
have always killed it before it timed out so I don't know whether it
blocks indefinitely or there is only long timeout.

I may look at this later, but I use notmuch remotely only from time to
time so this is not priority for me.

-Michal


Remote usage script updated

2011-02-05 Thread Michal Sojka
On Thu, 27 Jan 2011, Jesse Rosenthal wrote:
> Dear all,
> 
> Just a note to say that I finally got around to updating the remote
> usage script on the wiki to what I'm using now. With "--format=raw" in,
> it's all pretty straightforward. 

Hi Jesse,

thanks for this. I wanted to do this since the --format=raw was merged,
but didn't find the time for it. I have a few comments:

> The only things the script does now are:
> 
> 1. Produces a slight pause in the "notmuch show" output to avoid that
>weird bug where emacs leaves off every tenth message or so.

I added a comment to the script to explain the sleep there. Otherwise,
people may delete the command because they think it is not needed.


> 2. Locally caches raw messages (i.e. when --format=raw) is called. This
>usually happens when getting attachments, so this is a nice way to
>avoid having to download large attachments repeatedly.

This is a good idea.

>Note this just caches based on msg-id (or a hash thereof, to avoid
>strange characters in file names). That means that if an attachment
>is deleted on the server, the cache will be out of date. An easy way
>to fix this would be to make the cache file name a concatenation of
>the msg-id hash (check that first) and the hash of the actual message
>(check that if the msg-id hash is there). I might put this in in the
>future, especially if anyone else is using the script.
> 
> 3. Escapes dollar signs in the msg-id to make shell-quoting over ssh
>work.

I think that quoting with
  printf -v args "%q " "$@"
instead of sed would work more reliably (see bash(1)).

> I've actually switched over to keeping my messages on my IMAP server and
> using this remote script on all of my computers. It avoids any need for
> syncing. It's been working very well for me so far.
> 
> A future feature might be to integrate the ControlMaster feature of
> openssh into the script, instead of having to open a connection
> manually, but there are some complications there (dead sockets still
> around if you go offline, etc.).

In another project I worked around the dead sockets this way:

sshgw() {
local socket="$HOME/.ssh/cangw-connection"

if [[ ! -S $socket ]] || ! ssh -x -a -S $socket root at 192.168.2.3 true; 
then
# Create master connection to speed up subsequent commands.
ssh -N -f -M -S $socket root at 192.168.2.3 >/dev/null 2>&1
fi
ssh -x -a -S $socket root at 192.168.2.3 "$@"
}

I guess this should work even for notmuch.

-Michal


Re: Remote usage script updated

2011-02-05 Thread Michal Sojka
On Sat, 05 Feb 2011, Michal Sojka wrote:
 On Thu, 27 Jan 2011, Jesse Rosenthal wrote:
  A future feature might be to integrate the ControlMaster feature of
  openssh into the script, instead of having to open a connection
  manually, but there are some complications there (dead sockets still
  around if you go offline, etc.).
 
 In another project I worked around the dead sockets this way:
 
 sshgw() {
 local socket=$HOME/.ssh/cangw-connection
 
 if [[ ! -S $socket ]] || ! ssh -x -a -S $socket root@192.168.2.3 true; 
 then
   # Create master connection to speed up subsequent commands.
   ssh -N -f -M -S $socket root@192.168.2.3 /dev/null 21
 fi
 ssh -x -a -S $socket root@192.168.2.3 $@
 }

Hmm, this code worked well with dropbear ssh server but it seems that
with openssh server the result is not that good. Namely, if the master
connection is dead, the command running true blocked for a long time. I
have always killed it before it timed out so I don't know whether it
blocks indefinitely or there is only long timeout.

I may look at this later, but I use notmuch remotely only from time to
time so this is not priority for me.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-01-28 Thread Carl Worth
On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal  
wrote:
> Just a note to say that I finally got around to updating the remote
> usage script on the wiki to what I'm using now. With "--format=raw" in,
> it's all pretty straightforward. The only things the script does now
> are:

Thanks for passing on this note, Jesse!

> I've actually switched over to keeping my messages on my IMAP server and
> using this remote script on all of my computers. It avoids any need for
> syncing. It's been working very well for me so far.

It sounds entirely useful for a large class of users. Is there any
reason we shouldn't just host the script in the notmuch repository
itself? Let me know if there's anything I can do.

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



Re: Remote usage script updated

2011-01-28 Thread Jesse Rosenthal

On Fri, 28 Jan 2011 12:54:17 +1000, Carl Worth cwo...@cworth.org wrote:
 On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal jrosent...@jhu.edu wrote
 It sounds entirely useful for a large class of users. Is there any
 reason we shouldn't just host the script in the notmuch repository
 itself? Let me know if there's anything I can do.
 
 -Carl

I just realized I single-replied to Carl (after all the time I spent
making sure I didn't group-reply by accident).

Just to repeat briefly what I said there -- there was a slight problem
with yesterday's script (wild-cards were eaten by the shell) that should
now be fixed. And I'd be happy to house it in the repository if that
seems like it would be useful to people. But I'm curious about whether
we should have a separate tools/utilities folder. I know this came up
before with notmuch-deliver and other similar tools. What was the
outcome of that discussion?

Best,
Jesse
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Remote usage script updated

2011-01-27 Thread Jesse Rosenthal
Dear all,

Just a note to say that I finally got around to updating the remote
usage script on the wiki to what I'm using now. With "--format=raw" in,
it's all pretty straightforward. The only things the script does now
are:

1. Produces a slight pause in the "notmuch show" output to avoid that
   weird bug where emacs leaves off every tenth message or so.

2. Locally caches raw messages (i.e. when --format=raw) is called. This
   usually happens when getting attachments, so this is a nice way to
   avoid having to download large attachments repeatedly.

   Note this just caches based on msg-id (or a hash thereof, to avoid
   strange characters in file names). That means that if an attachment
   is deleted on the server, the cache will be out of date. An easy way
   to fix this would be to make the cache file name a concatenation of
   the msg-id hash (check that first) and the hash of the actual message
   (check that if the msg-id hash is there). I might put this in in the
   future, especially if anyone else is using the script.

3. Escapes dollar signs in the msg-id to make shell-quoting over ssh
   work.

I've actually switched over to keeping my messages on my IMAP server and
using this remote script on all of my computers. It avoids any need for
syncing. It's been working very well for me so far.

A future feature might be to integrate the ControlMaster feature of
openssh into the script, instead of having to open a connection
manually, but there are some complications there (dead sockets still
around if you go offline, etc.).

Best,
Jesse