[notmuch] RFC: output json from notmuch?

2009-12-14 Thread David Bremner
On Mon, 14 Dec 2009 14:30:45 -0800, Carl Worth  wrote:

> I'm still not sure of the need to depend on a library just to *generate*
> any particular format. I would expect that job to be so constrained as
> to be almost trivial. I won't necessarily block a patch based on that,
> but I think we should at least look at how easy it would be to just emit
> the syntax manually.

OK, I'll wait for Scott's patch, and see how he is using cJSON. Just
stealing a few functions might be the way to go.

The cJSON code is less horrifying after running through indent. Now all
we need is "indent --carl" :)

d



[notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 20:42:07 -0400, David Bremner  wrote:
> On a different topic (and in response to Carl and your earlier
> discussion), as a counter-weight to the desire to avoid dependencies
> (which I agree with), I also think we should be careful about how many
> embedded copies of code there are, from the point of keeping up with
> security problems.  This is probably a bias I picked up from Debian.

Embedded copies of code already packaged in Debian is something to
avoid, definitely. I'm not actively against having dependencies for
functionality that makes sense.

For example, the current libsha1 code in notmuch has been the subject of
a debate about embedded code copies here on the list already. If
somebody would like to maintain that code as a Debian package, then I
would be happy to depend on it that way rather than having an embedded
copy inside notmuch.

For something like JSON output, I really can't see how we need an
external library. The job we're talking about is changing our current
delimiters and then fixing our code to properly quote delimiters
appearing in the content. That doesn't sound like a job that needs a
library.

(Meanwhile, if we were parsing JSON, then I'd be happy to depend on a
library for that.)

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



[notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson  
wrote:
> I have a patch for a --output=(text|json) for both notmuch-show and
> notmuch-search. I mentioned it earlier on the list, and no one seemed to have
> any interest.

Hi Scott,

I remember you mentioning this earlier, but I didn't remember seeing the
actual patch? Did I miss it? I also didn't find it again when searching
and re-reading the thread. hopefully this isn't a case of my mail
client's search feature failing me... ;-)

> Is it worth updating to HEAD and trying again?

If you'd like to, please feel free. I'd be glad to evaluate different
options here.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



[notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner  wrote:
> It would be nice to have more structured output from notmuch-show.  I
> decided to investigate sexp (i.e. lisp) and json output.

Thanks, David!

> Then I found that json parsing is provided by the library json.el
> shipped with emacs23, so I decided to play with json a bit.

That sounds very promising. I wasn't aware this existed inside emacs
already.

> I settled on the jansson library (http://www.digip.org/jansson/)
> because it seemed to have a sane api and documentation, but there are
> many choices.  I'll follow up with the actual patch, but the idea is to
> replace the printfs in show_message with calls to set (key,value) pairs
> in a json object, and output it at the end.

I'm still not sure of the need to depend on a library just to *generate*
any particular format. I would expect that job to be so constrained as
to be almost trivial. I won't necessarily block a patch based on that,
but I think we should at least look at how easy it would be to just emit
the syntax manually.

> So, do people think this is a reasonable idea to persue?

I do, yes. The current lack of full quoting is an obnoxious bug.

> 1) it adds a dependency, but not a heavy one. jansson is about 300k
> installed on debian/i386

That might be acceptable. And we might rewrite the patch to avoid it as
well. I'd like to see what that would take.

> 2) It might mean that people using emacs22 might have to score
> json.el from somewhere; I'm not sure.

Currently, it sounds like notmuch is entirely unusable from emacs22, so
I'm really not too concerned about one more thing that requires emacs
23.

> 3) There is some increase in memory use since the whole thread has
> to be built as a json object before being output.

Right. That would be another reason to just stream the output
manually. The syntax doesn't require things like knowing the size of an
array before emitting the first element, right?

> 4) Of course jansson is doing it's own reference counting memory
> managment, and not using talloc. But we already have glib...

It's not evil to not use talloc. So not an issue there.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



[notmuch] RFC: output json from notmuch?

2009-12-14 Thread Marten Veldthuis
Excerpts from David Bremner's message of Mon Dec 14 00:21:02 +0100 2009:
> So, do people think this is a reasonable idea to persue?

JSON was actually the first thing I thought of when I first saw the
output of notmuch-show. I think it's a much more natural fit for notmuch
than say, sexps, since most languages will have libraries for JSON,
which will be nicer to interfaces other than the emacs one.
-- 
- Marten


Re: [notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner da...@tethera.net wrote:
 It would be nice to have more structured output from notmuch-show.  I
 decided to investigate sexp (i.e. lisp) and json output.

Thanks, David!

 Then I found that json parsing is provided by the library json.el
 shipped with emacs23, so I decided to play with json a bit.

That sounds very promising. I wasn't aware this existed inside emacs
already.

 I settled on the jansson library (http://www.digip.org/jansson/)
 because it seemed to have a sane api and documentation, but there are
 many choices.  I'll follow up with the actual patch, but the idea is to
 replace the printfs in show_message with calls to set (key,value) pairs
 in a json object, and output it at the end.

I'm still not sure of the need to depend on a library just to *generate*
any particular format. I would expect that job to be so constrained as
to be almost trivial. I won't necessarily block a patch based on that,
but I think we should at least look at how easy it would be to just emit
the syntax manually.

 So, do people think this is a reasonable idea to persue?

I do, yes. The current lack of full quoting is an obnoxious bug.

 1) it adds a dependency, but not a heavy one. jansson is about 300k
 installed on debian/i386

That might be acceptable. And we might rewrite the patch to avoid it as
well. I'd like to see what that would take.

 2) It might mean that people using emacs22 might have to score
 json.el from somewhere; I'm not sure.

Currently, it sounds like notmuch is entirely unusable from emacs22, so
I'm really not too concerned about one more thing that requires emacs
23.

 3) There is some increase in memory use since the whole thread has
 to be built as a json object before being output.

Right. That would be another reason to just stream the output
manually. The syntax doesn't require things like knowing the size of an
array before emitting the first element, right?

 4) Of course jansson is doing it's own reference counting memory
 managment, and not using talloc. But we already have glib...

It's not evil to not use talloc. So not an issue there.

-Carl


pgpf3vKQgsEeq.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson sc...@quadhome.com wrote:
 I have a patch for a --output=(text|json) for both notmuch-show and
 notmuch-search. I mentioned it earlier on the list, and no one seemed to have
 any interest.

Hi Scott,

I remember you mentioning this earlier, but I didn't remember seeing the
actual patch? Did I miss it? I also didn't find it again when searching
and re-reading the thread. hopefully this isn't a case of my mail
client's search feature failing me... ;-)

 Is it worth updating to HEAD and trying again?

If you'd like to, please feel free. I'd be glad to evaluate different
options here.

-Carl


pgpJXn1ZOgOKy.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-14 Thread Carl Worth
On Sun, 13 Dec 2009 20:42:07 -0400, David Bremner brem...@unb.ca wrote:
 On a different topic (and in response to Carl and your earlier
 discussion), as a counter-weight to the desire to avoid dependencies
 (which I agree with), I also think we should be careful about how many
 embedded copies of code there are, from the point of keeping up with
 security problems.  This is probably a bias I picked up from Debian.

Embedded copies of code already packaged in Debian is something to
avoid, definitely. I'm not actively against having dependencies for
functionality that makes sense.

For example, the current libsha1 code in notmuch has been the subject of
a debate about embedded code copies here on the list already. If
somebody would like to maintain that code as a Debian package, then I
would be happy to depend on it that way rather than having an embedded
copy inside notmuch.

For something like JSON output, I really can't see how we need an
external library. The job we're talking about is changing our current
delimiters and then fixing our code to properly quote delimiters
appearing in the content. That doesn't sound like a job that needs a
library.

(Meanwhile, if we were parsing JSON, then I'd be happy to depend on a
library for that.)

-Carl


pgpUv5TfbA3jw.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-14 Thread David Bremner
On Mon, 14 Dec 2009 14:30:45 -0800, Carl Worth cwo...@cworth.org wrote:

 I'm still not sure of the need to depend on a library just to *generate*
 any particular format. I would expect that job to be so constrained as
 to be almost trivial. I won't necessarily block a patch based on that,
 but I think we should at least look at how easy it would be to just emit
 the syntax manually.

OK, I'll wait for Scott's patch, and see how he is using cJSON. Just
stealing a few functions might be the way to go.

The cJSON code is less horrifying after running through indent. Now all
we need is indent --carl :)

d

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner
On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson  
wrote:

> I have a patch for a --output=(text|json) for both notmuch-show and
> notmuch-search. I mentioned it earlier on the list, and no one seemed to have
> any interest.

Ahh, I missed that. I think it was just before I subscribed to the list,
and I skipped over the subject about perl and python.

> Is it worth updating to HEAD and trying again?

Sure, it sounds more complete than what I have. You could also wait and
see what feedback this thread gathers.

On a different topic (and in response to Carl and your earlier
discussion), as a counter-weight to the desire to avoid dependencies
(which I agree with), I also think we should be careful about how many
embedded copies of code there are, from the point of keeping up with
security problems.  This is probably a bias I picked up from Debian.


-- 
David Bremner  Professor, UNB Computer Science
bremner at unb.ca  
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director


[notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner
On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner  wrote:
> 
> Attachments:  json output of a message from carl, and the equivalent
> sexpr as parsed by json.el.  
> 
uh, right. Second time lucky.
-- next part --
A non-text attachment was scrubbed...
Name: msg.json
Type: application/octet-stream
Size: 2282 bytes
Desc: json encoded message
URL: 

-- next part --
A non-text attachment was scrubbed...
Name: msg.sexpr
Type: application/octet-stream
Size: 2064 bytes
Desc: msg converted to sexpr by json.el
URL: 

-- next part --

-- 
David Bremner  Professor, UNB Computer Science
bremner at unb.ca  
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director


[notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner

It would be nice to have more structured output from notmuch-show.  I
decided to investigate sexp (i.e. lisp) and json output.

Sexp has the obvious advantage that it is trivially parsable in emacs;
for other clients it is a little more work.  I started looking at
sfsexp (http://sexpr.sourceforge.net).  It looks ok; I got a little
irritated that it doesn't support cons cells (i.e. dotted pairs). 

Then I found that json parsing is provided by the library json.el
shipped with emacs23, so I decided to play with json a bit.

I settled on the jansson library (http://www.digip.org/jansson/)
because it seemed to have a sane api and documentation, but there are
many choices.  I'll follow up with the actual patch, but the idea is to
replace the printfs in show_message with calls to set (key,value) pairs
in a json object, and output it at the end.

This is not in any sense a production patch (e.g. it needs to actually
return a thread object rather than just dumping messages out; nothing at
all has been done on the emacs side), but it gives you some idea of
would be involved.

So, do people think this is a reasonable idea to persue?

1) it adds a dependency, but not a heavy one. jansson is about 300k
installed on debian/i386

2) It might mean that people using emacs22 might have to score
json.el from somewhere; I'm not sure.

3) There is some increase in memory use since the whole thread has
to be built as a json object before being output.

4) Of course jansson is doing it's own reference counting memory
managment, and not using talloc. But we already have glib...

Of course we won't really know if it is good idea until we try it, but
if it already looks like a no-go, I'll stop.

Attachments:  json output of a message from carl, and the equivalent
sexpr as parsed by json.el.  




-- 
David Bremner  Professor, UNB Computer Science
bremner at unb.ca  
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director


[notmuch] RFC: output json from notmuch?

2009-12-13 Thread Scott Robinson
Excerpts from David Bremner's message of Sun Dec 13 15:26:02 -0800 2009:
> On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner  
> wrote:
> > 
> > Attachments:  json output of a message from carl, and the equivalent
> > sexpr as parsed by json.el.  
> > 
> uh, right. Second time lucky.
> 

I have a patch for a --output=(text|json) for both notmuch-show and
notmuch-search. I mentioned it earlier on the list, and no one seemed to have
any interest.

Is it worth updating to HEAD and trying again?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/


[notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner

It would be nice to have more structured output from notmuch-show.  I
decided to investigate sexp (i.e. lisp) and json output.

Sexp has the obvious advantage that it is trivially parsable in emacs;
for other clients it is a little more work.  I started looking at
sfsexp (http://sexpr.sourceforge.net).  It looks ok; I got a little
irritated that it doesn't support cons cells (i.e. dotted pairs). 

Then I found that json parsing is provided by the library json.el
shipped with emacs23, so I decided to play with json a bit.

I settled on the jansson library (http://www.digip.org/jansson/)
because it seemed to have a sane api and documentation, but there are
many choices.  I'll follow up with the actual patch, but the idea is to
replace the printfs in show_message with calls to set (key,value) pairs
in a json object, and output it at the end.

This is not in any sense a production patch (e.g. it needs to actually
return a thread object rather than just dumping messages out; nothing at
all has been done on the emacs side), but it gives you some idea of
would be involved.

So, do people think this is a reasonable idea to persue?

1) it adds a dependency, but not a heavy one. jansson is about 300k
installed on debian/i386

2) It might mean that people using emacs22 might have to score
json.el from somewhere; I'm not sure.

3) There is some increase in memory use since the whole thread has
to be built as a json object before being output.

4) Of course jansson is doing it's own reference counting memory
managment, and not using talloc. But we already have glib...

Of course we won't really know if it is good idea until we try it, but
if it already looks like a no-go, I'll stop.

Attachments:  json output of a message from carl, and the equivalent
sexpr as parsed by json.el.  




-- 
David Bremner  Professor, UNB Computer Science
brem...@unb.ca 
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner
On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner da...@tethera.net wrote:
 
 Attachments:  json output of a message from carl, and the equivalent
 sexpr as parsed by json.el.  
 
uh, right. Second time lucky.


msg.json
Description: json encoded message


msg.sexpr
Description: msg converted to sexpr by json.el

-- 
David Bremner  Professor, UNB Computer Science
brem...@unb.ca 
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-13 Thread Scott Robinson
Excerpts from David Bremner's message of Sun Dec 13 15:26:02 -0800 2009:
 On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner da...@tethera.net wrote:
  
  Attachments:  json output of a message from carl, and the equivalent
  sexpr as parsed by json.el.  
  
 uh, right. Second time lucky.
 

I have a patch for a --output=(text|json) for both notmuch-show and
notmuch-search. I mentioned it earlier on the list, and no one seemed to have
any interest.

Is it worth updating to HEAD and trying again?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-13 Thread Marten Veldthuis
Excerpts from David Bremner's message of Mon Dec 14 00:21:02 +0100 2009:
 So, do people think this is a reasonable idea to persue?

JSON was actually the first thing I thought of when I first saw the
output of notmuch-show. I think it's a much more natural fit for notmuch
than say, sexps, since most languages will have libraries for JSON,
which will be nicer to interfaces other than the emacs one.
-- 
- Marten
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] RFC: output json from notmuch?

2009-12-13 Thread David Bremner
On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson sc...@quadhome.com wrote:

 I have a patch for a --output=(text|json) for both notmuch-show and
 notmuch-search. I mentioned it earlier on the list, and no one seemed to have
 any interest.

Ahh, I missed that. I think it was just before I subscribed to the list,
and I skipped over the subject about perl and python.

 Is it worth updating to HEAD and trying again?

Sure, it sounds more complete than what I have. You could also wait and
see what feedback this thread gathers.

On a different topic (and in response to Carl and your earlier
discussion), as a counter-weight to the desire to avoid dependencies
(which I agree with), I also think we should be careful about how many
embedded copies of code there are, from the point of keeping up with
security problems.  This is probably a bias I picked up from Debian.


-- 
David Bremner  Professor, UNB Computer Science
brem...@unb.ca 
http://www.cs.unb.ca/~bremner   Cross Appointment, UNB Mathematics
http://www.mitacs.ca/  MITACS Atlantic Scientific Director
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch