I don't know if this helps, but... email-canonicalization.tar.gz:Files numbered in order from MTA hops, same order as the received headers. Two sets, with the patch and another without.
1- a copy of the email from Outlook Express on the sender system.2- a signing canonicalization from dkim-milter on the first sendmail MTA (two files header & body) 3- a verification canonicalization from dkim-milter on the second MTA (two files header & body) 4- a verification canonicalization from Mail::DKIM from SpamAssassin::Plugin::DKIM from milter-spamc on the third MTA
5- fourth MTA is Exchange, no debugging here6- a copy of the email from Thunderbird on the recipient system. Reviewing the patched and unpatched versions of the 6- file shows the verification process on the second MTA and the success of milter-spamc to allow DKIM_VERIFY or not to verify (respectively).
The real failure here is that dkim-milter saves it's canonicalization files as separate header and body files, while Mail::DKIM's debug mode prints --- BEGIN <snip> --- and --- END <snip> --- tags, which I've commented out, but still stores the two in the same file. This makes it almost impossible to detect what the header/body separator has been passed. So, many of these canonicalizations have limited value.
I was hoping the packet captures reveal more than this, but I don't think they do.
packet-captures.tar.gz:two traces, one with the original milter-spamc 1.10.376 release, the other with milter-spamc 1.10.376 with my patch. spamd on TCP783, milter-spamc on TCP3336. These should be independent captures of the sendmail to milter-spamc communication, the mitler-spamc to spamd communication, the spamd to milter-spamc response, and the mitler-spamc to sendmail response.
I suggest capturing the raw bytes that SpamAssassin sees, and post them here to be reproduced. --j. Ben Lentz writes:Nope, the code in the milter appears to use \r\n only.So, it's entirely possible that the messages I'm trying to verify are sent using \n, the milter changes them all to \r\n when it passes to SpamAssassin, and the DKIM checks fail.SpamAssassin will use *either* \r\n or \n, depending on what's passed to it -- but it has to be consistent. If the first header line (iirc) contains \n, it'll require that throughout. Is the milter using \r\n in some parts and \n elsewhere? --j. Ben Lentz writes:Greetings,I am currently using spamassassin 3.1.7 with SnertSoft's milter-spamc 1.10.376 and sendmail 8.13.8. I am having problems getting DKIM_VERIFIED to work, and after some trial and error, I compared the canonicalized files from a signer and verifier system, and found that the data being passed to SpamAssassin (and in turn to Plugin/DKIM.pm and Mail::DKIM) contained what appeared to be an extra CRLF between the headers and body.This is almost a "normal" problem with DKIM, as it's sensitive (by design) to signs of tampering like this.Applying a patch against the source for milter-spamc, removing what I believe is the line of code that's injecting this CRLF when the data is passed from libmitler to spamassassin:--- milter-spamc.c.orig 2006-10-17 17:07:09.000000000 -0400 +++ milter-spamc.c 2006-10-17 17:26:22.000000000 -0400 @@ -649,7 +649,7 @@ if (data->work.skipMessage) return SMFIS_CONTINUE; - (void) BufAddBytes(data->headers, "\r\n", 0, 2); + /* (void) BufAddBytes(data->headers, "\r\n", 0, 2); */ /* Insert a simulated Received: header for this server into the * header block being sent to spamd. It appears that this header Results in SpamAssassin doing this: X-Spam-Report: Content preview: test [...] ____ Content analysis details: (-0.1 points, 4.0 required) ____ pts rule name description---- ---------------------- --------------------------------------------------...snip... 2.5 MISSING_HB_SEP Missing blank line between message header and body 0.0 DKIM_POLICY_SIGNALL Domain Keys Identified Mail: policy says domain signs all mails -0.0 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.0 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.0 DKIM_POLICY_TESTING Domain Keys Identified Mail: policy says domain is testing DKWhich is a little unexpected... MISSING_HB_SEP & DKIM_VERIFIED at the same time.However, I've also discovered that if I keep this line, but change the CRLF to a LF:--- com/snert/src/milter-spamc/milter-spamc.c.orig 2006-10-18 15:03:37.000000000 -0400 +++ com/snert/src/milter-spamc/milter-spamc.c 2006-10-18 15:03:49.000000000 -0400@@ -649,7 +649,7 @@ if (data->work.skipMessage) return SMFIS_CONTINUE;- (void) BufAddBytes(data->headers, "\r\n", 0, 2);+ (void) BufAddBytes(data->headers, "\n", 0, 1);/* Insert a simulated Received: header for this server into the* header block being sent to spamd. It appears that this headerEverything appears to work, MISSING_HB_SEP goes away, and DKIM_VERIFIED works for signed mail.However, after providing all this information to Anthony Howe, developer of milter-spamc he's responded with:I'm going to reject this patch on the grounds that I claim the DKIM test in SpamAssassin is wrong. RFC 2822 line endings for ALL headers, body lines, and the blank line separating the two are CRLF, not LF. Since the first call to filterBody() when processing a message does NOT contain the CRLF that separates the headers from the body, that CRLF is correctly added to the buffer when filterEndHeaders() is called. Its SpamAssassin that would appear not to be correctly handling message newlines breaking on LF instead of CRLF.The part of his theory that doesn't sit right with me is that if I re-scan the email that's handed back to sendmail after milter-spamc verifies it, DKIM_VERIFIED works fine. It appears that whatever milter-spamc is handing to SpamAssassin won't DKIM_VERIFy, but that the final output of milter-spamc will DKIM_VERIFy. To me, this is an indication of munge when handing off to SpamAssassin, and a removal of that munge when handed back to sendmail. I know I've just repeated the same point three times, but I'm trying to articulate it correctly.I'd like to help debug and/or troubleshoot this issue, even though I feel my patch to milter-spamc resolves the issue for my systems.Any thoughts would be greatly appreciated. If you feel it'd be more appropriate to post to the users list, I'll do that instead, but I thought that this issue might be more dev-y.Thanks in advance.
email-canonicalization.tar.gz
Description: GNU Zip compressed data
inbound_smtp.cap
Description: Binary data
packet-captures.tar.gz
Description: GNU Zip compressed data
