Re: [exim] spool format error (on some list messages)
Hi, On 07.06.2018 23:28, exim-users--- via Exim-users wrote: > Exim mainlog show the corrupted message as second mail sent over one TCP > connection (linux kernel mailing list server is the only server that > sends more than one mail per TCP connection, other servers do not send > those volumes). I do not follow all messages on the list, thus there may > be other errors/corruptions (the queue error I had initially are the > most obvious, other corruption which do not lead to technical errors). > Quick grep in the Mail dir shows significant number of messages which > seem to have some unexpected strings in the header. I see corruption in > this specific header for other messages as well, all have in common that > there was one than one message sent over one single TCP connection. > > I am setting a debug header containing $primary_hostname in an acl > stanza to see if there is some corruption in this heades as well. The corruption only happens on the sa-exim included header, the header inserted for debugging is inserted properly. Best regards, Thomas -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
On 06/06/2018 09:00 PM, Heiko Schlittermann via Exim-users wrote: > @Jeremy: Maybe we should announce that sa_exim will have > some end-of-life in the near future? I'm happy to add to the docs chapter that discusses the spool file formats that they are specifically regarded as not being a stable interface and liable to change from time to time. Along with the obvious implication that we may break any program (such as sa_exim) that tries to use them as such. The supported interface to SA is described in http://exim.org/exim-html-current/doc/html/spec_html/ch-content_scanning_at_acl_time.html -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
exim-users--- via Exim-users (Do 31 Mai 2018 21:52:51
CEST):
..
>
> >> 1fOL7J-0001BL-DC-H
> > …
> >> 031 X-Spam-Relay-Country: US US **
> >> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format
> >> description of
> >> NRCPUS header
> >> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
> >> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
> > [ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
> > this as the end of the header section of the message.
> >> 042 Acked-by: Andi Kleen
> >> 044 Cc: Adrian Hunter
> >> 036 Cc: David Ahern
> >> 034 Cc: He Kuang
> >> 053 Cc: Hendrik Brueckner
> >> 038 Cc: Jin Yao
> > …
>
> You are right, the X-SA-Exim-Scanned header is truncated (after "on", I
> missed that before) it is set by sa-exim (code snipplet from sa-exim.c
> with line numbers):
Ah, that *I* didn't see, that there's a fragment of the header to be
added. Hm. The 's-' is part of the primary hostname?
> --cut
> 31 /* Exim includes */
> 32 #include "local_scan.h"
> 33 extern FILE *smtp_out; /* Exim's incoming SMTP
> output file */
> 34 extern int body_linecount; /* Line count in body */
> 35 extern uschar *primary_hostname;
> ...
> 1277 header_add(' ', "X-SA-Exim-Scanned: Yes (on %s)\n",
> primary_hostname);
> --cut
Ok, the spool wire format is off, you said. I'm not sure about the
mechanigs of sa_exim, that is, I do not have any clue *which* file it
sees and modifies. And/or if we built some optimisations which assume
that the spooled files (spooled in $spooldir/scan) are not altered.
For better theories about what's going on we need to know which files
sa_exim accesses.
If this is important and worth to be solved,, it would need some further
investigation.
@Jeremy: Maybe we should announce that sa_exim will have
some end-of-life in the near future?
> All corrupted messages at least lack "primary-hostname" and the newline,
> some have other parts of the message in there. Any simple way to use a
> saved message to produce some more debugging information?
You can try to use something like
swaks --data ./saved-message -f … -t … --pipe 'exim -bh 1.1.1.1'
I'm not sure, if exim stops processing right before or right after the
local_scan() call.
As you do not want to test the ACL, exim -N could be your friend.
> achieve the sa-exim functionality (on the fly spamassassin scanning and
> greylisting depending on spamassassin scores)? Spamassassin integration
> via exiscan and greylisting as described in
> https://github.com/Exim/exim/wiki/SimpleGreylisting or greylistd? Any
> best practice on this topic? What I liked on sa-exim is, that there is
> no initial greylisting for unknown senders/hosts when they send mails
> with reasonable low spamassasin scores.
I do greylisting based on the announced content size. But your approach
might be good too.
I wrote some Perl function(s) to support greylisting in Exim, these
functions work reliable for years already. Tell me, if you're
interested, I"d update the docs and the scripts a bit and publish it.
(To be true, it is published already, but the docs are outdated.)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 -
signature.asc
Description: PGP signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
Hello Heiko,
thanks for your fast analysis.
On 31.05.2018 17:38, Heiko Schlittermann via Exim-users wrote:
> I'm not sure how the sa_exim processing works, I do not use it for long
> time now. Does it see the original spooled message and modifies it?
> After this step, Exim does its own processessing, splitting the message
> into -H and -D?
Yes, original message is analyzed via spamd and altered (headers added)
before further exim processing. Depeding on spamassassin score messages
are temporarily rejected (greylisting).
> I'd see sa_exim as the suspicious. Maybe bad cooperation between sa_exim
> and Exim when we use wire format spool files (do we?) or when the
> message arrives in chunks. I think, we had some other issues in this
> context.
Wire format spool files is currently not enabled.
> For verification, can you add to some ACL the
>
> warnsenders = linux-kernel@XXX
> control = no_mbox_unspool
>
> directive? This way the message should stay in the $spooldir/scan
> folder, even after scanning. (I'm not sure if this is the way sa_exim
> works, it is just guesswork and it could help to identify the issue.)
Added it, did not disable sa-exim yet to get some more examples for the
issue. Saved messages do not have the X-SA-Exim headers.
>> 1fOL7J-0001BL-DC-H
> …
>> 031 X-Spam-Relay-Country: US US **
>> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description
>> of
>> NRCPUS header
>> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
>> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
> [ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
> this as the end of the header section of the message.
>> 042 Acked-by: Andi Kleen
>> 044 Cc: Adrian Hunter
>> 036 Cc: David Ahern
>> 034 Cc: He Kuang
>> 053 Cc: Hendrik Brueckner
>> 038 Cc: Jin Yao
> …
You are right, the X-SA-Exim-Scanned header is truncated (after "on", I
missed that before) it is set by sa-exim (code snipplet from sa-exim.c
with line numbers):
--cut
31 /* Exim includes */
32 #include "local_scan.h"
33 extern FILE *smtp_out; /* Exim's incoming SMTP
output file */
34 extern int body_linecount; /* Line count in body */
35 extern uschar *primary_hostname;
...
1277 header_add(' ', "X-SA-Exim-Scanned: Yes (on %s)\n",
primary_hostname);
--cut
All correct messages have a header:
X-SA-Exim-Scanned: Yes (on primary-hostname)
All corrupted messages at least lack "primary-hostname" and the newline,
some have other parts of the message in there. Any simple way to use a
saved message to produce some more debugging information?
However, as sa-exim is kind of unmaintained (it served my needs very
well for >10 years now, though). What would be a similar alternative to
achieve the sa-exim functionality (on the fly spamassassin scanning and
greylisting depending on spamassassin scores)? Spamassassin integration
via exiscan and greylisting as described in
https://github.com/Exim/exim/wiki/SimpleGreylisting or greylistd? Any
best practice on this topic? What I liked on sa-exim is, that there is
no initial greylisting for unknown senders/hosts when they send mails
with reasonable low spamassasin scores.
Best regards,
Thomas
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
Hi,
it looks as if the last SA-Exim header eliminated the blank line that
separates header and body.
I'm not sure how the sa_exim processing works, I do not use it for long
time now. Does it see the original spooled message and modifies it?
After this step, Exim does its own processessing, splitting the message
into -H and -D?
I'd see sa_exim as the suspicious. Maybe bad cooperation between sa_exim
and Exim when we use wire format spool files (do we?) or when the
message arrives in chunks. I think, we had some other issues in this
context.
For verification, can you add to some ACL the
warnsenders = linux-kernel@XXX
control = no_mbox_unspool
directive? This way the message should stay in the $spooldir/scan
folder, even after scanning. (I'm not sure if this is the way sa_exim
works, it is just guesswork and it could help to identify the issue.)
And as another step, can you disable sa_exim for the linux-kernel mails
(or for all mails alltogether), so we can identify a little bit more
about the issue?
> 1fOL7J-0001BL-DC-H
…
> 031 X-Spam-Relay-Country: US US **
> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description
> of
> NRCPUS header
> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
[ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
this as the end of the header section of the message.
> 042 Acked-by: Andi Kleen
> 044 Cc: Adrian Hunter
> 036 Cc: David Ahern
> 034 Cc: He Kuang
> 053 Cc: Hendrik Brueckner
> 038 Cc: Jin Yao
…
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 -
signature.asc
Description: PGP signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
[exim] spool format error (on some list messages)
Hello, as this gets quite long a short overiew in advance: After upgrading to exim 4.90_1 (Ubuntu 18.04), I observe message corruption in the queue in rare cases. All corrupted messages are from linux kernel mailing list. I can eliminate file system as reason for corruption. Whole story: Since updating exim to version 4.90_1 (Ubuntu upgrade to 18.04), in rare cases (less than 1 per 1 messages deliveries) messages get stuck in the queue (with message "spool format error: size=6045" when I list the queue). It seems, that files in /var/spool/exim4/input get corrupted (header and body parts get mixed up). I can eliminate filesystems corruption as root cause (no other indication for file corruption, filesystem checks w/o errors). So far I only observe corruption on linux kernel mailing list posts (example is attached), all are patch messages (mailers are git-send-email in various versions and tip-git-log-daemon). --cut exim mainlog excerpt: 2018-05-31 12:47:37.521 [4547] 1fOL7J-0001BL-DC SA: Debug: SAEximRunCond expand returned: '1' 2018-05-31 12:47:37.522 [4547] 1fOL7J-0001BL-DC SA: Debug: check succeeded, running spamc 2018-05-31 12:47:39.330 [4547] 1fOL7J-0001BL-DC SA: Action: scanned but message isn't spam: score=-15.2 required=5.0 (scanned in 2/2 secs | Message-Id: tip-j7o92acm2vnxjv70y4o3s...@git.kernel.org). From (host=vger.kernel.org [209.132.180.67]) for linux-kernel@XXX 2018-05-31 12:47:39.368 [4547] 1fOL7J-0001BL-DC <= linux-kernel-ow...@vger.kernel.org H=vger.kernel.org [209.132.180.67]:37639 I=[1.2.3.4]:25 P=esmtp K S=7580 M8S=8 id=tip-j7o92acm2vnxjv70y4o3s...@git.kernel.org from for XXX 2018-05-31 12:47:39.381 [4612] 1fOL7J-0001BL-DC Format error in spool file 1fOL7J-0001BL-DC-H: size=6763 --cut --cut mail.log excerpt: May 31 12:47:37 deep-thought spamd[4493]: spamd: processing message for Debian-exim:116 May 31 12:47:39 deep-thought spamd[4493]: spamd: result: . -15 - BAYES_00,GREYLIST_ISWHITE,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI scantime=1.8,size=5135,user=Debian-exim,uid=116,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=36468,mid=,bayes=0.00,autolearn=ham autolearn_force=no --cut I run the Ubuntus exim4-daemon-heavy flavor with sa_exim for greylisting/spamassassin integration and ClamAV for basic malware scanning. SPF checking is enabled. All corrupted messages were accepted without errors or warnings in exim and other (ClamAV, spamassassin, ...) logs. No external message altering in /var/spool/exim4/input (where corrupted messages get stuck). --cut "exim4 --version" Exim version 4.90_1 #2 built 14-Feb-2018 16:01:14 Copyright (c) University of Cambridge, 1995 - 2017 (c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2017 Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013) Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY SOCKS TCP_Fast_Open Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql sqlite Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa tls Routers: accept dnslookup ipliteral iplookup manualroute queryprogram redirect Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp Fixed never_users: 0 Configure owner: 0:0 Size of off_t: 8 Configuration file is /var/lib/exim4/config.autogenerated --cut Any hints comments highly appreciated, I can provide more examples or details if necessary. Best regards, Thomas 1fOL7J-0001BL-DC-D Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b Gitweb: https://git.kernel.org/tip/18a7057420f8b67f15d17087bf5c0863db752c8b Author: Arnaldo Carvalho de Melo AuthorDate: Fri, 25 May 2018 16:37:36 -0300 Committer: Arnaldo Carvalho de Melo CommitDate: Wed, 30 May 2018 15:40:26 -0300 perf tools: Fix perf.data format description of NRCPUS header In the perf.data HEADER_CPUDESC feadure header we store first the number of available CPUs in the system, then the number of CPUs at the time of writing the header, not the other way around. Reported-by: Thomas-Mich Richter Acked-by: Andi Kleen Cc: Adrian Hunter Cc: David Ahern Cc: He Kuang Cc: Hendrik Brueckner Cc: Jin Yao Cc: Jiri Olsa Cc: Kim Phillips Cc: Lakshman Annadorai Cc: Namhyung Kim Cc: Simon Que Cc: Stephane Eranian Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-j7o92acm2vnxjv70y4o3s...@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo --- tools/perf/Documentation/perf.data-file-format.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt index c57904a526ce..dfb218feaad9 100644 --- a/tools/perf/Documentation/perf.data-file-format.txt +++ b/tools/perf/Documentation/perf.data-file-format.txt @@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m) A structure
