Re: [exim] spool format error (on some list messages)
Hi, On 07.06.2018 23:28, exim-users--- via Exim-users wrote: > Exim mainlog show the corrupted message as second mail sent over one TCP > connection (linux kernel mailing list server is the only server that > sends more than one mail per TCP connection, other servers do not send > those volumes). I do not follow all messages on the list, thus there may > be other errors/corruptions (the queue error I had initially are the > most obvious, other corruption which do not lead to technical errors). > Quick grep in the Mail dir shows significant number of messages which > seem to have some unexpected strings in the header. I see corruption in > this specific header for other messages as well, all have in common that > there was one than one message sent over one single TCP connection. > > I am setting a debug header containing $primary_hostname in an acl > stanza to see if there is some corruption in this heades as well. The corruption only happens on the sa-exim included header, the header inserted for debugging is inserted properly. Best regards, Thomas -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
On 06/06/2018 09:00 PM, Heiko Schlittermann via Exim-users wrote: > @Jeremy: Maybe we should announce that sa_exim will have > some end-of-life in the near future? I'm happy to add to the docs chapter that discusses the spool file formats that they are specifically regarded as not being a stable interface and liable to change from time to time. Along with the obvious implication that we may break any program (such as sa_exim) that tries to use them as such. The supported interface to SA is described in http://exim.org/exim-html-current/doc/html/spec_html/ch-content_scanning_at_acl_time.html -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
exim-users--- via Exim-users (Do 31 Mai 2018 21:52:51
CEST):
..
>
> >> 1fOL7J-0001BL-DC-H
> > …
> >> 031 X-Spam-Relay-Country: US US **
> >> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format
> >> description of
> >> NRCPUS header
> >> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
> >> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
> > [ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
> > this as the end of the header section of the message.
> >> 042 Acked-by: Andi Kleen
> >> 044 Cc: Adrian Hunter
> >> 036 Cc: David Ahern
> >> 034 Cc: He Kuang
> >> 053 Cc: Hendrik Brueckner
> >> 038 Cc: Jin Yao
> > …
>
> You are right, the X-SA-Exim-Scanned header is truncated (after "on", I
> missed that before) it is set by sa-exim (code snipplet from sa-exim.c
> with line numbers):
Ah, that *I* didn't see, that there's a fragment of the header to be
added. Hm. The 's-' is part of the primary hostname?
> --cut
> 31 /* Exim includes */
> 32 #include "local_scan.h"
> 33 extern FILE *smtp_out; /* Exim's incoming SMTP
> output file */
> 34 extern int body_linecount; /* Line count in body */
> 35 extern uschar *primary_hostname;
> ...
> 1277 header_add(' ', "X-SA-Exim-Scanned: Yes (on %s)\n",
> primary_hostname);
> --cut
Ok, the spool wire format is off, you said. I'm not sure about the
mechanigs of sa_exim, that is, I do not have any clue *which* file it
sees and modifies. And/or if we built some optimisations which assume
that the spooled files (spooled in $spooldir/scan) are not altered.
For better theories about what's going on we need to know which files
sa_exim accesses.
If this is important and worth to be solved,, it would need some further
investigation.
@Jeremy: Maybe we should announce that sa_exim will have
some end-of-life in the near future?
> All corrupted messages at least lack "primary-hostname" and the newline,
> some have other parts of the message in there. Any simple way to use a
> saved message to produce some more debugging information?
You can try to use something like
swaks --data ./saved-message -f … -t … --pipe 'exim -bh 1.1.1.1'
I'm not sure, if exim stops processing right before or right after the
local_scan() call.
As you do not want to test the ACL, exim -N could be your friend.
> achieve the sa-exim functionality (on the fly spamassassin scanning and
> greylisting depending on spamassassin scores)? Spamassassin integration
> via exiscan and greylisting as described in
> https://github.com/Exim/exim/wiki/SimpleGreylisting or greylistd? Any
> best practice on this topic? What I liked on sa-exim is, that there is
> no initial greylisting for unknown senders/hosts when they send mails
> with reasonable low spamassasin scores.
I do greylisting based on the announced content size. But your approach
might be good too.
I wrote some Perl function(s) to support greylisting in Exim, these
functions work reliable for years already. Tell me, if you're
interested, I"d update the docs and the scripts a bit and publish it.
(To be true, it is published already, but the docs are outdated.)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 -
signature.asc
Description: PGP signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
Hello Heiko,
thanks for your fast analysis.
On 31.05.2018 17:38, Heiko Schlittermann via Exim-users wrote:
> I'm not sure how the sa_exim processing works, I do not use it for long
> time now. Does it see the original spooled message and modifies it?
> After this step, Exim does its own processessing, splitting the message
> into -H and -D?
Yes, original message is analyzed via spamd and altered (headers added)
before further exim processing. Depeding on spamassassin score messages
are temporarily rejected (greylisting).
> I'd see sa_exim as the suspicious. Maybe bad cooperation between sa_exim
> and Exim when we use wire format spool files (do we?) or when the
> message arrives in chunks. I think, we had some other issues in this
> context.
Wire format spool files is currently not enabled.
> For verification, can you add to some ACL the
>
> warnsenders = linux-kernel@XXX
> control = no_mbox_unspool
>
> directive? This way the message should stay in the $spooldir/scan
> folder, even after scanning. (I'm not sure if this is the way sa_exim
> works, it is just guesswork and it could help to identify the issue.)
Added it, did not disable sa-exim yet to get some more examples for the
issue. Saved messages do not have the X-SA-Exim headers.
>> 1fOL7J-0001BL-DC-H
> …
>> 031 X-Spam-Relay-Country: US US **
>> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description
>> of
>> NRCPUS header
>> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
>> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
> [ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
> this as the end of the header section of the message.
>> 042 Acked-by: Andi Kleen
>> 044 Cc: Adrian Hunter
>> 036 Cc: David Ahern
>> 034 Cc: He Kuang
>> 053 Cc: Hendrik Brueckner
>> 038 Cc: Jin Yao
> …
You are right, the X-SA-Exim-Scanned header is truncated (after "on", I
missed that before) it is set by sa-exim (code snipplet from sa-exim.c
with line numbers):
--cut
31 /* Exim includes */
32 #include "local_scan.h"
33 extern FILE *smtp_out; /* Exim's incoming SMTP
output file */
34 extern int body_linecount; /* Line count in body */
35 extern uschar *primary_hostname;
...
1277 header_add(' ', "X-SA-Exim-Scanned: Yes (on %s)\n",
primary_hostname);
--cut
All correct messages have a header:
X-SA-Exim-Scanned: Yes (on primary-hostname)
All corrupted messages at least lack "primary-hostname" and the newline,
some have other parts of the message in there. Any simple way to use a
saved message to produce some more debugging information?
However, as sa-exim is kind of unmaintained (it served my needs very
well for >10 years now, though). What would be a similar alternative to
achieve the sa-exim functionality (on the fly spamassassin scanning and
greylisting depending on spamassassin scores)? Spamassassin integration
via exiscan and greylisting as described in
https://github.com/Exim/exim/wiki/SimpleGreylisting or greylistd? Any
best practice on this topic? What I liked on sa-exim is, that there is
no initial greylisting for unknown senders/hosts when they send mails
with reasonable low spamassasin scores.
Best regards,
Thomas
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] spool format error (on some list messages)
Hi,
it looks as if the last SA-Exim header eliminated the blank line that
separates header and body.
I'm not sure how the sa_exim processing works, I do not use it for long
time now. Does it see the original spooled message and modifies it?
After this step, Exim does its own processessing, splitting the message
into -H and -D?
I'd see sa_exim as the suspicious. Maybe bad cooperation between sa_exim
and Exim when we use wire format spool files (do we?) or when the
message arrives in chunks. I think, we had some other issues in this
context.
For verification, can you add to some ACL the
warnsenders = linux-kernel@XXX
control = no_mbox_unspool
directive? This way the message should stay in the $spooldir/scan
folder, even after scanning. (I'm not sure if this is the way sa_exim
works, it is just guesswork and it could help to identify the issue.)
And as another step, can you disable sa_exim for the linux-kernel mails
(or for all mails alltogether), so we can identify a little bit more
about the issue?
> 1fOL7J-0001BL-DC-H
…
> 031 X-Spam-Relay-Country: US US **
> 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description
> of
> NRCPUS header
> 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +)
> 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter
[ HERE THE BLANK LINE WAS EATEN, so Exim doesn't recognize
this as the end of the header section of the message.
> 042 Acked-by: Andi Kleen
> 044 Cc: Adrian Hunter
> 036 Cc: David Ahern
> 034 Cc: He Kuang
> 053 Cc: Hendrik Brueckner
> 038 Cc: Jin Yao
…
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 -
signature.asc
Description: PGP signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
