Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
I keep saying I can't test qmail with telnet -- I mean to say the
command line. I can't run qmail-smtpd from the command line and give it
a message. Piping a text file through it can also fail if the file
isn't in DOS format. That's one of the first reasons I wrote the
sendrecv program that's used by the spamdyke testing scripts.
Sorry for the misstatement.
-- Sam Clippinger
Sam Clippinger wrote:
Since I've received several complaints about the . insertion when
timeouts occur, I'll remove it. It was an error on my part.
As for inserting carriage returns, I stand by what I said before. Since
RFC 2821 doesn't elaborate on the additional problems that have been
created, I can't take its vague statement at face value. RFCs are not
the law, they are documents written to describe how the internet should
work in a perfect world. As a result, many SMTP servers violate RFCs in
order to accommodate badly written clients. It's a matter of picking
your battles; trying to force everyone to send carriage returns by
rejecting their email doesn't work. It only punishes innocent users who
don't know or care about RFCs. I don't believe inserting carriage
returns causes any problems. If anyone can demonstrate that it does,
I'll remove the feature immediately.
However, if you feel strongly about it, you are free to remove it. Just
change line 339 in spamdyke.c to an else instead of an else if.
Then remove the final else block starting on line 344. That's it.
-- Sam Clippinger
PS: I can't test qmail from a telnet prompt because Linux doesn't end
its lines with CRLF, just LF. I can issue the HELO, MAIL, RCPT and DATA
commands but as soon as I try to type any message text, qmail cuts me
off with an error message.
Chris Robinson wrote:
If qmail-smtpd gets a timeout it simply issues the 451 and exits without
invoking qmail-queue. To pretend it had got a CRLF.CRLF would break
RFC 821 and 2821. Here's exactly what it does:
- qmail-sptpd.c snip --
void die_alarm() { out(451 timeout (#4.4.2)\r\n); flush(); _exit(1); }
int saferead(fd,buf,len) int fd; char *buf; int len;
{
r = timeoutread(timeout,fd,buf,len);
if (r == -1) if (errno == error_timeout) die_alarm();
}
---
This is all done from within the DATA function blast() (I'm not sure
if that's explosive or invective :-))
Incidentally RFC 2821 states:
The custom of accepting lines ending only in LF, as a concession to
non-conforming behavior on the part of some UNIX systems, has proven
to cause more interoperability problems than it solves, and SMTP
server systems MUST NOT do this...
I'm surprised you can't test qmail with telnet. I used to do it quite
regularly to check out users' complaints. Now we use SMTPAUTH it's too
much of a pain to type all that encoded username/password stuff.
Chris Robinson
CRLF.CRLF
- Original Message -
*From:* Sam Clippinger mailto:[EMAIL PROTECTED]
*To:* spamdyke users mailto:spamdyke-users@spamdyke.org
*Sent:* Friday, February 08, 2008 07:02
*Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to
qmailafter timeout
Well as I said, I assumed qmail would send the first part of the
message
if the remote server disconnected without completing it but I never
tested it. If that's not what qmail does, I'll remove it from spamdyke.
Inserting carriage returns is a different kettle of fish, however. I
personally disagree with DJB's position about strictly interpreting the
RFCs -- I believe software should strictly follow RFCs when producing
output and loosely follow them when accepting it. I think his position
on this has created more problems than it has solved. Among other
things, it really annoys me that I can't test a qmail server with a
telnet session.
There are only two situations I can imagine where inserting the
carriage
returns could possibly cause a problem. First, ESMTP allows the sender
to specify the size of the message body within the DATA command.
Inserting carriage returns may cause the message to be bigger than the
number indicates, which may cause the mail server to truncate the
message. Second, cryptographic software (e.g. PGP, GPG) work by
producing a checksum from the message content. Inserting carriage
returns may invalidate the signature. I haven't tested either of those
things.
-- Sam Clippinger
Chris Robinson wrote:
OK that explains everything.
I honestly admire spamdyke enormously; for its clean, readable and
efficient code, and for the way it turned my servers into racehorses
rather than donkeys labouring under a load average around 8 to 10. I
don't know what I'd do without it now
Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
Thanks Hartmut, I think you may be right about the sendrecv changes, specially if the child process somehow closed the connection to qmail in a way that satisfied qmail-smtpd. Could it be that the child was actually sending the SMTP terminating dot without receiving it from the departed client? Whatever. Chris - Original Message - From: Andreas To: spamdyke users Sent: Thursday, February 07, 2008 14:15 Subject: Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout Hi Hartmut, I had the same phenomen on 1 of my qmail-servers as long as I had a nameserver related problem. I had up to 750 processes of spamdyke. I had to reboot the server to get rid of those processes, just killing them was not enough, they immediately reappeared. Sind my nemserver works properly I have, on the same machine, just 10-20 processes of spamdyke and it works fine now. My spamdyke- version is 3.15. Bye Andreas Am Donnerstag, den 07.02.2008, 08:36 +0100 schrieb Hartmut Wernisch: I hade customers which got emails again and again in spamdyke version prior to 3.1.4. The maximum timeout I have been setting was 600. Whitelisting didn't help. Version 3.1.4 of spamdyke fixed this behaviour for me. Maybe because of the changes of sendrecv in release 3.1.4? On 06 Feb 08, Chris Robinson wrote: Maybe I didn't express myself properly. Spamdyke was timeing out the connection perfectly correctly because the client was inactive for more than 60 seconds which was spamdyke's default idle timeout. So the client's Outlook sees the disconnection and retries, often resulting in another partial copy of the same message. That is why I've probably fixed it by setting idle-timeout-secs=1200, which should handle even the slowest clients. But it's not the cause, it's the effect that concerns me. The partial message received before the disconnection (e.g. 50 Kb of a 200Kb email) has been piped to qmail-smtpd which then actually delivers it. How could qmail-smtpd deliver a message that has not been properly terminated? Does spamdyke simply close the pipe to qmail-smtpd, as would happen if qmail-smtpd were connected directly to the client and qmail-smtpd initiated the timeout? Somehow a fundamental principle of SMTP MTAs is being breached, namely that an incomplete session will never result in delivery, local or remote, of a partial message. My details: xinetd.d - server = /var/qmail/bin/tcp-env server_args = -Rt0 /usr/local/bin/spamdyke -f /usr/np/mail/spamdyke/spamdyke.conf /var/qmail/bin/qmail-smtpdauth /var/qmail/bin/checkpassword /bin/true Clamav and Spamassassin are only run after qmail-local pipes it to procmail via the .qmail file, so they can't be affecting the issue. Of course neither of those programs are run when the recipient is remote and qmail-remote handles it. But even remote recipients are receiving multiple, partial copies of the same message Cheers and thanks for the response, Chris Robinson - Original Message - From: Sam Clippinger [EMAIL PROTECTED] To: spamdyke users spamdyke-users@spamdyke.org Sent: Wednesday, February 06, 2008 18:11 Subject: Re: [spamdyke-users] Spamdyke passes partial emails to qmail after timeout Most likely, this is a virus/spam filter issue. Some qmail installations pass the incoming email to Spamassassin or ClamAV before acknowledging the delivery. When that happens, spamdyke's idle timeout can trigger a disconnection if the scanner takes too long (this is common for large attachments). The best way to be sure is to enable full logging (with full-log-dir) and examine the logs for disconnected deliveries. The log will contain timestamps that will show long each response was delayed. Of course, it's also possible you've found a bug. :) In that case, I'll need to find a way to reproduce it, so I may need you to (privately) send me a few full logs along with details of your spamdyke configuration. -- Sam Clippinger Chris Robinson wrote: Hi Everyone, I run mail servers for about 30 companies (qmail /spamdyke 3.1.0 / Fedora 2.4.21) and started experiencing a problem whereby some users complained that they could send an email and it was received by the recipient corrupted multiple times in varying sizes until eventually the final version would arrive correct in full. To cut a long story short I eventually tracked it down from a string in a user's Outlook log Talk faster next time. A google revealed all. It wasn't coming from qmail but from spamdyke, which explained why I couldn't grep it in the qmail source. I've probably fixed
Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
OK that explains everything. I honestly admire spamdyke enormously; for its clean, readable and efficient code, and for the way it turned my servers into racehorses rather than donkeys labouring under a load average around 8 to 10. I don't know what I'd do without it now. But I must honestly say that I think spamdyke should not interfere with the data stream once it's done it's job on the envelope and headers. The first thing in the README is the About Spamdyke which says ... It acts as a transparent middleman, observing the conversation without interference ..., and I feel it should honour this strictly. To be honest I'm even have my doubts about it silently inserting a carriage return before any bare line feed characters it sees. Maybe all this could be an option such as strict-transparency in the config file? Chris Robinson - Original Message - From: Sam Clippinger To: spamdyke users Sent: Thursday, February 07, 2008 17:14 Subject: Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout The changes to sendrecv aren't related to this -- sendrecv is a program I wrote only for testing spamdyke (as part of the testing scripts). It isn't involved in spamdyke's operation at all. The changes to sendrecv in 3.1.4 were needed because spamdyke was corrupting the SSL handshake. sendrecv contained a matching bug that covered the corruption and allowed the SSL handshake to proceed correctly. You're seeing this behavior because when spamdyke disconnects a remote client due to a timeout (either idle timeout or maximum connection time), it sends a terminating dot and a QUIT command to qmail. This was a judgement call on my part -- I felt that it was better to close qmail correctly and deliver whatever data had been sent than to potentially lose it if the client didn't retry. Not all clients correctly handle errors during/after the DATA command. I also assumed qmail already behaved this way, though I didn't test it. If that assumption is wrong and this is causing problems, I can easily change it. -- Sam Clippinger Chris Robinson wrote: Thanks Hartmut, I think you may be right about the sendrecv changes, specially if the child process somehow closed the connection to qmail in a way that satisfied qmail-smtpd. Could it be that the child was actually sending the SMTP terminating dot without receiving it from the departed client? Whatever. Chris - Original Message - *From:* Andreas mailto:[EMAIL PROTECTED] *To:* spamdyke users mailto:spamdyke-users@spamdyke.org *Sent:* Thursday, February 07, 2008 14:15 *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout Hi Hartmut, I had the same phenomen on 1 of my qmail-servers as long as I had a nameserver related problem. I had up to 750 processes of spamdyke. I had to reboot the server to get rid of those processes, just killing them was not enough, they immediately reappeared. Sind my nemserver works properly I have, on the same machine, just 10-20 processes of spamdyke and it works fine now. My spamdyke- version is 3.15. Bye Andreas Am Donnerstag, den 07.02.2008, 08:36 +0100 schrieb Hartmut Wernisch: I hade customers which got emails again and again in spamdyke version prior to 3.1.4. The maximum timeout I have been setting was 600. Whitelisting didn't help. Version 3.1.4 of spamdyke fixed this behaviour for me. Maybe because of the changes of sendrecv in release 3.1.4? On 06 Feb 08, Chris Robinson wrote: Maybe I didn't express myself properly. Spamdyke was timeing out the connection perfectly correctly because the client was inactive for more than 60 seconds which was spamdyke's default idle timeout. So the client's Outlook sees the disconnection and retries, often resulting in another partial copy of the same message. That is why I've probably fixed it by setting idle-timeout-secs=1200, which should handle even the slowest clients. But it's not the cause, it's the effect that concerns me. The partial message received before the disconnection (e.g. 50 Kb of a 200Kb email) has been piped to qmail-smtpd which then actually delivers it. How could qmail-smtpd deliver a message that has not been properly terminated? Does spamdyke simply close the pipe to qmail-smtpd, as would happen if qmail-smtpd were connected directly to the client and qmail-smtpd initiated the timeout? Somehow a fundamental principle of SMTP MTAs is being breached
Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
The changes to sendrecv aren't related to this -- sendrecv is a program I wrote only for testing spamdyke (as part of the testing scripts). It isn't involved in spamdyke's operation at all. The changes to sendrecv in 3.1.4 were needed because spamdyke was corrupting the SSL handshake. sendrecv contained a matching bug that covered the corruption and allowed the SSL handshake to proceed correctly. You're seeing this behavior because when spamdyke disconnects a remote client due to a timeout (either idle timeout or maximum connection time), it sends a terminating dot and a QUIT command to qmail. This was a judgement call on my part -- I felt that it was better to close qmail correctly and deliver whatever data had been sent than to potentially lose it if the client didn't retry. Not all clients correctly handle errors during/after the DATA command. I also assumed qmail already behaved this way, though I didn't test it. If that assumption is wrong and this is causing problems, I can easily change it. -- Sam Clippinger Chris Robinson wrote: Thanks Hartmut, I think you may be right about the sendrecv changes, specially if the child process somehow closed the connection to qmail in a way that satisfied qmail-smtpd. Could it be that the child was actually sending the SMTP terminating dot without receiving it from the departed client? Whatever. Chris - Original Message - *From:* Andreas mailto:[EMAIL PROTECTED] *To:* spamdyke users mailto:spamdyke-users@spamdyke.org *Sent:* Thursday, February 07, 2008 14:15 *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout Hi Hartmut, I had the same phenomen on 1 of my qmail-servers as long as I had a nameserver related problem. I had up to 750 processes of spamdyke. I had to reboot the server to get rid of those processes, just killing them was not enough, they immediately reappeared. Sind my nemserver works properly I have, on the same machine, just 10-20 processes of spamdyke and it works fine now. My spamdyke- version is 3.15. Bye Andreas Am Donnerstag, den 07.02.2008, 08:36 +0100 schrieb Hartmut Wernisch: I hade customers which got emails again and again in spamdyke version prior to 3.1.4. The maximum timeout I have been setting was 600. Whitelisting didn't help. Version 3.1.4 of spamdyke fixed this behaviour for me. Maybe because of the changes of sendrecv in release 3.1.4? On 06 Feb 08, Chris Robinson wrote: Maybe I didn't express myself properly. Spamdyke was timeing out the connection perfectly correctly because the client was inactive for more than 60 seconds which was spamdyke's default idle timeout. So the client's Outlook sees the disconnection and retries, often resulting in another partial copy of the same message. That is why I've probably fixed it by setting idle-timeout-secs=1200, which should handle even the slowest clients. But it's not the cause, it's the effect that concerns me. The partial message received before the disconnection (e.g. 50 Kb of a 200Kb email) has been piped to qmail-smtpd which then actually delivers it. How could qmail-smtpd deliver a message that has not been properly terminated? Does spamdyke simply close the pipe to qmail-smtpd, as would happen if qmail-smtpd were connected directly to the client and qmail-smtpd initiated the timeout? Somehow a fundamental principle of SMTP MTAs is being breached, namely that an incomplete session will never result in delivery, local or remote, of a partial message. My details: xinetd.d - server = /var/qmail/bin/tcp-env server_args = -Rt0 /usr/local/bin/spamdyke -f /usr/np/mail/spamdyke/spamdyke.conf /var/qmail/bin/qmail-smtpdauth /var/qmail/bin/checkpassword /bin/true Clamav and Spamassassin are only run after qmail-local pipes it to procmail via the .qmail file, so they can't be affecting the issue. Of course neither of those programs are run when the recipient is remote and qmail-remote handles it. But even remote recipients are receiving multiple, partial copies of the same message Cheers and thanks for the response, Chris Robinson - Original Message - From: Sam Clippinger [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] To: spamdyke users spamdyke-users@spamdyke.org mailto:spamdyke-users@spamdyke.org Sent: Wednesday, February 06, 2008 18:11 Subject: Re: [spamdyke-users] Spamdyke passes partial emails to qmail after
Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
Well as I said, I assumed qmail would send the first part of the message if the remote server disconnected without completing it but I never tested it. If that's not what qmail does, I'll remove it from spamdyke. Inserting carriage returns is a different kettle of fish, however. I personally disagree with DJB's position about strictly interpreting the RFCs -- I believe software should strictly follow RFCs when producing output and loosely follow them when accepting it. I think his position on this has created more problems than it has solved. Among other things, it really annoys me that I can't test a qmail server with a telnet session. There are only two situations I can imagine where inserting the carriage returns could possibly cause a problem. First, ESMTP allows the sender to specify the size of the message body within the DATA command. Inserting carriage returns may cause the message to be bigger than the number indicates, which may cause the mail server to truncate the message. Second, cryptographic software (e.g. PGP, GPG) work by producing a checksum from the message content. Inserting carriage returns may invalidate the signature. I haven't tested either of those things. -- Sam Clippinger Chris Robinson wrote: OK that explains everything. I honestly admire spamdyke enormously; for its clean, readable and efficient code, and for the way it turned my servers into racehorses rather than donkeys labouring under a load average around 8 to 10. I don't know what I'd do without it now. But I must honestly say that I think spamdyke should not interfere with the data stream once it's done it's job on the envelope and headers. The first thing in the README is the About Spamdyke which says ... It acts as a transparent middleman, observing the conversation without interference ..., and I feel it should honour this strictly. To be honest I'm even have my doubts about it silently inserting a carriage return before any bare line feed characters it sees. Maybe all this could be an option such as strict-transparency in the config file? Chris Robinson - Original Message - *From:* Sam Clippinger mailto:[EMAIL PROTECTED] *To:* spamdyke users mailto:spamdyke-users@spamdyke.org *Sent:* Thursday, February 07, 2008 17:14 *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout The changes to sendrecv aren't related to this -- sendrecv is a program I wrote only for testing spamdyke (as part of the testing scripts). It isn't involved in spamdyke's operation at all. The changes to sendrecv in 3.1.4 were needed because spamdyke was corrupting the SSL handshake. sendrecv contained a matching bug that covered the corruption and allowed the SSL handshake to proceed correctly. You're seeing this behavior because when spamdyke disconnects a remote client due to a timeout (either idle timeout or maximum connection time), it sends a terminating dot and a QUIT command to qmail. This was a judgement call on my part -- I felt that it was better to close qmail correctly and deliver whatever data had been sent than to potentially lose it if the client didn't retry. Not all clients correctly handle errors during/after the DATA command. I also assumed qmail already behaved this way, though I didn't test it. If that assumption is wrong and this is causing problems, I can easily change it. -- Sam Clippinger Chris Robinson wrote: Thanks Hartmut, I think you may be right about the sendrecv changes, specially if the child process somehow closed the connection to qmail in a way that satisfied qmail-smtpd. Could it be that the child was actually sending the SMTP terminating dot without receiving it from the departed client? Whatever. Chris - Original Message - *From:* Andreas mailto:[EMAIL PROTECTED] *To:* spamdyke users mailto:spamdyke-users@spamdyke.org *Sent:* Thursday, February 07, 2008 14:15 *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout Hi Hartmut, I had the same phenomen on 1 of my qmail-servers as long as I had a nameserver related problem. I had up to 750 processes of spamdyke. I had to reboot the server to get rid of those processes, just killing them was not enough, they immediately reappeared. Sind my nemserver works properly I have, on the same machine, just 10-20 processes of spamdyke and it works fine now. My spamdyke- version is 3.15. Bye Andreas Am Donnerstag, den 07.02.2008, 08:36 +0100 schrieb Hartmut Wernisch
Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter timeout
If qmail-smtpd gets a timeout it simply issues the 451 and exits without
invoking qmail-queue. To pretend it had got a CRLF.CRLF would break RFC 821
and 2821. Here's exactly what it does:
- qmail-sptpd.c snip --
void die_alarm() { out(451 timeout (#4.4.2)\r\n); flush(); _exit(1); }
int saferead(fd,buf,len) int fd; char *buf; int len;
{
r = timeoutread(timeout,fd,buf,len);
if (r == -1) if (errno == error_timeout) die_alarm();
}
---
This is all done from within the DATA function blast() (I'm not sure if
that's explosive or invective :-))
Incidentally RFC 2821 states:
The custom of accepting lines ending only in LF, as a concession to
non-conforming behavior on the part of some UNIX systems, has proven
to cause more interoperability problems than it solves, and SMTP
server systems MUST NOT do this...
I'm surprised you can't test qmail with telnet. I used to do it quite regularly
to check out users' complaints. Now we use SMTPAUTH it's too much of a pain to
type all that encoded username/password stuff.
Chris Robinson
CRLF.CRLF
- Original Message -
From: Sam Clippinger
To: spamdyke users
Sent: Friday, February 08, 2008 07:02
Subject: Re: [spamdyke-users] Spamdyke passes partial emails to qmailafter
timeout
Well as I said, I assumed qmail would send the first part of the message
if the remote server disconnected without completing it but I never
tested it. If that's not what qmail does, I'll remove it from spamdyke.
Inserting carriage returns is a different kettle of fish, however. I
personally disagree with DJB's position about strictly interpreting the
RFCs -- I believe software should strictly follow RFCs when producing
output and loosely follow them when accepting it. I think his position
on this has created more problems than it has solved. Among other
things, it really annoys me that I can't test a qmail server with a
telnet session.
There are only two situations I can imagine where inserting the carriage
returns could possibly cause a problem. First, ESMTP allows the sender
to specify the size of the message body within the DATA command.
Inserting carriage returns may cause the message to be bigger than the
number indicates, which may cause the mail server to truncate the
message. Second, cryptographic software (e.g. PGP, GPG) work by
producing a checksum from the message content. Inserting carriage
returns may invalidate the signature. I haven't tested either of those
things.
-- Sam Clippinger
Chris Robinson wrote:
OK that explains everything.
I honestly admire spamdyke enormously; for its clean, readable and
efficient code, and for the way it turned my servers into racehorses
rather than donkeys labouring under a load average around 8 to 10. I
don't know what I'd do without it now.
But I must honestly say that I think spamdyke should not interfere with
the data stream once it's done it's job on the envelope and headers. The
first thing in the README is the About Spamdyke which says ... It acts
as a transparent middleman, observing the conversation without
interference ..., and I feel it should honour this strictly. To be
honest I'm even have my doubts about it silently inserting a carriage
return before any bare line feed characters it sees.
Maybe all this could be an option such as strict-transparency in the
config file?
Chris Robinson
- Original Message -
*From:* Sam Clippinger mailto:[EMAIL PROTECTED]
*To:* spamdyke users mailto:spamdyke-users@spamdyke.org
*Sent:* Thursday, February 07, 2008 17:14
*Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to
qmailafter timeout
The changes to sendrecv aren't related to this -- sendrecv is a program
I wrote only for testing spamdyke (as part of the testing scripts). It
isn't involved in spamdyke's operation at all. The changes to sendrecv
in 3.1.4 were needed because spamdyke was corrupting the SSL handshake.
sendrecv contained a matching bug that covered the corruption and
allowed the SSL handshake to proceed correctly.
You're seeing this behavior because when spamdyke disconnects a remote
client due to a timeout (either idle timeout or maximum connection
time), it sends a terminating dot and a QUIT command to qmail. This
was a judgement call on my part -- I felt that it was better to close
qmail correctly and deliver whatever data had been sent than to
potentially lose it if the client didn't retry. Not all clients
correctly handle errors during/after the DATA command.
I also assumed qmail already behaved this way, though I didn't test it.
If that assumption is wrong and this is causing
