I keep saying I can't test qmail with telnet -- I mean to say the
command line. I can't run qmail-smtpd from the command line and give it
a message. Piping a text file through it can also fail if the file
isn't in "DOS" format. That's one of the first reasons I wrote the
"sendrecv" program that's used by the spamdyke testing scripts.
Sorry for the misstatement.
-- Sam Clippinger
Sam Clippinger wrote:
> Since I've received several complaints about the "." insertion when
> timeouts occur, I'll remove it. It was an error on my part.
>
> As for inserting carriage returns, I stand by what I said before. Since
> RFC 2821 doesn't elaborate on the additional problems that have been
> created, I can't take its vague statement at face value. RFCs are not
> the law, they are documents written to describe how the internet should
> work in a perfect world. As a result, many SMTP servers violate RFCs in
> order to accommodate badly written clients. It's a matter of picking
> your battles; trying to force everyone to send carriage returns by
> rejecting their email doesn't work. It only punishes innocent users who
> don't know or care about RFCs. I don't believe inserting carriage
> returns causes any problems. If anyone can demonstrate that it does,
> I'll remove the feature immediately.
>
> However, if you feel strongly about it, you are free to remove it. Just
> change line 339 in spamdyke.c to an "else" instead of an "else if".
> Then remove the final "else" block starting on line 344. That's it.
>
> -- Sam Clippinger
>
> PS: I can't test qmail from a telnet prompt because Linux doesn't end
> its lines with CRLF, just LF. I can issue the HELO, MAIL, RCPT and DATA
> commands but as soon as I try to type any message text, qmail cuts me
> off with an error message.
>
> Chris Robinson wrote:
>> If qmail-smtpd gets a timeout it simply issues the 451 and exits without
>> invoking qmail-queue. To pretend it had got a <CRLF>.<CRLF> would break
>> RFC 821 and 2821. Here's exactly what it does:
>>
>> --------- qmail-sptpd.c snip --------------
>>
>> void die_alarm() { out("451 timeout (#4.4.2)\r\n"); flush(); _exit(1); }
>>
>> int saferead(fd,buf,len) int fd; char *buf; int len;
>> {
>> ............
>> r = timeoutread(timeout,fd,buf,len);
>> if (r == -1) if (errno == error_timeout) die_alarm();
>> ............
>> }
>>
>> ---------------------------------------------------
>>
>> This is all done from within the DATA function "blast()" (I'm not sure
>> if that's explosive or invective :-))
>>
>> Incidentally RFC 2821 states:
>>
>> "The custom of accepting lines ending only in <LF>, as a concession to
>> non-conforming behavior on the part of some UNIX systems, has proven
>> to cause more interoperability problems than it solves, and SMTP
>> server systems MUST NOT do this..."
>>
>> I'm surprised you can't test qmail with telnet. I used to do it quite
>> regularly to check out users' complaints. Now we use SMTPAUTH it's too
>> much of a pain to type all that encoded username/password stuff.
>>
>> Chris Robinson
>> <CRLF>.<CRLF>
>>
>> ----- Original Message -----
>> *From:* Sam Clippinger <mailto:[EMAIL PROTECTED]>
>> *To:* spamdyke users <mailto:[email protected]>
>> *Sent:* Friday, February 08, 2008 07:02
>> *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to
>> qmailafter timeout
>>
>> Well as I said, I assumed qmail would send the first part of the
>> message
>> if the remote server disconnected without completing it but I never
>> tested it. If that's not what qmail does, I'll remove it from spamdyke.
>>
>> Inserting carriage returns is a different kettle of fish, however. I
>> personally disagree with DJB's position about strictly interpreting the
>> RFCs -- I believe software should strictly follow RFCs when producing
>> output and loosely follow them when accepting it. I think his position
>> on this has created more problems than it has solved. Among other
>> things, it really annoys me that I can't test a qmail server with a
>> telnet session.
>>
>> There are only two situations I can imagine where inserting the
>> carriage
>> returns could possibly cause a problem. First, ESMTP allows the sender
>> to specify the size of the message body within the DATA command.
>> Inserting carriage returns may cause the message to be bigger than the
>> number indicates, which may cause the mail server to truncate the
>> message. Second, cryptographic software (e.g. PGP, GPG) work by
>> producing a checksum from the message content. Inserting carriage
>> returns may invalidate the signature. I haven't tested either of those
>> things.
>>
>> -- Sam Clippinger
>>
>> Chris Robinson wrote:
>> > OK that explains everything.
>> >
>> > I honestly admire spamdyke enormously; for its clean, readable and
>> > efficient code, and for the way it turned my servers into racehorses
>> > rather than donkeys labouring under a load average around 8 to 10. I
>> > don't know what I'd do without it now.
>> >
>> > But I must honestly say that I think spamdyke should not
>> interfere with
>> > the data stream once it's done it's job on the envelope and
>> headers. The
>> > first thing in the README is the About Spamdyke which says ...
>> "It acts
>> > as a transparent middleman, observing the conversation without
>> > interference" ..., and I feel it should honour this strictly. To be
>> > honest I'm even have my doubts about it "silently inserting a
>> carriage
>> > return before any bare line feed characters it sees".
>> >
>> > Maybe all this could be an option such as "strict-transparency"
>> in the
>> > config file?
>> >
>> > Chris Robinson
>> >
>> > ----- Original Message -----
>> > *From:* Sam Clippinger <mailto:[EMAIL PROTECTED]>
>> > *To:* spamdyke users <mailto:[email protected]>
>> > *Sent:* Thursday, February 07, 2008 17:14
>> > *Subject:* Re: [spamdyke-users] Spamdyke passes partial emails to
>> > qmailafter timeout
>> >
>> > The changes to sendrecv aren't related to this -- sendrecv is
>> a program
>> > I wrote only for testing spamdyke (as part of the testing
>> scripts). It
>> > isn't involved in spamdyke's operation at all. The changes
>> to sendrecv
>> > in 3.1.4 were needed because spamdyke was corrupting the SSL
>> handshake.
>> > sendrecv contained a matching bug that covered the
>> corruption and
>> > allowed the SSL handshake to proceed correctly.
>> >
>> > You're seeing this behavior because when spamdyke disconnects
>> a remote
>> > client due to a timeout (either idle timeout or maximum
>> connection
>> > time), it sends a terminating dot and a "QUIT" command to
>> qmail. This
>> > was a judgement call on my part -- I felt that it was better
>> to close
>> > qmail correctly and deliver whatever data had been sent than to
>> > potentially lose it if the client didn't retry. Not all clients
>> > correctly handle errors during/after the DATA command.
>> >
>> > I also assumed qmail already behaved this way, though I
>> didn't test it.
>> > If that assumption is wrong and this is causing problems, I can
>> > easily
>> > change it.
>> >
>> > -- Sam Clippinger
>> >
>> > Chris Robinson wrote:
>> > > Thanks Hartmut,
>> > >
>> > > I think you may be right about the sendrecv changes,
>> specially if
>> > the
>> > > child process somehow closed the connection to qmail in a
>> way that
>> > > satisfied qmail-smtpd. Could it be that the child was actually
>> > sending
>> > > the SMTP terminating dot without receiving it from the
>> departed
>> > client?
>> > >
>> > > Whatever.
>> > >
>> > > Chris
>> > >
>> > > ----- Original Message -----
>> > > *From:* Andreas <mailto:[EMAIL PROTECTED]>
>> > > *To:* spamdyke users <mailto:[email protected]>
>> > > *Sent:* Thursday, February 07, 2008 14:15
>> > > *Subject:* Re: [spamdyke-users] Spamdyke passes
>> partial emails to
>> > > qmailafter timeout
>> > >
>> > > Hi Hartmut,
>> > >
>> > > I had the same phenomen on 1 of my qmail-servers as
>> long as I
>> > had a
>> > > nameserver related problem.
>> > > I had up to 750 processes of spamdyke.
>> > > I had to reboot the server to get rid of those processes,
>> > just killing
>> > > them was not enough, they immediately reappeared.
>> > > Sind my nemserver works properly I have, on the same
>> machine,
>> > just 10-20
>> > > processes of spamdyke and it works fine now.
>> > > My spamdyke- version is 3.15.
>> > > Bye
>> > >
>> > > Andreas
>> > > Am Donnerstag, den 07.02.2008, 08:36 +0100 schrieb Hartmut
>> > Wernisch:
>> > > > I hade customers which got emails again and again in
>> > spamdyke version
>> > > > prior to 3.1.4. The maximum timeout I have been setting
>> > was 600.
>> > > > Whitelisting didn't help.
>> > > >
>> > > > Version 3.1.4 of spamdyke fixed this behaviour for
>> me. Maybe
>> > > because of
>> > > > the changes of sendrecv in release 3.1.4?
>> > > >
>> > > > On 06 Feb 08, Chris Robinson wrote:
>> > > > > Maybe I didn't express myself properly. Spamdyke
>> was timeing
>> > > out the
>> > > > > connection perfectly correctly because the client was
>> > inactive
>> > > for more than
>> > > > > 60 seconds which was spamdyke's default idle
>> timeout. So the
>> > > client's
>> > > > > Outlook sees the disconnection and retries, often
>> > resulting in
>> > > another
>> > > > > partial copy of the same message. That is why
>> I've probably
>> > > fixed it by
>> > > > > setting "idle-timeout-secs=1200", which should handle
>> > even the
>> > > slowest
>> > > > > clients.
>> > > > >
>> > > > > But it's not the cause, it's the effect that concerns
>> > me. The
>> > > partial
>> > > > > message received before the disconnection (e.g.
>> 50 Kb of a
>> > > 200Kb email) has
>> > > > > been piped to qmail-smtpd which then actually
>> delivers
>> > it. How
>> > > could
>> > > > > qmail-smtpd deliver a message that has not been
>> properly
>> > > terminated? Does
>> > > > > spamdyke simply close the pipe to qmail-smtpd, as
>> would
>> > happen if
>> > > > > qmail-smtpd were connected directly to the client and
>> > > qmail-smtpd initiated
>> > > > > the timeout?
>> > > > >
>> > > > > Somehow a fundamental principle of SMTP MTAs is being
>> > breached,
>> > > namely that
>> > > > > an incomplete session will never result in delivery,
>> > local or
>> > > remote, of a
>> > > > > partial message.
>> > > > >
>> > > > > My details:
>> > > > >
>> > > > > xinetd.d
>> > > > > ---------
>> > > > > server = /var/qmail/bin/tcp-env
>> > > > > server_args = -Rt0 /usr/local/bin/spamdyke -f
>> > > > > /usr/np/mail/spamdyke/spamdyke.conf
>> > /var/qmail/bin/qmail-smtpdauth
>> > > > > /var/qmail/bin/checkpassword /bin/true
>> > > > >
>> > > > > Clamav and Spamassassin are only run after
>> qmail-local
>> > pipes it
>> > > to procmail
>> > > > > via the .qmail file, so they can't be affecting the
>> > issue. Of
>> > > course neither
>> > > > > of those programs are run when the recipient is
>> remote and
>> > > qmail-remote
>> > > > > handles it. But even remote recipients are receiving
>> > multiple,
>> > > partial
>> > > > > copies of the same message
>> > > > >
>> > > > > Cheers and thanks for the response,
>> > > > > Chris Robinson
>> > > > >
>> > > > > ----- Original Message -----
>> > > > > From: "Sam Clippinger" <[EMAIL PROTECTED]
>> <mailto:[EMAIL PROTECTED]>
>> > <mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]>>
>> > > > > To: "spamdyke users" <[email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > <mailto:[email protected]>>
>> > > > > Sent: Wednesday, February 06, 2008 18:11
>> > > > > Subject: Re: [spamdyke-users] Spamdyke passes partial
>> > emails to
>> > > qmail after
>> > > > > timeout
>> > > > >
>> > > > >
>> > > > > > Most likely, this is a virus/spam filter
>> issue. Some
>> > qmail
>> > > > > > installations pass the incoming email to
>> Spamassassin or
>> > > ClamAV before
>> > > > > > acknowledging the delivery. When that happens,
>> spamdyke's
>> > > idle timeout
>> > > > > > can trigger a disconnection if the scanner
>> takes too long
>> > > (this is
>> > > > > > common for large attachments).
>> > > > > >
>> > > > > > The best way to be sure is to enable full
>> logging (with
>> > > "full-log-dir")
>> > > > > > and examine the logs for disconnected
>> deliveries. The log
>> > > will contain
>> > > > > > timestamps that will show long each response
>> was delayed.
>> > > > > >
>> > > > > > Of course, it's also possible you've found a
>> bug. :)
>> > In that
>> > > case, I'll
>> > > > > > need to find a way to reproduce it, so I may
>> need you to
>> > > (privately)
>> > > > > > send me a few full logs along with details of your
>> > spamdyke
>> > > configuration.
>> > > > > >
>> > > > > > -- Sam Clippinger
>> > > > > >
>> > > > > > Chris Robinson wrote:
>> > > > > > > Hi Everyone,
>> > > > > > >
>> > > > > > > I run mail servers for about 30 companies (qmail
>> > /spamdyke
>> > > 3.1.0 /
>> > > > > Fedora
>> > > > > > > 2.4.21) and started experiencing a problem
>> whereby some
>> > > users complained
>> > > > > > > that they could send an email and it was
>> received by the
>> > > recipient
>> > > > > corrupted
>> > > > > > > multiple times in varying sizes until
>> eventually the
>> > final
>> > > version would
>> > > > > > > arrive correct in full. To cut a long story
>> short I
>> > > eventually tracked
>> > > > > it
>> > > > > > > down from a string in a user's Outlook log
>> "Talk faster
>> > > next time". A
>> > > > > google
>> > > > > > > revealed all. It wasn't coming from qmail but
>> from
>> > > spamdyke, which
>> > > > > explained
>> > > > > > > why I couldn't grep it in the qmail source.
>> > > > > > >
>> > > > > > > I've probably fixed it by setting
>> > "idle-timeout-secs=1200".
>> > > But what
>> > > > > worries
>> > > > > > > me is why the recipient got anything except the
>> > final good
>> > > email. If
>> > > > > > > spamdyke issues that 421 error then breaks the
>> > connection,
>> > > the user's
>> > > > > > > Outlook can justifiably assume that the email
>> won't have
>> > > been accepted
>> > > > > by
>> > > > > > > the server, nor sent to the recipient. But what
>> > seems to
>> > > happen is that
>> > > > > the
>> > > > > > > part that has been received down the pipe so
>> far has
>> > been
>> > > passed by
>> > > > > spamdyke
>> > > > > > > to qmail-smtpd which has passed it to
>> qmail-queue and
>> > > thence to the
>> > > > > local or
>> > > > > > > remote recipient. Surely no "250 OK 123 qp
>> 456" has come
>> > > out of qmail?
>> > > > > > >
>> > > > > > > Is this a spamdyke issue or a qmail issue?
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > > Chris Robinson
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > _______________________________________________
>> > > > > > > spamdyke-users mailing list
>> > > > > > > [email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > <mailto:[email protected]>
>> > > > > > >
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > > > > > _______________________________________________
>> > > > > > spamdyke-users mailing list
>> > > > > > [email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > > > >
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > > > > >
>> > > > >
>> > > > > _______________________________________________
>> > > > > spamdyke-users mailing list
>> > > > > [email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > > >
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > > > >
>> > > > _______________________________________________
>> > > > spamdyke-users mailing list
>> > > > [email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > > http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > > >
>> > >
>> > > _______________________________________________
>> > > spamdyke-users mailing list
>> > > [email protected]
>> <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > <mailto:[email protected]>
>> > > http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > >
>> > >
>> > >
>> >
>> ------------------------------------------------------------------------
>> > >
>> > > _______________________________________________
>> > > spamdyke-users mailing list
>> > > [email protected]
>> <mailto:[email protected]>
>> <mailto:[email protected]>
>> > > http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> > _______________________________________________
>> > spamdyke-users mailing list
>> > [email protected]
>> <mailto:[email protected]>
>> <mailto:[email protected]>
>> > http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> >
>> >
>> >
>> ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > spamdyke-users mailing list
>> > [email protected] <mailto:[email protected]>
>> > http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> _______________________________________________
>> spamdyke-users mailing list
>> [email protected] <mailto:[email protected]>
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> spamdyke-users mailing list
>> [email protected]
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
> _______________________________________________
> spamdyke-users mailing list
> [email protected]
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
_______________________________________________
spamdyke-users mailing list
[email protected]
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
