Re: [spamdyke-users] timeout results in duplicates
I think I understand. You're using simscan to run ClamAV and SpamAssassin during the delivery process, so the message will be rejected immediately if either of those tests fails. If one of the tests takes too long, spamdyke times out. Unfortunately I still don't see how to do anything about this. The scanner must be being started after the mail client finishes sending the message data and before qmail responds with 250 to indicate it was accepted. From spamdyke's point of view, there's no way it can tell why the data stopped flowing or why there's been no response from qmail -- it could be due to a slow scanner, a failing hard drive, high load, low memory, a blocked filesystem call or a million other reasons. There's just no way to determine if the delay will end soon or ever -- if something is deadlocked it's possible there will never be any response. Linux systems can get into exactly that situation when an NFS server disappears, for example. I would like to eventually add hooks to spamdyke to call external filters like SpamAssassin, so it could trigger these scans itself instead of using simscan to do it. If that were done, spamdyke be able to close/kill SpamAssassin if it took too long, then continue accepting the message. Until that day however, I think the only solution is to increase your idle timeout setting. -- Sam Clippinger On Feb 8, 2014, at 4:34 PM, Eric Shubert e...@shubes.net wrote: On 02/08/2014 02:40 PM, Sam Clippinger wrote: I'm a little unclear here -- what scanning are you doing and when does it take place? I'm not crystal clear either about exactly how everything's happening. Simscan is invoking clamav and spamassassin. Simscan is implemented via QMAILQUEUE=/var/qmail/bin/simscan. How can spamdyke tell the difference between a delay caused by something on your server versus a delay from the remote sender? I've no idea. I'm guessing that simscan isn't given control until the message is completely received. It's at that point, when the message has been completely received but not yet queued, that I think the idle timeout should be disabled. The problem appears to be that when when spamdyke does idle timeout, the qmail-queue process can still successfully deliver a message (when it's past the point described above). spamdyke should only initiate a timeout when it can (still) keep a message from being delivered. Here's a sample from the log which might make things a little clearer: 02-07 14:15:14 tcpserver: status: 1/100 02-07 14:15:14 tcpserver: pid 19001 from 70.58.xxx169 02-07 14:15:14 tcpserver: ok 19001 tacs-mail.datamatters.us:192.168.73.7:25 :70.58.xxx.169::44872 02-07 14:15:23 CHKUSER accepted sender: from x...@x.com:: remote ..com:unknown:70.58.xxx.169 rcpt : sender accepted 02-07 14:15:23 CHKUSER accepted any rcpt: from x...@.com:: remote ..com:unknown:70.58.xxx.169 rcpt x...@.com : accepted any recipient for this domain 02-07 14:15:23 policy_check: remote x...@.com - local x...@.com (UNAUTHENTICATED SENDER) 02-07 14:15:23 policy_check: policy allows transmission 02-07 14:16:25 spamdyke[19001]: TIMEOUT from: x...@.com to: x...@.com origin_ip: 70.58.xxx.169 origin_rdns: .com auth: (unknown) encryption: TLS reason: TIMEOUT 02-07 14:17:58 simscan:[19002]:CLEAN (5.50/12.00):154.6404s:***SPAM*** Fwd_ 70.58.xxx.169:x...@.com:x...@.com 02-07 14:17:58 tcpserver: end 19001 status 0 Usually, the simscan message comes before spamdyke. BL is that the message is delivered, and the sender is notified of a failure, causing duplicates in the inbox. Thanks Sam. Gotta run. -- Sam Clippinger On Feb 7, 2014, at 4:37 PM, Eric Shubert e...@shubes.net mailto:e...@shubes.net wrote: With spamdyke 4.3.1, I've come across an email which takes an inordinate amount of time to scan, for whatever reason. I had idle-timeout=60, so spamdyke would timeout the session, and a minute or so later the scan completes, and the message is delivered. This causes duplicates though, as the sender isn't aware of the successful delivery. I've bumped up the idle-timeout to 180, which I expect will remedy the situation. I wonder, though, if this setting could or should be suspended during the time which spamdyke is waiting for delivery to happen. Perhaps there should be 2 settings - one for the incoming side and one for the delivery side? I like keeping this setting on the low side to keep senders from tying up incoming processes, yet the setting doesn't seem to make any sense when waiting for scanning/delivery, especially when spamdyke can't cancel that part of things. Thanks Sam. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org mailto:spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] timeout results in duplicates
On 02/08/2014 02:40 PM, Sam Clippinger wrote: I'm a little unclear here -- what scanning are you doing and when does it take place? I'm not crystal clear either about exactly how everything's happening. Simscan is invoking clamav and spamassassin. Simscan is implemented via QMAILQUEUE=/var/qmail/bin/simscan. How can spamdyke tell the difference between a delay caused by something on your server versus a delay from the remote sender? I've no idea. I'm guessing that simscan isn't given control until the message is completely received. It's at that point, when the message has been completely received but not yet queued, that I think the idle timeout should be disabled. The problem appears to be that when when spamdyke does idle timeout, the qmail-queue process can still successfully deliver a message (when it's past the point described above). spamdyke should only initiate a timeout when it can (still) keep a message from being delivered. Here's a sample from the log which might make things a little clearer: 02-07 14:15:14 tcpserver: status: 1/100 02-07 14:15:14 tcpserver: pid 19001 from 70.58.xxx169 02-07 14:15:14 tcpserver: ok 19001 tacs-mail.datamatters.us:192.168.73.7:25 :70.58.xxx.169::44872 02-07 14:15:23 CHKUSER accepted sender: from x...@x.com:: remote ..com:unknown:70.58.xxx.169 rcpt : sender accepted 02-07 14:15:23 CHKUSER accepted any rcpt: from x...@.com:: remote ..com:unknown:70.58.xxx.169 rcpt x...@.com : accepted any recipient for this domain 02-07 14:15:23 policy_check: remote x...@.com - local x...@.com (UNAUTHENTICATED SENDER) 02-07 14:15:23 policy_check: policy allows transmission 02-07 14:16:25 spamdyke[19001]: TIMEOUT from: x...@.com to: x...@.com origin_ip: 70.58.xxx.169 origin_rdns: .com auth: (unknown) encryption: TLS reason: TIMEOUT 02-07 14:17:58 simscan:[19002]:CLEAN (5.50/12.00):154.6404s:***SPAM*** Fwd_ 70.58.xxx.169:x...@.com:x...@.com 02-07 14:17:58 tcpserver: end 19001 status 0 Usually, the simscan message comes before spamdyke. BL is that the message is delivered, and the sender is notified of a failure, causing duplicates in the inbox. Thanks Sam. Gotta run. -- Sam Clippinger On Feb 7, 2014, at 4:37 PM, Eric Shubert e...@shubes.net mailto:e...@shubes.net wrote: With spamdyke 4.3.1, I've come across an email which takes an inordinate amount of time to scan, for whatever reason. I had idle-timeout=60, so spamdyke would timeout the session, and a minute or so later the scan completes, and the message is delivered. This causes duplicates though, as the sender isn't aware of the successful delivery. I've bumped up the idle-timeout to 180, which I expect will remedy the situation. I wonder, though, if this setting could or should be suspended during the time which spamdyke is waiting for delivery to happen. Perhaps there should be 2 settings - one for the incoming side and one for the delivery side? I like keeping this setting on the low side to keep senders from tying up incoming processes, yet the setting doesn't seem to make any sense when waiting for scanning/delivery, especially when spamdyke can't cancel that part of things. Thanks Sam. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org mailto:spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] timeout results in duplicates
With spamdyke 4.3.1, I've come across an email which takes an inordinate amount of time to scan, for whatever reason. I had idle-timeout=60, so spamdyke would timeout the session, and a minute or so later the scan completes, and the message is delivered. This causes duplicates though, as the sender isn't aware of the successful delivery. I've bumped up the idle-timeout to 180, which I expect will remedy the situation. I wonder, though, if this setting could or should be suspended during the time which spamdyke is waiting for delivery to happen. Perhaps there should be 2 settings - one for the incoming side and one for the delivery side? I like keeping this setting on the low side to keep senders from tying up incoming processes, yet the setting doesn't seem to make any sense when waiting for scanning/delivery, especially when spamdyke can't cancel that part of things. Thanks Sam. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
