Re: [spamdyke-users] timeout results in duplicates

[Sam Clippinger]

I think I understand.  You're using simscan to run ClamAV and SpamAssassin 
during the delivery process, so the message will be rejected immediately if 
either of those tests fails.  If one of the tests takes too long, spamdyke 
times out.

Unfortunately I still don't see how to do anything about this.  The scanner 
must be being started after the mail client finishes sending the message data 
and before qmail responds with "250" to indicate it was accepted.  From 
spamdyke's point of view, there's no way it can tell why the data stopped 
flowing or why there's been no response from qmail -- it could be due to a slow 
scanner, a failing hard drive, high load, low memory, a blocked filesystem call 
or a million other reasons.  There's just no way to determine if the delay will 
end soon or ever -- if something is deadlocked it's possible there will never 
be any response.  Linux systems can get into exactly that situation when an NFS 
server disappears, for example.

I would like to eventually add hooks to spamdyke to call external filters like 
SpamAssassin, so it could trigger these scans itself instead of using simscan 
to do it.  If that were done, spamdyke be able to close/kill SpamAssassin if it 
took too long, then continue accepting the message.  Until that day however, I 
think the only solution is to increase your idle timeout setting.

-- Sam Clippinger




On Feb 8, 2014, at 4:34 PM, Eric Shubert  wrote:

> On 02/08/2014 02:40 PM, Sam Clippinger wrote:
>> I'm a little unclear here -- what scanning are you doing and when does
>> it take place?
> 
> I'm not crystal clear either about exactly how everything's happening.
> 
> Simscan is invoking clamav and spamassassin. Simscan is implemented via 
> QMAILQUEUE="/var/qmail/bin/simscan".
> 
>> How can spamdyke tell the difference between a delay
>> caused by something on your server versus a delay from the remote sender?
> 
> I've no idea. I'm guessing that simscan isn't given control until the 
> message is completely received. It's at that point, when the message has 
> been completely received but not yet queued, that I think the idle 
> timeout should be disabled.
> 
> The problem appears to be that when when spamdyke does idle timeout, the 
> qmail-queue process can still successfully deliver a message (when it's 
> past the point described above). spamdyke should only initiate a timeout 
> when it can (still) keep a message from being delivered.
> 
> Here's a sample from the log which might make things a little clearer:
> 02-07 14:15:14 tcpserver: status: 1/100
> 02-07 14:15:14 tcpserver: pid 19001 from 70.58.xxx169
> 02-07 14:15:14 tcpserver: ok 19001 
> tacs-mail.datamatters.us:192.168.73.7:25 :70.58.xxx.169::44872
> 02-07 14:15:23 CHKUSER accepted sender: from  remote 
>  rcpt <> : sender accepted
> 02-07 14:15:23 CHKUSER accepted any rcpt: from  remote 
>  rcpt  : accepted 
> any recipient for this domain
> 02-07 14:15:23 policy_check: remote x...@.com -> local x...@.com 
> (UNAUTHENTICATED SENDER)
> 02-07 14:15:23 policy_check: policy allows transmission
> 02-07 14:16:25 spamdyke[19001]: TIMEOUT from: x...@.com to: 
> x...@.com origin_ip: 70.58.xxx.169 origin_rdns: .com auth: 
> (unknown) encryption: TLS reason: TIMEOUT
> 02-07 14:17:58 simscan:[19002]:CLEAN (5.50/12.00):154.6404s:***SPAM*** 
> Fwd_  70.58.xxx.169:x...@.com:x...@.com
> 02-07 14:17:58 tcpserver: end 19001 status 0
> 
> Usually, the simscan message comes before spamdyke. BL is that the 
> message is delivered, and the sender is notified of a failure, causing 
> duplicates in the inbox.
> 
> Thanks Sam. Gotta run.
> 
>> -- Sam Clippinger
>> 
>> 
>> 
>> 
>> On Feb 7, 2014, at 4:37 PM, Eric Shubert
>> > > wrote:
>> 
>>> With spamdyke 4.3.1, I've come across an email which takes an inordinate
>>> amount of time to scan, for whatever reason. I had idle-timeout=60, so
>>> spamdyke would timeout the session, and a minute or so later the scan
>>> completes, and the message is delivered. This causes duplicates though,
>>> as the sender isn't aware of the successful delivery.
>>> 
>>> I've bumped up the idle-timeout to 180, which I expect will remedy the
>>> situation.
>>> 
>>> I wonder, though, if this setting could or should be suspended during
>>> the time which spamdyke is waiting for delivery to happen. Perhaps there
>>> should be 2 settings - one for the incoming side and one for the
>>> delivery side? I like keeping this setting on the low side to keep
>>> senders from tying up incoming processes, yet the setting doesn't seem
>>> to make any sense when waiting for scanning/delivery, especially when
>>> spamdyke can't cancel that part of things.
>>> 
>>> Thanks Sam.
>>> 
>>> --
>>> -Eric 'shubes'
>>> 
>>> ___
>>> spamdyke-users mailing list
>>> spamdyke-users@spamdyke.org
>>> 
>>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>> 
>> 
>>

Re: [spamdyke-users] timeout results in duplicates

[Eric Shubert]

On 02/08/2014 02:40 PM, Sam Clippinger wrote:
> I'm a little unclear here -- what scanning are you doing and when does
> it take place?

I'm not crystal clear either about exactly how everything's happening.

Simscan is invoking clamav and spamassassin. Simscan is implemented via 
QMAILQUEUE="/var/qmail/bin/simscan".

> How can spamdyke tell the difference between a delay
> caused by something on your server versus a delay from the remote sender?

I've no idea. I'm guessing that simscan isn't given control until the 
message is completely received. It's at that point, when the message has 
been completely received but not yet queued, that I think the idle 
timeout should be disabled.

The problem appears to be that when when spamdyke does idle timeout, the 
qmail-queue process can still successfully deliver a message (when it's 
past the point described above). spamdyke should only initiate a timeout 
when it can (still) keep a message from being delivered.

Here's a sample from the log which might make things a little clearer:
02-07 14:15:14 tcpserver: status: 1/100
02-07 14:15:14 tcpserver: pid 19001 from 70.58.xxx169
02-07 14:15:14 tcpserver: ok 19001 
tacs-mail.datamatters.us:192.168.73.7:25 :70.58.xxx.169::44872
02-07 14:15:23 CHKUSER accepted sender: from  remote 
 rcpt <> : sender accepted
02-07 14:15:23 CHKUSER accepted any rcpt: from  remote 
 rcpt  : accepted 
any recipient for this domain
02-07 14:15:23 policy_check: remote x...@.com -> local x...@.com 
(UNAUTHENTICATED SENDER)
02-07 14:15:23 policy_check: policy allows transmission
02-07 14:16:25 spamdyke[19001]: TIMEOUT from: x...@.com to: 
x...@.com origin_ip: 70.58.xxx.169 origin_rdns: .com auth: 
(unknown) encryption: TLS reason: TIMEOUT
02-07 14:17:58 simscan:[19002]:CLEAN (5.50/12.00):154.6404s:***SPAM*** 
Fwd_  70.58.xxx.169:x...@.com:x...@.com
02-07 14:17:58 tcpserver: end 19001 status 0

Usually, the simscan message comes before spamdyke. BL is that the 
message is delivered, and the sender is notified of a failure, causing 
duplicates in the inbox.

Thanks Sam. Gotta run.

> -- Sam Clippinger
>
>
>
>
> On Feb 7, 2014, at 4:37 PM, Eric Shubert
>  > wrote:
>
>> With spamdyke 4.3.1, I've come across an email which takes an inordinate
>> amount of time to scan, for whatever reason. I had idle-timeout=60, so
>> spamdyke would timeout the session, and a minute or so later the scan
>> completes, and the message is delivered. This causes duplicates though,
>> as the sender isn't aware of the successful delivery.
>>
>> I've bumped up the idle-timeout to 180, which I expect will remedy the
>> situation.
>>
>> I wonder, though, if this setting could or should be suspended during
>> the time which spamdyke is waiting for delivery to happen. Perhaps there
>> should be 2 settings - one for the incoming side and one for the
>> delivery side? I like keeping this setting on the low side to keep
>> senders from tying up incoming processes, yet the setting doesn't seem
>> to make any sense when waiting for scanning/delivery, especially when
>> spamdyke can't cancel that part of things.
>>
>> Thanks Sam.
>>
>> --
>> -Eric 'shubes'
>>
>> ___
>> spamdyke-users mailing list
>> spamdyke-users@spamdyke.org
>> 
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>
>
>
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>


-- 
-Eric 'shubes'

___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users

Re: [spamdyke-users] timeout results in duplicates

[Sam Clippinger]

I'm a little unclear here -- what scanning are you doing and when does it take 
place?  How can spamdyke tell the difference between a delay caused by 
something on your server versus a delay from the remote sender?

-- Sam Clippinger




On Feb 7, 2014, at 4:37 PM, Eric Shubert  wrote:

> With spamdyke 4.3.1, I've come across an email which takes an inordinate 
> amount of time to scan, for whatever reason. I had idle-timeout=60, so 
> spamdyke would timeout the session, and a minute or so later the scan 
> completes, and the message is delivered. This causes duplicates though, 
> as the sender isn't aware of the successful delivery.
> 
> I've bumped up the idle-timeout to 180, which I expect will remedy the 
> situation.
> 
> I wonder, though, if this setting could or should be suspended during 
> the time which spamdyke is waiting for delivery to happen. Perhaps there 
> should be 2 settings - one for the incoming side and one for the 
> delivery side? I like keeping this setting on the low side to keep 
> senders from tying up incoming processes, yet the setting doesn't seem 
> to make any sense when waiting for scanning/delivery, especially when 
> spamdyke can't cancel that part of things.
> 
> Thanks Sam.
> 
> -- 
> -Eric 'shubes'
> 
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users

___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users

[spamdyke-users] timeout results in duplicates

[Eric Shubert]

With spamdyke 4.3.1, I've come across an email which takes an inordinate 
amount of time to scan, for whatever reason. I had idle-timeout=60, so 
spamdyke would timeout the session, and a minute or so later the scan 
completes, and the message is delivered. This causes duplicates though, 
as the sender isn't aware of the successful delivery.

I've bumped up the idle-timeout to 180, which I expect will remedy the 
situation.

I wonder, though, if this setting could or should be suspended during 
the time which spamdyke is waiting for delivery to happen. Perhaps there 
should be 2 settings - one for the incoming side and one for the 
delivery side? I like keeping this setting on the low side to keep 
senders from tying up incoming processes, yet the setting doesn't seem 
to make any sense when waiting for scanning/delivery, especially when 
spamdyke can't cancel that part of things.

Thanks Sam.

-- 
-Eric 'shubes'

___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users