Re: [Assp-test] fixes in assp 2.5.4 build 16277

[cw]

Hi Thomas,

Cracked it. It turns out it is all down to debugging. I remembered I turned
that on at the same time as upgrading the version. I've turned that off
again and both ASSP instances are back up to full speed. Phew.

Not sure why debugging had such a major impact, it never has before and
I've had it on and off the past week.

All the best,
Colin.

On Mon, Oct 3, 2016 at 4:13 PM, cw  wrote:

> I've been watching what the process is doing. It seems to be writing to
> the logfiles, or failing to. I have this:
>
> write(63, "2016-10-03 15:41:31 [Worker_7] <"..., 102) = 102
>
> This repeats 999 times until:
>
> write(63, "2016-10-03 15:41:39 [Worker_7] <"..., 101) = 101
>
> There is no line with a matching timestamp in the log file anywhere.
>
> This seems to be a repeating pattern of 9 lines:
>
>
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 102) = 102
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 67) = 67
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 69) = 69
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 154) = 154
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 63) = 63
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 65) = 65
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 69) = 69
> write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 102) = 102
>
> Until the very last pattern which has 101 instead of 102.
>
> I see this over and over again with different lines and worker numbers.
>
> If I look in the debug for this I see the following occurring for that
> thread:
>
> > (at 1475505692.00246)2016-10-03 15:41:32 [Worker_7]  em...@address.tld - noScan
> > (at 1475505692.00461)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.00639)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.00669)2016-10-03 15:41:32 [Worker_7]  
> [CR][LF]
> > (at 1475505692.00695)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.00728)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.00766)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.00799)2016-10-03 15:41:32 [Worker_7]  em...@address.tld - noScan
> > (at 1475505692.00872)2016-10-03 15:41:32 [Worker_7]  em...@address.tld - noScan
> > (at 1475505692.01008)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01085)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01128)2016-10-03 15:41:32 [Worker_7]  
> [CR][LF]
> > (at 1475505692.01167)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01202)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01239)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01282)2016-10-03 15:41:32 [Worker_7]  em...@address.tld - noScan
> > (at 1475505692.01472)2016-10-03 15:41:32 [Worker_7]  em...@address.tld - noScan
> > (at 1475505692.01613)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01746)2016-10-03 15:41:32 [Worker_7]  > (at 1475505692.01954)2016-10-03 15:41:32 [Worker_7]  
> [CR][LF]
> > (at 1475505692.02082)2016-10-03 15:41:32 [Worker_7] 
> I've changed the email address and zero'd the content. So it looks like
> normal behaviour of receiving a message but why the 999 writes that look
> like they should be going to the log file?
>
> Whilst watching all this, I've just had ASSP go completely unresponsive,
> something I've never had before. There's nothing going to the debug file.
> Strace is showing lots of resource temporarily unavailable then it freed
> up, dumped a load of lines to the log file before hanging again. It seems
> to be stuck in a cycle with that now.
>
> I'm also seeing things logged to MainThread_stuck_err.log which is a new
> one for me. They are mostly exactly 62 or 65 seconds with one at 61 seconds.
>
> I can't see much in it. There is always at least one thread with the
> following:
>
> 2016-10-03 16:04:34 Worker(1): last loop start before 0 seconds - signals:
> can:1, state:0, never: - last debug step is : wh: 0 - write:  - wait: 0.002
>
> All the other messages vary.
>
> It is starting to get noticed and reported now so must be having a
> considerable knock on. Absolutely baffled as to what is going on.
>
>
> On Mon, Oct 3, 2016 at 3:42 PM, cw  wrote:
>
>> Hi Thomas,
>>
>> I don't know what is going on. I've dropped back to 16275 and external
>> email starts working again but it is extremely slow. I can't get to the GUI
>> and output to the maillog is slow.
>>
>> I've even dropped back to 16270 and it is still the same. Are there any
>> caches or database entries that could be affected by this that I need to
>> clear?
>>
>> Nothing else changed other than stopping assp, downloading the latest
>> from sourceforge and then starting up ASSP.
>>
>> On Mon, Oct 3, 2016 at 3:19 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> I'm sure, there is no change in build 16277 than can cause this.
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>>

Re: [Assp-test] fixes in assp 2.5.4 build 16277

[cw]

I've been watching what the process is doing. It seems to be writing to the
logfiles, or failing to. I have this:

write(63, "2016-10-03 15:41:31 [Worker_7] <"..., 102) = 102

This repeats 999 times until:

write(63, "2016-10-03 15:41:39 [Worker_7] <"..., 101) = 101

There is no line with a matching timestamp in the log file anywhere.

This seems to be a repeating pattern of 9 lines:


write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 102) = 102
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 67) = 67
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 69) = 69
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 154) = 154
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 63) = 63
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 65) = 65
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 69) = 69
write(63, "2016-10-03 15:41:32 [Worker_7] <"..., 102) = 102

Until the very last pattern which has 101 instead of 102.

I see this over and over again with different lines and worker numbers.

If I look in the debug for this I see the following occurring for that
thread:

> (at 1475505692.00246)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00461)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00639)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00669)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00695)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00728)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00766)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00799)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.00872)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01008)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01085)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01128)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01167)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01202)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01239)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01282)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01472)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01613)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01746)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.01954)2016-10-03 15:41:32 [Worker_7]  (at 1475505692.02082)2016-10-03 15:41:32 [Worker_7]  wrote:

> Hi Thomas,
>
> I don't know what is going on. I've dropped back to 16275 and external
> email starts working again but it is extremely slow. I can't get to the GUI
> and output to the maillog is slow.
>
> I've even dropped back to 16270 and it is still the same. Are there any
> caches or database entries that could be affected by this that I need to
> clear?
>
> Nothing else changed other than stopping assp, downloading the latest from
> sourceforge and then starting up ASSP.
>
> On Mon, Oct 3, 2016 at 3:19 PM, Thomas Eckardt  > wrote:
>
>> I'm sure, there is no change in build 16277 than can cause this.
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:cw 
>> An: ASSP development mailing list 
>> Datum:  03.10.2016 15:39
>> Betreff:Re: [Assp-test] fixes in assp 2.5.4 build 16277
>>
>>
>>
>> Hi Thomas,
>>
>> After upgrading to this build ASSP was only listening on my internal
>> private subnet and was not accepting connections from the outside world.
>> I've had to go straight back down to 16275.
>>
>>
>> On Mon, Oct 3, 2016 at 1:44 PM, Thomas Eckardt
>> 
>> wrote:
>>
>> > Hi all,
>> >
>> > fixed in assp 2.5.4 build 16277:
>> >
>> > changed:
>> >
>> > - for all SSL/TLS connection a 'read ahead' mechanism is implemented to
>> > speed up mail processing
>> >   for small SSL-frame size (< 8kB) - at least by ten times
>> >
>> > added:
>> >
>> > - 'neverQueueSize','Never internaly Queue Mails larger than this Size'
>> >  Default is 20971520 (20MB) - lowest possible value is 100.
>> >  Any mail that is announced to be or grows larger than this size in
>> byte,
>> > will not be queued for actions
>> >  and checks that requires the complete mail to be internaly queued.
>> >  skipped actions are: DKIM signature generation and charset conversions
>> >  skipped checks are: all Plugins in level 2 (complete mail) and the full
>> > mail DKIM check
>> >  Please also check npSize and npSizeOut.
>> >
>> > removed:
>> >
>> > - the hidden config parameter 'neverQueueSize' is now moved to the GUI
>> >
>> > - 'OutgoingBufSizeNew' is removed from the code
>> >
>> > Thomas
>> >
>> > DISCLAIMER:
>> > ***
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use of
>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > ***
>> >
>> >
>> > 
>> > --
>> > Check out the vibrant tech 

Re: [Assp-test] fixes in assp 2.5.4 build 16277

[cw]

Hi Thomas,

I don't know what is going on. I've dropped back to 16275 and external
email starts working again but it is extremely slow. I can't get to the GUI
and output to the maillog is slow.

I've even dropped back to 16270 and it is still the same. Are there any
caches or database entries that could be affected by this that I need to
clear?

Nothing else changed other than stopping assp, downloading the latest from
sourceforge and then starting up ASSP.

On Mon, Oct 3, 2016 at 3:19 PM, Thomas Eckardt 
wrote:

> I'm sure, there is no change in build 16277 than can cause this.
>
> Thomas
>
>
>
>
>
> Von:cw 
> An: ASSP development mailing list 
> Datum:  03.10.2016 15:39
> Betreff:Re: [Assp-test] fixes in assp 2.5.4 build 16277
>
>
>
> Hi Thomas,
>
> After upgrading to this build ASSP was only listening on my internal
> private subnet and was not accepting connections from the outside world.
> I've had to go straight back down to 16275.
>
>
> On Mon, Oct 3, 2016 at 1:44 PM, Thomas Eckardt
> 
> wrote:
>
> > Hi all,
> >
> > fixed in assp 2.5.4 build 16277:
> >
> > changed:
> >
> > - for all SSL/TLS connection a 'read ahead' mechanism is implemented to
> > speed up mail processing
> >   for small SSL-frame size (< 8kB) - at least by ten times
> >
> > added:
> >
> > - 'neverQueueSize','Never internaly Queue Mails larger than this Size'
> >  Default is 20971520 (20MB) - lowest possible value is 100.
> >  Any mail that is announced to be or grows larger than this size in
> byte,
> > will not be queued for actions
> >  and checks that requires the complete mail to be internaly queued.
> >  skipped actions are: DKIM signature generation and charset conversions
> >  skipped checks are: all Plugins in level 2 (complete mail) and the full
> > mail DKIM check
> >  Please also check npSize and npSizeOut.
> >
> > removed:
> >
> > - the hidden config parameter 'neverQueueSize' is now moved to the GUI
> >
> > - 'OutgoingBufSizeNew' is removed from the code
> >
> > Thomas
> >
> > DISCLAIMER:
> > ***
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > ***
> >
> >
> > 
> > --
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > ___
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.5.4 build 16277

[Thomas Eckardt]

I'm sure, there is no change in build 16277 than can cause this.

Thomas





Von:cw 
An: ASSP development mailing list 
Datum:  03.10.2016 15:39
Betreff:Re: [Assp-test] fixes in assp 2.5.4 build 16277



Hi Thomas,

After upgrading to this build ASSP was only listening on my internal
private subnet and was not accepting connections from the outside world.
I've had to go straight back down to 16275.


On Mon, Oct 3, 2016 at 1:44 PM, Thomas Eckardt 

wrote:

> Hi all,
>
> fixed in assp 2.5.4 build 16277:
>
> changed:
>
> - for all SSL/TLS connection a 'read ahead' mechanism is implemented to
> speed up mail processing
>   for small SSL-frame size (< 8kB) - at least by ten times
>
> added:
>
> - 'neverQueueSize','Never internaly Queue Mails larger than this Size'
>  Default is 20971520 (20MB) - lowest possible value is 100.
>  Any mail that is announced to be or grows larger than this size in 
byte,
> will not be queued for actions
>  and checks that requires the complete mail to be internaly queued.
>  skipped actions are: DKIM signature generation and charset conversions
>  skipped checks are: all Plugins in level 2 (complete mail) and the full
> mail DKIM check
>  Please also check npSize and npSizeOut.
>
> removed:
>
> - the hidden config parameter 'neverQueueSize' is now moved to the GUI
>
> - 'OutgoingBufSizeNew' is removed from the code
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, 
legally
> privileged and protected in law and are intended solely for the use of 
the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Very minor request: ClamAV more verbose logging?

[K Post]

Okay, I'm back to standard logging on ClamAV then
Thanks

On Mon, Oct 3, 2016 at 2:55 AM, Thomas Eckardt 
wrote:

> >1) Is verbose logging slowing things
>
> The MainThread goes slower than more is logged
>
> >1) and causing the daemon to be unreachable
>
> No.
>
> >1) is this happening with standard logging too and just not
> >logged?
>
> Yes.
>
> >2) Is this normal? If not, what should I do to fix this?
>
> This is normal. Every time the clamd reloads signatures or does the self
> check (default 600 seconds) it becomes unavailable.
> If you've configured to use more than one clamd, the next will be used.
> If non of the configured clamd is available, you'll get the warning:
> ClamAV Temporary Off :
>
> Thomas
>
>
>
>
> Von:K Post 
> An: ASSP development mailing list 
> Datum:  02.10.2016 20:44
> Betreff:Re: [Assp-test] Very minor request: ClamAV more verbose
> logging?
>
>
>
> Thanks for the reply.
>
> Doesn't ASSP know what it's sending to the scanner though?  And it's ASSP
> that writes to maillog, not ClamAV right?
>
> Separately, I turned ClamAV logging to verbose just to see the logs, and
> with this setting as such, I'm getting:
> Warning: the ClamAV daemon at 127.0.0.1:3310 seems to be down
> a couple of times an hour.
>
> I see no issues with ClamAV, but always get worried about warnings. With
> ClamAV logging set to standard, I don't get these warnings.  So the
> questions for me become:
> 1) Is verbose logging slowing things and causing the daemon to be
> unreachable or is this happening with standard logging too and just not
> logged?
> 2) Is this normal? If not, what should I do to fix this?
>
>
>
> On Sun, Oct 2, 2016 at 3:05 AM, Thomas Eckardt
> 
> wrote:
>
> > The scanning engine does not know where the content comes from
> > (attachment, decompressed attachment, body check, text parts, mail
> > analyzer, archive post processor ... . )
> >
> > So - no chance to have this information there.
> >
> > Thomas
> >
> >
> >
> > Von:K Post 
> > An: ASSP development mailing list 
> > Datum:  01.10.2016 22:02
> > Betreff:[Assp-test] Very minor request: ClamAV more verbose
> > logging?
> >
> >
> >
> > With verbose logging for clamav on, we get lines like:
> > ClamAV: scanned 1146936 bytes in whitelisted message - OK
> >
> > Would it be possible to add the name of the file being scanned?
> > ClamAV: scanned 1146936 bytes in whitelisted message - invoice.pdf - OK
> > 
> > --
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > ___
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > ***
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > ***
> >
> >
> > 
> > --
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > ___
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> 

Re: [Assp-test] fixes in assp 2.5.4 build 16277

[cw]

Hi Thomas,

After upgrading to this build ASSP was only listening on my internal
private subnet and was not accepting connections from the outside world.
I've had to go straight back down to 16275.


On Mon, Oct 3, 2016 at 1:44 PM, Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.5.4 build 16277:
>
> changed:
>
> - for all SSL/TLS connection a 'read ahead' mechanism is implemented to
> speed up mail processing
>   for small SSL-frame size (< 8kB) - at least by ten times
>
> added:
>
> - 'neverQueueSize','Never internaly Queue Mails larger than this Size'
>  Default is 20971520 (20MB) - lowest possible value is 100.
>  Any mail that is announced to be or grows larger than this size in byte,
> will not be queued for actions
>  and checks that requires the complete mail to be internaly queued.
>  skipped actions are: DKIM signature generation and charset conversions
>  skipped checks are: all Plugins in level 2 (complete mail) and the full
> mail DKIM check
>  Please also check npSize and npSizeOut.
>
> removed:
>
> - the hidden config parameter 'neverQueueSize' is now moved to the GUI
>
> - 'OutgoingBufSizeNew' is removed from the code
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Hi Thomas,

I've got 16277 running straight away. Oddly I've not been able to get
connected to the GUI on either server since upgrading I just get timeouts.

I'm not entirely convinced this build is working properly for me. I'm
seeing only one email every few minutes rather than the constant output to
the logfiles.

Unfortunately I only saw that stuck message once and it cleared itself
quite quickly/before ASSP decided it was shutting itself down. Most of the
time there are no stuck workers and the status page shows green.



On Mon, Oct 3, 2016 at 2:05 PM, Thomas Eckardt 
wrote:

> >action: header (Content-Disposition -attr)
>
> 2.5.4 16277 will no longer use this output to state the worker
>
> please post the line produced with the current build
>
> "action: header (Content-Disposition -attr)" may be stated by a virus or
> attachment check, done in the body check of assp.pl or the ASSP_AFC plugin
>
> Thomas
>
>
>
>
>
> Von:cw 
> An: ASSP development mailing list 
> Datum:  30.09.2016 17:02
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Mixed results on this. So far no problems with running workers being
> logged
> but the GUI has become incredibly unresponsive. By unresponsive I mean I
> waited a good couple of minutes for the shutdown_list page to load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to
> roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw  wrote:
>
> > I wish I'd spotted this before writing out the other message. I'll give
> it
> > a test now for you.
> >
> > On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> > thomas.ecka...@thockar.com> wrote:
> >
> >> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
> >> CVS /test
> >>
> >> Thomas
> >>
> >>
> >>
> >> Von:cw 
> >> An: ASSP development mailing list 
> >> Datum:  29.09.2016 16:40
> >> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> >> servers
> >>
> >>
> >>
> >> Hi Thomas,
> >> I moved up to 16270 following this thread of discussion but then had a
> day
> >> working away. I've come back to huge issues with delays, mails not
> going
> >> through and many, many of these in the logs:
> >>
> >> Info: unable to detect any running worker for a new connection - wait
> (max
> >> 30 seconds)
> >>
> >> When I say many, I have over 21,000 lines in today's log file. I also
> >> found
> >> the GUI unresponsive or not connecting at all and ASSP restarting quite
> >> regularly.
> >>
> >> I've dropped back to 16256 and things are instantly better. Do you
> think
> >> going up to 16273 might improve things over 16270 or am I better
> holding
> >> off for now?
> >> All the best,
> >> Colin.
> >>
> >> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
> >> 
> >> wrote:
> >>
> >> > I just released 2.5.2 build 16273 at CVS test folder
> >> >
> >> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >> >
> >> > This release should make a very large difference for SSL/TLS mails
> sent
> >> by
> >> > hosts that uses small SSL-frame size.
> >> >
> >> > Tell me your test results.
> >> >
> >> >
> >> > Thomas
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Von:K Post 
> >> > An: ASSP development mailing list
> 
> >> > Datum:  28.09.2016 19:42
> >> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses
> /
> >> > servers
> >> >
> >> >
> >> >
> >> > But I want a postman driving a Ferarri with monster truck tires that
> can
> >> > roll over the traffic (and if wishes are being granted, I'd prefer
> the
> >> car
> >> > in a deep blue instead of classic red).
> >> >
> >> > We regularly see people attaching large files or a bunch of smaller
> ones
> >> > that add up to a big email, I'm talking lots and lots of different
> >> people
> >> > from outside the organization sending to us, and this happens on a
> daily
> >> > basis.  It's especially popular with photos and huge scans multi-page
> >> > 600dpi (which people don't understand can be done at low resolution).
> >> > Often it's people sending in scanned official documents for us to
> review
> >> > an
> >> > help them.  They're not our staff, they're the people we help.  They
> >> have
> >> > a
> >> > tendency of not following any instructions, and ignore the fact that
> we
> >> > have a web 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>action: header (Content-Disposition -attr)

2.5.4 16277 will no longer use this output to state the worker

please post the line produced with the current build

"action: header (Content-Disposition -attr)" may be stated by a virus or 
attachment check, done in the body check of assp.pl or the ASSP_AFC plugin

Thomas





Von:cw 
An: ASSP development mailing list 
Datum:  30.09.2016 17:02
Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Mixed results on this. So far no problems with running workers being 
logged
but the GUI has become incredibly unresponsive. By unresponsive I mean I
waited a good couple of minutes for the shutdown_list page to load.
The dot on the main page is red yet the workers page is all green.
Scratch that, it has refreshed again and I have a worker stuck:
Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
filename name (stuck)
30s later and it is healthy again..

On the server I haven't upgraded the shutdown_list page comes up within
seconds. I'm not sure whether to leave it running or whether this is
evidence of the same kind of unresponsiveness that cause me to have to 
roll
back earlier this week.

On Fri, Sep 30, 2016 at 3:29 PM, cw  wrote:

> I wish I'd spotted this before writing out the other message. I'll give 
it
> a test now for you.
>
> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
>> CVS /test
>>
>> Thomas
>>
>>
>>
>> Von:cw 
>> An: ASSP development mailing list 
>> Datum:  29.09.2016 16:40
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> Hi Thomas,
>> I moved up to 16270 following this thread of discussion but then had a 
day
>> working away. I've come back to huge issues with delays, mails not 
going
>> through and many, many of these in the logs:
>>
>> Info: unable to detect any running worker for a new connection - wait 
(max
>> 30 seconds)
>>
>> When I say many, I have over 21,000 lines in today's log file. I also
>> found
>> the GUI unresponsive or not connecting at all and ASSP restarting quite
>> regularly.
>>
>> I've dropped back to 16256 and things are instantly better. Do you 
think
>> going up to 16273 might improve things over 16270 or am I better 
holding
>> off for now?
>> All the best,
>> Colin.
>>
>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>> 
>> wrote:
>>
>> > I just released 2.5.2 build 16273 at CVS test folder
>> >
>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>> >
>> > This release should make a very large difference for SSL/TLS mails 
sent
>> by
>> > hosts that uses small SSL-frame size.
>> >
>> > Tell me your test results.
>> >
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:K Post 
>> > An: ASSP development mailing list 

>> > Datum:  28.09.2016 19:42
>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses 
/
>> > servers
>> >
>> >
>> >
>> > But I want a postman driving a Ferarri with monster truck tires that 
can
>> > roll over the traffic (and if wishes are being granted, I'd prefer 
the
>> car
>> > in a deep blue instead of classic red).
>> >
>> > We regularly see people attaching large files or a bunch of smaller 
ones
>> > that add up to a big email, I'm talking lots and lots of different
>> people
>> > from outside the organization sending to us, and this happens on a 
daily
>> > basis.  It's especially popular with photos and huge scans multi-page
>> > 600dpi (which people don't understand can be done at low resolution).
>> > Often it's people sending in scanned official documents for us to 
review
>> > an
>> > help them.  They're not our staff, they're the people we help.  They
>> have
>> > a
>> > tendency of not following any instructions, and ignore the fact that 
we
>> > have a web based system for the process.  We can't control it and the
>> > powers that be don't want us lowering the 30 MB threshold across the
>> > board.  Lot of these people use gmail.com addresses and google allows
>> for
>> > up to 25 MB - https://support.google.com/mail/answer/6584
>> >
>> > I think it's really interesting that google seems to use this
>> inefficient
>> > small packet size for SSL, allows for 25MB emails, is a big proponent 
of
>> > SSL, and at the same time doesn't allow mails to take more than 15
>> minutes
>> > to transfer.  Now that you've made things >much< more efficient on 
the
>> > ASSP
>> > side, I'm hoping that all will be okay.  I just get annoyed by
>> > inefficiency.
>> >
>> >
>> > I'm going to tryrunning with npSize of zero, the no queuing size set
>> very
>> > high and see how that goes.  I want to insure that even the biggest
>> 

[Assp-test] fixes in assp 2.5.4 build 16277

[Thomas Eckardt]

Hi all,

fixed in assp 2.5.4 build 16277:

changed:

- for all SSL/TLS connection a 'read ahead' mechanism is implemented to 
speed up mail processing
  for small SSL-frame size (< 8kB) - at least by ten times

added:

- 'neverQueueSize','Never internaly Queue Mails larger than this Size'
 Default is 20971520 (20MB) - lowest possible value is 100. 
 Any mail that is announced to be or grows larger than this size in byte, 
will not be queued for actions 
 and checks that requires the complete mail to be internaly queued.
 skipped actions are: DKIM signature generation and charset conversions
 skipped checks are: all Plugins in level 2 (complete mail) and the full 
mail DKIM check
 Please also check npSize and npSizeOut.

removed:

- the hidden config parameter 'neverQueueSize' is now moved to the GUI

- 'OutgoingBufSizeNew' is removed from the code 

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] PB-IP-Score suddenly dropping

[Thomas Eckardt]

>Wouldn't it be better to have a "gliding" score, i.e. with every

nice idea - thrown away in 2008 for the PBBlack
for some temporary hashes (domainIP ... outgoing mail limiter ) it is 
done this way

assume the IP has been connected 100 times - we'll get something like

time1 value1 time2 value2 ...time100 value 100

It does'nt matter, if this one record or pointered records in multiple 
hashes or arrays. To get the resulting IP-score, we have to read and to 
sum all. Not a big deal? Right, if we would have only one worker. But all 
workers accessing this hash. So we would have to lock all involved records 
while calculating and the rewrite.
Next issue is the record lenght in RDB (mysql..) - it must be 
variable, the standard assp DB-code is unable to handle this for all DB 
types.

>amnesty  

Yes, this is better than having false positives. Most times the next 
mail(s) after the 'amnesty' will be catched. If not, you'll get one spam 
in 6 hours.
IP based blocking can become a very big issue in multi-domain 
environments, because someones SPAM may be others HAM - sent from the same 
IP.

You may set 'PenaltyExpiration' to a very high value - but be carefull 


Thomas


Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  03.10.2016 13:40
Betreff:Re: [Assp-test] PB-IP-Score suddenly dropping



Hi Thomas,
thanks for explaining this behaviour. Let's see if I get this right...

Let's assume an IP reveals constant misbehaviour adding a PB-IP-Score of 
60
every hour. It started off at 0 so after 6 hours (default 
PenaltyExpiration)
the score would be 360 and rising, but *surprise* after PeneltyExpiration
time the IP gets a complete amnesty and is allowed to restart clean at 0
again.

That does not reflect that IP's behaviour. I think the amnesty can be
justified as soon as the offending IP sends one good message. But if it 
does
not?
Wouldn't it be better to have a "gliding" score, i.e. with every
recalculation all entries which are older than PenaltyExpiration minutes 
get
substracted from the overall score, but the rest is kept? (Obvious 
problem:
codewise this could be complicated, because you would have to keep track 
of
every single increment of the score and its timestamp.)

Best regards
Dirk
 
-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 3. Oktober 2016 11:50
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

The PBBlack record is removed after 'PenaltyExpiration' minutes of the
record creation (NOT the last update).

Thomas





Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  03.10.2016 10:19
Betreff:Re: [Assp-test] PB-IP-Score suddenly dropping



Hi Thomas,

if there was a good message causing this, then I should see the IP in
question in my log before the drop. But there is not a single line.The 
score
is high, nothing happens, the score is low. 
This happened again today and I grep'ed the log for e.g.  118.71.251
(leaving out the last byte of the IP to see everything from a /24 area
around it):

2016-10-03 02:52:52 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 480, added 60 in this session
2016-10-03 02:52:52 [Worker_1] 118.71.251.67 disconnected:
session:7F11A94F5860 118.71.251.67 - processing time 24 seconds
2016-10-03 06:37:38 [Worker_1] Connected: session:7F11A94EBAB0
118.71.251.67:20540 > 192.168.12.242:25 > 127.0.0.1:125
2016-10-03 06:37:38 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs
2016-10-03 06:37:39 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 60, added 60 for AUTHErrors

Nobody from 118.71.251 shows up between 02:52 and 06:37. Still the score
drops from 480 to 0. There is however some background work being done 
during
that time, e.g.:

2016-10-03 03:31:11 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=81, deleted=19
2016-10-03 06:31:15 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=76, deleted=12

Can you give me a hint what to look at to better understand this? I have 
set
PenaltyDuration = 60 and PenaltyExpiration=720. What else can be of
influence here?

Thanks a lot
Dirk

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Gesendet: Montag, 3. Oktober 2016 09:12
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

There are dozend of reasons why this can happen.
Most common is 

Re: [Assp-test] PB-IP-Score suddenly dropping

[Dirk Kulmsee]

Hi Thomas,
thanks for explaining this behaviour. Let's see if I get this right...

Let's assume an IP reveals constant misbehaviour adding a PB-IP-Score of 60
every hour. It started off at 0 so after 6 hours (default PenaltyExpiration)
the score would be 360 and rising, but *surprise* after PeneltyExpiration
time the IP gets a complete amnesty and is allowed to restart clean at 0
again.

That does not reflect that IP's behaviour. I think the amnesty can be
justified as soon as the offending IP sends one good message. But if it does
not?
Wouldn't it be better to have a "gliding" score, i.e. with every
recalculation all entries which are older than PenaltyExpiration minutes get
substracted from the overall score, but the rest is kept? (Obvious problem:
codewise this could be complicated, because you would have to keep track of
every single increment of the score and its timestamp.)

Best regards
Dirk
 
-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 3. Oktober 2016 11:50
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

The PBBlack record is removed after 'PenaltyExpiration' minutes of the
record creation (NOT the last update).

Thomas





Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  03.10.2016 10:19
Betreff:Re: [Assp-test] PB-IP-Score suddenly dropping



Hi Thomas,

if there was a good message causing this, then I should see the IP in
question in my log before the drop. But there is not a single line.The score
is high, nothing happens, the score is low. 
This happened again today and I grep'ed the log for e.g.  118.71.251
(leaving out the last byte of the IP to see everything from a /24 area
around it):

2016-10-03 02:52:52 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 480, added 60 in this session
2016-10-03 02:52:52 [Worker_1] 118.71.251.67 disconnected:
session:7F11A94F5860 118.71.251.67 - processing time 24 seconds
2016-10-03 06:37:38 [Worker_1] Connected: session:7F11A94EBAB0
118.71.251.67:20540 > 192.168.12.242:25 > 127.0.0.1:125
2016-10-03 06:37:38 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for External
IPs
2016-10-03 06:37:39 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 PB-IP-Score for '118.71.251.67'
is 60, added 60 for AUTHErrors

Nobody from 118.71.251 shows up between 02:52 and 06:37. Still the score
drops from 480 to 0. There is however some background work being done during
that time, e.g.:

2016-10-03 03:31:11 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=81, deleted=19
2016-10-03 06:31:15 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=76, deleted=12

Can you give me a hint what to look at to better understand this? I have set
PenaltyDuration = 60 and PenaltyExpiration=720. What else can be of
influence here?

Thanks a lot
Dirk

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Gesendet: Montag, 3. Oktober 2016 09:12
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

There are dozend of reasons why this can happen.
Most common is 'PenaltyExpiration'.
If there is a good mail transfered by an IP, the IP score is deleted to
prevent false positives. Where good means - no doubed, like 'contentOnly',
RWL, SPF, DKIM 

Thomas.


Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  02.10.2016 20:04
Betreff:[Assp-test] PB-IP-Score suddenly dropping



Hi all,

I just tracked some IPs through my logfiles just to see how they build up
their score. Something strange is happening:

Case 1: between 09:51:13 and 12:49:10 PB-IP-Score drops from 600 to 0
without any visible reason

2016-10-02 06:13:54 [Worker_1] Connected: session:7F11F4A35FA0
118.71.251.67:53467 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 06:13:54 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for External
IPs

2016-10-02 06:13:55 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 PB-IP-Score for '118.71.251.67'
is 540, added 60 for AUTHErrors

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 540, 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Hi Thomas,

I've put 16275 on one of my servers this morning. I'm seeing the no running
workers error every 10-20 minutes causing ASSP to shut down. I've sat there
with the worker status page open and I'm not seeing any workers getting
stuck that coincide with this problem. In fact when the error is scrolling
past in the logs the workers are happily carrying on with whatever emails
they are processing.

Whilst typing, worker 5 has just gone to Bayes(OK_Run) Stuck with a timer
of around 240 before it cleared. This didn't trigger the fault just like
the other stuck message didn't really coincide and cleared itself.

I've upgraded the second server and so far am not seeing the delays that I
did with 16270. I'll keep an eye on it. Sure wish I could figure out what
is causing the no running workers error though.

All the best,
Colin

On Sun, Oct 2, 2016 at 7:44 PM, K Post  wrote:

> It's been about a full day now with 75.  I see nothing but greatness.
>
> On Sun, Oct 2, 2016 at 2:59 AM, Thomas Eckardt  >
> wrote:
>
> > >then I see 9 seconds of idle/damping,
> >
> > If the complete mail is queued, this time is required to do the post
> > checks and conversions (charset conversion, plugins level 2, DKIM) and to
> > send the mail data to your MTA.
> >
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:K Post 
> > An: ASSP development mailing list 
> > Datum:  01.10.2016 22:01
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > I've been testing 16275 for a couple hours now.  FAST FAST FAST and
> stable
> > so far.   I see no errors in the maillog.
> >
> > Some notes / questions
> >
> > 1)  11MB attachment (14 MB email after encoding) transfers from google,
> > total time 25-29 seconds.  It seems that the whole email transfers in
> > about
> > 20 seconds, then I see 9 seconds of idle/damping,   I've never before
> seen
> > (or noticed I suppose) the idle column grow under normal circumstances.
> > Did you change the way messages are processed, in terms of doing
> > scans/analysis post transfer?  This idle counter doesn't matter to me,
> > speed's great, but I want to insure there's nothing wrong - and I'm
> > curious
> > if something changed on this front.
> >
> > 2) I see no notable difference in speed when I disable TLS for google
> now.
> > This is TERRIFIC.  There's certainly processing overhead for SSL, but
> your
> > code is so fast now that it's not noticeable.
> >
> > 3) Any point in adding a column to shutdown_list saying what the task at
> > hand is? AFC?  ClamAV?  Etc?
> >
> > For reference - we resumed this discussion August 1 (there were previous
> > attempts at getting this resolved, but they didn't go anywhere).  About 2
> > months later, 70 or so messages in this thread, a ton of your time,
> > dedication, and brainpower, we went from a gmail TLS message totaling 14
> > MB
> > taking upwards of 13 minutes and being unreliable now take only 25
> seconds
> > and be rock solid!  I don't know how you figured all of this out, put up
> > with my persistence, or had the time and brainpower to spare, but you
> seem
> > to have conquered this terrible problem!
> >
> > Tausend dank!  Ich bin Ihnen sehr dankbar für.
> >
> >
> >
> >
> > On Sat, Oct 1, 2016 at 11:23 AM, Thomas Eckardt
> > 
> > wrote:
> >
> > > Ken, 2.5.2 build 16275 is in CVS /test
> > >
> > > Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused
> > > some sockets - some times - at any state - to become unreadable.
> > > I've done extensive test on this - but I was not able to find a
> > > reproduceable reason for the behavior.
> > > I removed this code in build 16275. The read ahead is still available
> > for
> > > TLS/SSL and it is used per default by assp. The mechanism is a bit
> > > different like in build 16273/74.
> > > Every TLS/SSL socket gets additionally 50 milliseconds to read and
> > decode
> > > a much as available data from the underlying BIO
> > >
> > > On my nice old slow system, I was able to receive a 20MB mail from
> gmail
> > > in 110 seconds.
> > > Even for yahoo (they use 4096 byte SSL frames) it makes a big
> > difference.
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > > Von:K Post 
> > > An: ASSP development mailing list  >
> > > Datum:  30.09.2016 19:21
> > > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > > servers
> > >
> > >
> > >
> > > 70 and 71 is fine here (Windows).73 was SUPER fast with SSL
> messages
> > > from gmail, but then we got the idle / delay issues and had to revert
> to
> > > 71.  Haven't had a chance to try 74 (need to wait for after hours)
> > >
> > >
> > > On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring 
> > > wrote:
> > >
> > > > 16256 works acceptably but shuts down once or twice a 

Re: [Assp-test] PB-IP-Score suddenly dropping

[Thomas Eckardt]

The PBBlack record is removed after 'PenaltyExpiration' minutes of the 
record creation (NOT the last update).

Thomas





Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  03.10.2016 10:19
Betreff:Re: [Assp-test] PB-IP-Score suddenly dropping



Hi Thomas,

if there was a good message causing this, then I should see the IP in
question in my log before the drop. But there is not a single line.The 
score
is high, nothing happens, the score is low. 
This happened again today and I grep'ed the log for e.g.  118.71.251
(leaving out the last byte of the IP to see everything from a /24 area
around it):

2016-10-03 02:52:52 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 480, added 60 in this session
2016-10-03 02:52:52 [Worker_1] 118.71.251.67 disconnected:
session:7F11A94F5860 118.71.251.67 - processing time 24 seconds
2016-10-03 06:37:38 [Worker_1] Connected: session:7F11A94EBAB0
118.71.251.67:20540 > 192.168.12.242:25 > 127.0.0.1:125
2016-10-03 06:37:38 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs
2016-10-03 06:37:39 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 60, added 60 for AUTHErrors

Nobody from 118.71.251 shows up between 02:52 and 06:37. Still the score
drops from 480 to 0. There is however some background work being done 
during
that time, e.g.:

2016-10-03 03:31:11 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=81, deleted=19
2016-10-03 06:31:15 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=76, deleted=12

Can you give me a hint what to look at to better understand this? I have 
set
PenaltyDuration = 60 and PenaltyExpiration=720. What else can be of
influence here?

Thanks a lot
Dirk

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 3. Oktober 2016 09:12
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

There are dozend of reasons why this can happen.
Most common is 'PenaltyExpiration'.
If there is a good mail transfered by an IP, the IP score is deleted to
prevent false positives. Where good means - no doubed, like 'contentOnly',
RWL, SPF, DKIM 

Thomas.


Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  02.10.2016 20:04
Betreff:[Assp-test] PB-IP-Score suddenly dropping



Hi all,

I just tracked some IPs through my logfiles just to see how they build up
their score. Something strange is happening:

Case 1: between 09:51:13 and 12:49:10 PB-IP-Score drops from 600 to 0
without any visible reason

2016-10-02 06:13:54 [Worker_1] Connected: session:7F11F4A35FA0
118.71.251.67:53467 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 06:13:54 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs

2016-10-02 06:13:55 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 540, added 60 for AUTHErrors

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 540, added 60 in this session

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 disconnected:
session:7F11F4A35FA0 118.71.251.67 - processing time 13 seconds

2016-10-02 06:22:56 [Worker_1] Delayed ip 118.71.251.67, because
PBBlack(540) is higher than DelayIP(500)- last penalty reason was:
AUTHErrors

2016-10-02 07:07:29 [Worker_1] Connected: session:7F11F4C41160
118.71.251.67:54518 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs

2016-10-02 07:07:29 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 600, added 60 for AUTHErrors

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 07:07:53 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 600, added 60 in this session

2016-10-02 07:07:53 [Worker_1] 

Re: [Assp-test] PB-IP-Score suddenly dropping

[Dirk Kulmsee]

Hi Thomas,

if there was a good message causing this, then I should see the IP in
question in my log before the drop. But there is not a single line.The score
is high, nothing happens, the score is low. 
This happened again today and I grep'ed the log for e.g.  118.71.251
(leaving out the last byte of the IP to see everything from a /24 area
around it):

2016-10-03 02:52:52 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 480, added 60 in this session
2016-10-03 02:52:52 [Worker_1] 118.71.251.67 disconnected:
session:7F11A94F5860 118.71.251.67 - processing time 24 seconds
2016-10-03 06:37:38 [Worker_1] Connected: session:7F11A94EBAB0
118.71.251.67:20540 > 192.168.12.242:25 > 127.0.0.1:125
2016-10-03 06:37:38 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for External
IPs
2016-10-03 06:37:39 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60
2016-10-03 06:37:39 [Worker_1] 118.71.251.67 PB-IP-Score for '118.71.251.67'
is 60, added 60 for AUTHErrors

Nobody from 118.71.251 shows up between 02:52 and 06:37. Still the score
drops from 480 to 0. There is however some background work being done during
that time, e.g.:

2016-10-03 03:31:11 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=81, deleted=19
2016-10-03 06:31:15 [Worker_1] PenaltyBox: cleaning BlackBox (PBBlack)
finished: IP's before=76, deleted=12

Can you give me a hint what to look at to better understand this? I have set
PenaltyDuration = 60 and PenaltyExpiration=720. What else can be of
influence here?

Thanks a lot
Dirk

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 3. Oktober 2016 09:12
An: ASSP development mailing list 
Betreff: Re: [Assp-test] PB-IP-Score suddenly dropping

There are dozend of reasons why this can happen.
Most common is 'PenaltyExpiration'.
If there is a good mail transfered by an IP, the IP score is deleted to
prevent false positives. Where good means - no doubed, like 'contentOnly',
RWL, SPF, DKIM 

Thomas.


Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  02.10.2016 20:04
Betreff:[Assp-test] PB-IP-Score suddenly dropping



Hi all,

I just tracked some IPs through my logfiles just to see how they build up
their score. Something strange is happening:

Case 1: between 09:51:13 and 12:49:10 PB-IP-Score drops from 600 to 0
without any visible reason

2016-10-02 06:13:54 [Worker_1] Connected: session:7F11F4A35FA0
118.71.251.67:53467 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 06:13:54 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for External
IPs

2016-10-02 06:13:55 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 PB-IP-Score for '118.71.251.67'
is 540, added 60 for AUTHErrors

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 540, added 60 in this session

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 disconnected:
session:7F11F4A35FA0 118.71.251.67 - processing time 13 seconds

2016-10-02 06:22:56 [Worker_1] Delayed ip 118.71.251.67, because
PBBlack(540) is higher than DelayIP(500)- last penalty reason was:
AUTHErrors

2016-10-02 07:07:29 [Worker_1] Connected: session:7F11F4C41160
118.71.251.67:54518 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for External
IPs

2016-10-02 07:07:29 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 PB-IP-Score for '118.71.251.67'
is 600, added 60 for AUTHErrors

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 07:07:53 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 600, added 60 in this session

2016-10-02 07:07:53 [Worker_1] 118.71.251.67 disconnected:
session:7F11F4C41160 118.71.251.67 - processing time 24 seconds

2016-10-02 09:51:13 [Worker_1] Delayed ip 118.71.251.67, because
PBBlack(600) is higher than DelayIP(500)- last penalty reason was:
AUTHErrors

2016-10-02 12:49:10 [Worker_1] Connected: session:7F11F573EEF0
118.71.251.67:2425 > 192.168.12.242:25 > 

Re: [Assp-test] PB-IP-Score suddenly dropping

[Thomas Eckardt]

There are dozend of reasons why this can happen.
Most common is 'PenaltyExpiration'.
If there is a good mail transfered by an IP, the IP score is deleted to 
prevent false positives. Where good means - no doubed, like 'contentOnly', 
RWL, SPF, DKIM 

Thomas.






Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  02.10.2016 20:04
Betreff:[Assp-test] PB-IP-Score suddenly dropping



Hi all,

 

I just tracked some IPs through my logfiles just to see how they build up
their score. Something strange is happening:

 

Case 1: between 09:51:13 and 12:49:10 PB-IP-Score drops from 600 to 0
without any visible reason

 

2016-10-02 06:13:54 [Worker_1] Connected: session:7F11F4A35FA0
118.71.251.67:53467 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 06:13:54 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs

2016-10-02 06:13:55 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 540, added 60 for AUTHErrors

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 06:13:55 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 540, added 60 in this session

2016-10-02 06:14:07 [Worker_1] 118.71.251.67 disconnected:
session:7F11F4A35FA0 118.71.251.67 - processing time 13 seconds

2016-10-02 06:22:56 [Worker_1] Delayed ip 118.71.251.67, because
PBBlack(540) is higher than DelayIP(500)- last penalty reason was:
AUTHErrors

2016-10-02 07:07:29 [Worker_1] Connected: session:7F11F4C41160
118.71.251.67:54518 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs

2016-10-02 07:07:29 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 600, added 60 for AUTHErrors

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 07:07:29 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 07:07:53 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 600, added 60 in this session

2016-10-02 07:07:53 [Worker_1] 118.71.251.67 disconnected:
session:7F11F4C41160 118.71.251.67 - processing time 24 seconds

2016-10-02 09:51:13 [Worker_1] Delayed ip 118.71.251.67, because
PBBlack(600) is higher than DelayIP(500)- last penalty reason was:
AUTHErrors

2016-10-02 12:49:10 [Worker_1] Connected: session:7F11F573EEF0
118.71.251.67:2425 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 12:49:10 [Worker_1] 118.71.251.67 Disabled SMTP AUTH for 
External
IPs

2016-10-02 12:49:10 [Worker_1] [unsupported_AUTH] 118.71.251.67 AUTH not
allowed

2016-10-02 12:49:10 [Worker_1] 118.71.251.67 Message-Score: added 60
(autValencePB) for too many AUTH errors from 118.71.251.0, total score for
this message is now 60

2016-10-02 12:49:10 [Worker_1] 118.71.251.67 PB-IP-Score for 
'118.71.251.67'
is 60, added 60 for AUTHErrors

2016-10-02 12:49:10 [Worker_1] 118.71.251.67 [SMTP Error] 502 AUTH not
supported

2016-10-02 12:49:11 [Worker_1] 118.71.251.67 info: start damping (12 s)

2016-10-02 12:49:34 [Worker_1] 118.71.251.67 info: PB-IP-Score for
'118.71.251.67' is 60, added 60 in this session

2016-10-02 12:49:34 [Worker_1] 118.71.251.67 disconnected:
session:7F11F573EEF0 118.71.251.67 - processing time 24 seconds

 

Case 2: between 15:02:57 and 15:41:09 PB-IP-Score drops from 600 to 0
without any visible reason

 

2016-10-02 11:49:40 [Worker_1] Connected: session:7F11F65EC988
46.32.239.160:64727 > 192.168.12.242:25 > 127.0.0.1:125

2016-10-02 11:49:40 [Worker_1] 46.32.239.160 Disabled SMTP AUTH for 
External
IPs

2016-10-02 11:49:41 [Worker_1] [unsupported_AUTH] 46.32.239.160 AUTH not
allowed

2016-10-02 11:49:41 [Worker_1] 46.32.239.160 Message-Score: added 60
(autValencePB) for too many AUTH errors from 46.32.239.0, total score for
this message is now 60

2016-10-02 11:49:41 [Worker_1] 46.32.239.160 PB-IP-Score for 
'46.32.239.160'
is 540, added 60 for AUTHErrors

2016-10-02 11:49:41 [Worker_1] 46.32.239.160 [SMTP Error] 502 AUTH not
supported

2016-10-02 11:49:41 [Worker_1] 46.32.239.160 info: start damping (12 s)

2016-10-02 11:50:05 [Worker_1] 46.32.239.160 info: PB-IP-Score for
'46.32.239.160' is 540, added 60 in this session

2016-10-02 11:50:05 [Worker_1] 46.32.239.160 disconnected:
session:7F11F65EC988 46.32.239.160 - processing time 25 seconds

2016-10-02 14:43:24 [Worker_1] Delayed ip 

Re: [Assp-test] Very minor request: ClamAV more verbose logging?

[Thomas Eckardt]

>1) Is verbose logging slowing things 

The MainThread goes slower than more is logged

>1) and causing the daemon to be unreachable

No.

>1) is this happening with standard logging too and just not
>logged?

Yes.

>2) Is this normal? If not, what should I do to fix this?

This is normal. Every time the clamd reloads signatures or does the self 
check (default 600 seconds) it becomes unavailable.
If you've configured to use more than one clamd, the next will be used.
If non of the configured clamd is available, you'll get the warning: 
ClamAV Temporary Off :

Thomas




Von:K Post 
An: ASSP development mailing list 
Datum:  02.10.2016 20:44
Betreff:Re: [Assp-test] Very minor request: ClamAV more verbose 
logging?



Thanks for the reply.

Doesn't ASSP know what it's sending to the scanner though?  And it's ASSP
that writes to maillog, not ClamAV right?

Separately, I turned ClamAV logging to verbose just to see the logs, and
with this setting as such, I'm getting:
Warning: the ClamAV daemon at 127.0.0.1:3310 seems to be down
a couple of times an hour.

I see no issues with ClamAV, but always get worried about warnings. With
ClamAV logging set to standard, I don't get these warnings.  So the
questions for me become:
1) Is verbose logging slowing things and causing the daemon to be
unreachable or is this happening with standard logging too and just not
logged?
2) Is this normal? If not, what should I do to fix this?



On Sun, Oct 2, 2016 at 3:05 AM, Thomas Eckardt 

wrote:

> The scanning engine does not know where the content comes from
> (attachment, decompressed attachment, body check, text parts, mail
> analyzer, archive post processor ... . )
>
> So - no chance to have this information there.
>
> Thomas
>
>
>
> Von:K Post 
> An: ASSP development mailing list 
> Datum:  01.10.2016 22:02
> Betreff:[Assp-test] Very minor request: ClamAV more verbose
> logging?
>
>
>
> With verbose logging for clamav on, we get lines like:
> ClamAV: scanned 1146936 bytes in whitelisted message - OK
>
> Would it be possible to add the name of the file being scanned?
> ClamAV: scanned 1146936 bytes in whitelisted message - invoice.pdf - OK
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, 
legally
> privileged and protected in law and are intended solely for the use of 
the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test