[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23331

[Thomas Eckardt]

fixed in assp 2.8.2 *SPAM-Eliminator* build 23331:

- the memory footprint of the rebuild spamdb task is reduced
- the output for the size of the tmpDB folder in the rebuild report was 
wrong
- some GUI corrections
- wrong configuration of the database drivers DBD::ODBC and DBD::ADO 
crashed assp instead falling back to flat files
- used database drivers were missing in the output of 
notes/loaded_perl_modules.txt
- it was no longer possible to monitor the rebuild spamdb task in the 
worker status screen, because of too frequently updates by 'getEmailAddr'
- unknown characterset (eg: handcrafted and invalid) defintions in emails 
caused unexpected exception in several code parts of assp
- improved parsing and processing of (handcrafted) html code in text/plain 
email parts

changed:

- The text processing engine for bayesian and HMM is improved. It is 
recommended to run a rebuildSpamDB after upgrading to this version. 


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Perl 5.38.0.1 for Windows - updated

[Thomas Eckardt]

Hi all,

a new build of perl 5.38.0.1 for windows is available at sourceforge

https://sourceforge.net/projects/assp/files/ASSP%20V2%20multithreading/ASSP%20V2%20module%20installation/strawberry-perl-5.38.0.1-64bit_gcc13-relocateable_4-assp.7z

This perl was build using the GNU GCC version 13 (the version available 
until today was compiled with GCC 8.3.0) and the modified perl module 
Perl::Dist::Strawberry

gcc version 13.1.0 (MinGW-W64 x86_64-msvcrt-posix-seh, built by Brecht 
Sanders)

Several perl modules got a correction to prevent unexpected errors and 
crashes - for example Win32::Unicode (SEGV in Win32::Unicode::Dir because 
of wrong pointers in C-code).

The module DBD::mysql was upgraded to version 5.002 using the GNU-compiled 
mysql (liblibmysql.dll) library version 8.0.35. The mysql lib-version 
8.0.35 is not able to connect to mysql servers version 5.1 (and lower) 
using a password.
Connecting to any version 8 mysql server is no problem (as well as version 
5.7). Connecting to mysql servers version 5.5 is not tested, but should 
work.
The installed liblibmysql.dll library version 8.0.35 requires an installed 
msvcr100.dll (Microsoft Visual C++ 2010 x64 Redistributable)

All libraries, header files and perl modules in this build are uptodate 
for 2023.11.12 14:00:00 GMT
and all components required to run assp are included.

The current location of this build is "C:\perl" - if you extract the build 
in to this location, there is nothing more to do (check the perl PATH env 
variable).
If you want or need to extract the build in to a different location - 
extract - and .
After extraction add the new perl PATH to your env.
Open a command prompt, cd in to the extracted folder and run 
"relocation.pl.bat" - DONE.

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23251

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23251:

- In IP-address lists it is now possible to define an ASN (Autonomous 
System Number) for an IP-address.
  The IP-address-range for the IP in the ASN is resolved and used instead 
of the defined IP-address.
  Even the ASN contains more than this IP-address range, only the range in 
which the defined IP-address is included is used.

GUI explanation:
...
It is also possible to let assp lookup the ASN (Autonomous System Number) 
for an IP-address (NOT the ASN number its self - like ASN:1234). The CIDR 
of the ASN will be used by assp.
To lookup the ASN for an IP-address, write ASN:x.x.x.x or ASN::bb::c
The ASN:ip-address notation can be also used for IP lists in a group 
definition.
...


added:

- ASSP_AFC.pm version 5.48 is now able to detect 'MHT MalDoc' (JPCERT/CC - 
https://blogs.jpcert.or.jp/en/2023/08/maldocinpdf.html) attacks.
  JPCERT/CC currently describs only the (one) case of handcrafted PDF 
files with MHT content.
  Simple tests have shown, that it is possible to include MHT's in many 
file types (for example images as well) and
  that MS-Office on Windows will open the MHT code, if the file extenson 
matches an MS-Office file extension - even the magic number of the file is 
not related to any MS-Office file.
  ASSP_AFC will detect MHT content in any attachment where such content is 
unexpected.
  Until now such files were only detected because of a possible missmatch 
between the file-magic-number (MIME-Type) and the file extension. 

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23225

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23225:

- because of a DNS-answer layout change of asn.routeviews.org, the ASN 
(Autonomous System Number) for the IP-address was no longer shown in the 
analyzer output
  and in the results of the "work with IP-addresses" dialog


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23214

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23214:


changed:

related to SF ticket(150) : DQS Service spamhaus.net is not working 
correctly

- The spamhaus offers a new service 'DQS Service spamhaus.net'. Since 
06/2023 they are blocking legacy (spamhaus.org) requests from different 
ISP's.
  As an alternative they are offering a free and a payed DQS Service. Both 
require registration at spamhaus and an accesskey has to be provided in 
each query.
  The accesskey has to be provided the following way : 
query-data.your-accesskey.spamhaus.url
  As a result, assp has treated the key as part of the provider host, and 
the key was included in the log as well as in reply codes for URIBL and 
DNSBL/RBL.
  To prevent this, the accesskey has to be surrounded by curly brakets in 
the RBL/URIBL-Service-Provider defintion
  like : $DATA$.{your-accesskey-here}.zen.dq.spamhaus.net - if the 
accesskey needs to be placed anywhere in the middle.
  Older definitions, which starts with the accesskey, like 
youraccesskey-here.$DATA$.provider.org will still work


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23198

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23198:

changed:

- an alias for the charset 'ISO-8859-8-I' is added to Encode, if the 
charset is not supported by the currently installed version of Encode

- the modus used to delay connections, if the connected IP matches the 
provided SPF-record, is changed
  until now, a hash was calculated over all IP-ranges in a SPF-record of a 
domain and the hash was used in tuplets and triplets
  this caused problems (delay loops), if other assp instances resolved the 
SPF-record from different DNS-servers or the SPF-record contained dynamic 
variables (the resulting hash has changed for any reason)

  from now on, the 'SPF:' tagged domain name is used for tuplets and 
triplets, if a SPF matching IP-address is found
  this way delaying is solved for a connection ( no matter if the 
SPF-record is anyhow different ) as long as the (re)connected IP-address 
matches the SPF-record of the domain

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23195

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23195:

- if a user,sender based (a privat) whitelist entry like 
user@local.domain,sender@other.domain was created using the 'work with 
lists' GUI dialog,
  an existing related personal-black-list entry was not removed

- if a mail contained an unknown charset definition (unknown to the perl 
module 'Encode') and the mail was stored in any corpus folder, the 
rebuildspamdb process
  died on an UTF8 exception - such mails/files may caused the same or 
similar exception in other assp operations
  recommendation: keep this perl module uptodate
  NOTICE: Encode is a perl core module. If you use a 
distribution based perl, the module will be normaly updated with the 
installation of a new perl version.
  Even there is a much more better version of this 
module available, most OS distributions are installing older versions of 
this module, if
  not the latest perl version is installed. It is 
safe to update this module to the latest available on any perl version - 
using cpan or cpanm.
  Even you use the latest perl version, there may 
be updates for the module Encode available!

  example: installed Encode version 3.17 (perl 5.36.0) and a mail contains 
the MIME-charset defintion ISO-8859-8-I (
https://en.wikipedia.org/wiki/ISO-8859-8-I)
   ISO-8859-8 is known to Encode, but it is not an alias for 
ISO-8859-8-I - so ISO-8859-8-I is unkown to this Encode version and this 
will cause the exception in assp
   Encode version 3.19 is aware of the MIME-charset ISO-8859-8-I 
by making ISO-8859-8 and alias for ISO-8859-8-I - but Encode version 3.19 
is only distributed with perl 5.38



Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23187

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23187:


- added support for Perl 5.38.0


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] perl 5.38.0

[Thomas Eckardt]

Hi all,

perl 5.38.0 was released some days ago.
I've finished compiling it for 64bit windows and added all components 
required by assp. If you want to try it out, it is available at 
https://sourceforge.net/projects/assp/files/ASSP%20V2%20multithreading/ASSP%20V2%20module%20installation/strawberry-perl-5.38.0.1-64bit-relocateable_4-assp.7z/download

openssl was upgraded to version 3.0.9

watch the file 'moduleLoadErrors.txt' for perl module load errors

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23184

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23184:

- if a backend-server (e.g. bad configured exim) offered the 
SMTP-extension PIPECONNECT and a client/server used the pipelining option, 
the connection failed after the DATA command

changed:

- the at least recommended version of the module Net::SSLeay is changed 
from 1.72 to 1.85

- the export extrem IP-List is now correctly sorted by IP

- it is now possible to find a match for an empty envelope recipient in 
'bombSenderRe'


added:

- it is now possible to set the OpenSSL security-level (default is 1) - 
using any of the following methodes
  - in lib/CorrectASSPcfg.pm:   $main::openssl_security_level = 1; # ( 
/1/2/3/4/5) used openssl security level - empty uses the libssl buildin 
value (default = 1)
  - SSLAdvancedServerConfigFile : SSL_CTX_set_security_level = 3;
  - SSLWEBConfigure: $parms->{'SSL_CTX_set_security_level'} = 3;
  - SSLSTATConfigure: $parms->{'SSL_CTX_set_security_level'} = 3;
  - SSLSMTPConfigure: $parms->{'SSL_CTX_set_security_level'} = 3;

- it is now possible to change the MIME-encoding of a mail before a 
DKIM-signature is added to it, if a specific MIME-header is found or the 
mail meets specific conditions
  to apply this behavior, define a sub DKIMconvCTE in 
lib/CorrectASSPcfg.pm
  if this sub is found by assp, it will be called by assp, providing the 
connection handle ($fh) and an array reference which can be modified 
inplace
  example:
 
sub DKIMconvCTE {
my ($fh, $convCTE) = @_;
my $this = $main::Con{$fh};
 
if (! $this->{mailfrom}) {

#  ct(1) ,   Regex for ct (2), target 
encoding (3)
push @{$convCTE}, 
['Content-Type','(?:text\/(?:ht|x)ml)','base64'];
#   push @{$convCTE}, ['.' ,  '.',   ''];
#   push ...
#   push ..
#   ...
}
} 

   in this example, if there is no envelope sender, assp will check the 
'Content-Type' (1) of the mail against the given regular expression (2) - 
and if
   a match is found, the MIME-encoding will be changed to 'base64' (3) 
before the DKIM-signature is added
   the reason for this implementation is: some NDR's, delivery 
notifications, report mails, OoO-mails (e.g. mails with clear text 
attached rfc822 mails or mail headers) may cause
   the DKIM-signature to become invalid after passing the final MTA (if 
the MTA reformated the mail for any reason)
   (there is currently only one known case)

Thomas

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23136

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23136:

- if openssl 3.0.0 or higher was installed, the installed openssl library 
version was not shown in the "Perl Modules" status page

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23130

[Thomas Eckardt]

Hi all,


fixed in assp 2.8.2 *SPAM-Eliminator* build 23130:

- the latest Net::DNS module version 1.38 throws a deprecation warning 
about the usage of the function rr->rdatastr (instead of using 
rr->rdstring) 
  this build uses rr->rdstring
  it is expected, that in any of the next versions of Net::DNS, the 
function 'rdatastr' will be removed from Net::DNS::RR, in this case ALL 
older assp builds will no longer work


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23116

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23116:

- the download URL's for the two- and third-level-tld files are changed to 
(the download was no longer woring using the old URL's)
https://www.surbl.org/static/two-level-tlds
https://www.surbl.org/static/three-level-tlds

- wrong crafted or looped SPF-Records may caused crashes while maintaining 
the SPF-Record-Cache (in tmpDB/files)

- Talos stopped redirecting http://senderbase.org - all related web-links 
in the GUI are changed

- the output of the analyzer for a wildcard (.*) good...-attachment-rule 
was wrong


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23089

[Thomas Eckardt]

Hi all,

fixed in assp 2.8.2 *SPAM-Eliminator* build 23089:

- if the bulkimport of the spamdb and hmmdb were disabled for 
MSSQL-Databases, the import failed because of the usage
  of wrong table names
 
- it was possible that preHeaderRe matched in the analyzer but not in the 
real mail processing, because of header line splitting


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.8.2 *SPAM-Eliminator* build 23072

[Thomas Eckardt]

Hi all

fixed in assp 2.8.2 *SPAM-Eliminator* build 23072:

**

*
* !!! ATTENTION !!! *
*
*this version of assp   *
*  requires at least*
*   *
*perl 5.12.0*
*   *
* it will NOT start on lower*
*  versions of perl *
*
* !!! ATTENTION !!! *
*

Because several required perl modules changed their minimum required perl 
version to 5.12, the minimum required perl version
to run assp is also changed to 5.12.0

**

- ASSP_AFC.pm is upgraded to version 5.46
  if symbols were used in a infected or not allowed attachment name, the 
removal of the attachment failed

- 'RemoteSupport' was not working if 'webAdminPort' was configured without 
a defined IP-address (like: SSL:5|5)

- several regular expression optimization were incorrect, because the perl 
module Regexp::Assemble used a fast but lazy
  algorythm per default - assp now forces Regexp::Assemble to use the 
right algorythm


- if code execution was configured in a weighted regular expression on 
perl 5.34 or 5.36, assp rejected the execution of
  the regex while searching for the configured weight value


- if a group name contained upper case characters and was use in 
'userAttach' , assp was unable resolve the group members,
  because it was searching for the (all) lower case group name



changed:

- the version changed from 2.6.x to 2.8.x and the code name is changed 
from *SPAM-Evaporator* to *SPAM-Eliminator*

- 'SepChar' now allows to define more than one character to support 
switching from one to another character, were it can
  be required to support both, the old and the new character, for some 
time


- 'removeDispositionNotification' can now also be used to remove any 
unwanted MIME-header by its tag

- 'URIBLCCTLDSRE' (the largest regular expression in the assp 
distribution) is now compiled in a separate process
  this prevents a stucking MainThread on some systems (under havy 
workload), while the regex is compiled in the MainThread


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP and chatgpt

[Thomas Eckardt]

1. a perl module OpenAI::API is already available - its not really nice 
because it uses Moo, but this can be changed
2. using OpenAI for spam detection would be very expensive - you are 
charged by tokens - reaching several million "Usage" and "Training" tokens 
per day is possible
...
9.  response time ?
...
X had this idea already - but - read.

OpenAI - pricing:

Model   TrainingUsage

Davinci $0.0300 / 1K tokens $0.1200 / 1K tokens

same costs for gpt-3.5-turbo

example from my today rebuild task:
Mar-07-23 04:07:00 Generating consolidated Hidden-Markov-Model database 
from 13,246,763 record model (~ 13.000.000)

one time training costs: ~ $US 390

having 10.000 mails per day, each with (only) 100 words -> 1.000.000 words 
or tokens (this is not really much, but an easy to calculate example)

= $US 120 per day

with one training per week (this is IMHO too less) the costs will be

390 * 52   = 20.280
120 * 365 = 43.800


~ 64.080 $US per year

with one training per day the costs would be $US 180.000

ASSP's HMM engine uses 600 words from each mail. OpenAI -> $US 720 -> 
overall costs per year $US 405.000


The idea its self is attractive - but who would use such an expensive 
service?


Let's say the ASSP_OpenAI Plugin is written and needs to be adjusted/fine 
tuned and tested for two weeks -> required per day are $US 1100, for two 
weeks $US ~15.000 !!!


Thomas






Von:"Graziano via Assp-test" 
An: assp-test@lists.sourceforge.net
Kopie:  "Graziano" 
Datum:  06.03.2023 09:55
Betreff:[Assp-test] ASSP and chatgpt



Hi
using a chatGPT API Perl module ( 
https://dev.to/davorg/writing-a-cpan-module-that-talks-to-chatgpt-gb5 ) 
with ASSP
to check every single email (chatgpt plus account should be required) 
could be an idea ?




___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP can't create socket on port 55555 for web admin.

[Thomas Eckardt]

>[init] IO::Socket::IP module version 0.41 installed and available

how looks the next line in the maillog.txt ?
I expect something like:  [init] this system binds universal IPv6 [::] to 
IPv6 and IPv4 (IPV6_V6ONLY is zero) ..

how is the config value for 'webAdminPort' set ?



Thomas





Von:"James Brown via Assp-test" 
An: assp-test@lists.sourceforge.net
Kopie:  "James Brown" 
Datum:  03.03.2023 23:12
Betreff:[Assp-test] ASSP can't create socket on port 5 for web 
admin.



I’ve moved to a new mail server, and can’t get some part of ASSP working.

Mar-03-23 22:48:47 [init] Listening for SMTP connections on [::]:25 , 
0.0.0.0:25
Mar-03-23 22:48:47 [init] Info: new SSL-Server-Context created for WEB 
connections
Mar-03-23 22:48:47 [init] Error: unable to create IPv6 socket to 
[::]:5 - IO::Socket::IP configuration failed
Mar-03-23 22:48:47 [init] Error: unable to create IPv4 socket to 
0.0.0.0:5 - IO::Socket::IP configuration failed
Mar-03-23 22:48:47 [init] Error: couldn't create server SSL-socket on port 
'5' -- maybe another service uses this listener or I'm not root 
(uid=0)? -- or a wrong IP address is defined? -- Address already in use
Mar-03-23 22:48:47 [init] Listening for stat HTTP connections on 
[::]:3 , 0.0.0.0:3
Mar-03-23 22:48:47 [init] Listening for SMTP relay connections on 
127.0.0.1:10025

Nothing is listening on port 5:

 % netstat -an | grep LISTEN
tcp4   0  0  *.31416*.*LISTEN  
 
tcp6   0  0  *.49217*.*LISTEN  
 
tcp4   0  0  *.49217*.*LISTEN  
 
tcp4   0  0  127.0.0.1.10025*.*LISTEN  
 
tcp4   0  0  *.3*.*LISTEN  
 
tcp6   0  0  *.3*.*LISTEN  
 
tcp4   0  0  *.25   *.*LISTEN  
 
tcp6   0  0  *.25   *.*LISTEN  
 
tcp46  0  0  *.3306 *.*LISTEN  
 
tcp4   0  0  127.0.0.1.33060*.*LISTEN  
 
tcp6   0  0  *.5000 *.*LISTEN  
 
tcp4   0  0  *.5000 *.*LISTEN  
 
tcp6   0  0  *.7000 *.*LISTEN  
 
tcp4   0  0  *.7000 *.*LISTEN  
 
tcp46  0  0  *.3283 *.*LISTEN  
 
tcp6   0  0  *.993  *.*LISTEN  
 
tcp4   0  0  *.993  *.*LISTEN  
 
tcp6   0  0  *.143  *.*LISTEN  
 
tcp4   0  0  *.143  *.*LISTEN  
 
tcp6   0  0  *.995  *.*LISTEN  
 
tcp4   0  0  *.995  *.*LISTEN  
 
tcp6   0  0  *.110  *.*LISTEN  
 
tcp4   0  0  *.110  *.*LISTEN  
 
tcp46  0  0  *.587  *.*LISTEN  
 
tcp46  0  0  *.465  *.*LISTEN  
 
tcp4   0  0  127.0.0.1.126  *.*LISTEN  
 
tcp4   0  0  127.0.0.1.10026*.*LISTEN  
 
tcp4   0  0  *.88   *.*LISTEN  
 
tcp6   0  0  *.88   *.*LISTEN  
 
tcp4   0  0  *.5900 *.*LISTEN  
 
tcp6   0  0  *.5900 *.*LISTEN  
 
tcp4   0  0  *.22   *.*LISTEN  
 
tcp6   0  0  *.22   *.*LISTEN 

I start ASSP with:

sudo /opt/homebrew/bin/perl /Applications/assp/assp.pl /Applications/assp

[startup] ASSP-professional version 2.6.8(23002) (Perl 5.036000) (on 
darwin) running on server:xxx
[init] IO::Socket::IP module version 0.41 installed and available

And in Activity Monitor I can see that Perl is running as root. 

What am I missing?

Thanks,

Jammes[Anhang "attu4spk.txt" gelöscht von Thomas Eckardt/eck] [Anhang 
"attyhzfq.txt" gelöscht von Thomas Eckardt/eck] 


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 23002

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 23002:

- ASSP_AFC.pm is upgraded to version 5.45

- on several linux version assp throws an error "Error: Schedule entry 
'0-59/10 * * * *' for MemoryUsageCheckSchedule is not valid"
  this should no longer happen



Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unable to create UDP Socket errors

[Thomas Eckardt]

found this in the assp dev forum

https://unix.stackexchange.com/questions/714901/dns-broken-when-using-ifupdown-and-systemd-resolved-after-upgrade-to-ubuntu-22-0?newreg=c4cc3942d61140858fd6815357211bba

it's related to ubuntu, but it may help to get an idea

Thomas




Von:"Mr. Courtney Creighton" 
An: assp-test@lists.sourceforge.net
Datum:  19.11.2022 04:02
Betreff:[Assp-test] Unable to create UDP Socket errors



Hi all,

I'm running ASSP on CENTOS 7, with Perl 5.30.1. I recently upgraded from 
b22137 to b22318.

Shortly afterwards, I'm seeing a new error from ASSP every few minutes:

[Worker_1] Error: DNS - unable to create any UDP socket to nameservers 
(1.1.1.10 111.111.111.22)

* Not my actual DNS servers (I tested with Google public DNS and get the 
same result)

As far as I can tell, mail is still working ok, and all ASSP features seem 
fine, even my DNSBL and URIBL are apparently still working.

This has been rather difficult to troubleshoot, but `ss -tulpn` doesn't 
show anything excessive or unexpected as far as UDP usage.

I did note that the CENTOS bind packages did see an update installed 
around this time as well. There was also a new kernel package installed. 
Perhaps there was a change in the behavior of that software that ASSP is 
having trouble dealing with?

I'm looking for ideas for continuing to troubleshoot this, and also 
wondering if anyone else is seeing anything similar.

-C



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22326

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22326:

- several interal statistics were unexpected cleared at startup, if 
'useDB4IntCache' was not set

- if a mail with the empty envelope recipient was received and the sending 
domain (from: or sender:) provided a DMARC record
  the DMARC check failed. The DMARC check is now skipped for such mails.

changed:

- the top ten statistc URL is expanded with a count parameter

/top10stats?count=10
  the number specifies the amount of the shown entries
  a negative value will show the less blocked entries
 
  the hidden variable $toptencount = 10; will set the default value
 
added:

- the 'work with addresses and domains' GUI-dialog now allow to request a 
blockreport in the web session.
 
  ... blockreport: or block: or report: or blr: or bl: in front of an 
address or in the reason field will generate
  a blockreport in a new browser window - a trailing number and/or regex 
specifys the days and filter

  NOTICE: these blockreports are internaly executed with EmailAdmin 
equivalent permission (no restriction)

- the 'work with IP-addresses' GUI-dialog supports the same blockreport 
option like the work with addresses and domains' GUI-dialog
  this option is and will be keeped undocumented

- admin users action permissions are enhanced with the 'action parameter' 
"webblockreport"
  so even an admin user is allowed to work with the addresses dialogs, it 
is possible to disallow the blockreport generation in the browser

- the 'work with IP-addresses' GUI-dialog is now able resolve SPF-records 
- simply write SPF:domain.org in to the input field
 

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unable to create UDP Socket errors

[Thomas Eckardt]

>even my DNSBL and URIBL are apparently still working

No - only the cache is used. If there is no DNS-Server available, no 
DNS-queries will be done by assp - otherwise the workers would stuck in 
DNS-queries.

1.1.1.10 111.111.111.22 - IMHO both are no DNS-servers - I get no answer 
from there (and so assp)

Configure your system to have a working DNS-setup (using bind and/or 
external DNS-servers). Check this using dig or any other useful tool.
Check the permission of the assp user to be able to query the configured 
DNS-servers.
Check the ASSP DNS-setup to use the right DNS-servers (system provided or 
defined by an admin)

There are no changes made in ASSP-DNS-engine.

>I tested with Google public DNS and get the same result

I don't recommend to use any public DNS-server, because it is possible 
that they are unable to query DNSBL,URIBL,Senderbase  . Some providers 
may block them, because of  havy usage (query count -> limit reached).

Thomas







Von:"Mr. Courtney Creighton" 
An: assp-test@lists.sourceforge.net
Datum:  19.11.2022 04:02
Betreff:[Assp-test] Unable to create UDP Socket errors



Hi all,

I'm running ASSP on CENTOS 7, with Perl 5.30.1. I recently upgraded from 
b22137 to b22318.

Shortly afterwards, I'm seeing a new error from ASSP every few minutes:

[Worker_1] Error: DNS - unable to create any UDP socket to nameservers 
(1.1.1.10 111.111.111.22)

* Not my actual DNS servers (I tested with Google public DNS and get the 
same result)

As far as I can tell, mail is still working ok, and all ASSP features seem 
fine, even my DNSBL and URIBL are apparently still working.

This has been rather difficult to troubleshoot, but `ss -tulpn` doesn't 
show anything excessive or unexpected as far as UDP usage.

I did note that the CENTOS bind packages did see an update installed 
around this time as well. There was also a new kernel package installed. 
Perhaps there was a change in the behavior of that software that ASSP is 
having trouble dealing with?

I'm looking for ideas for continuing to troubleshoot this, and also 
wondering if anyone else is seeing anything similar.

-C
[Anhang "attmrm6l.txt" gelöscht von Thomas Eckardt/eck] [Anhang 
"att9qtoe.txt" gelöscht von Thomas Eckardt/eck] 


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22318

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22318:

- some spam messages were not forwarded even 'ccSpamAlways' was configured
  NOTICE: if 'ccSpamAlways' is configured and 'SpamVirusLog' is not set to 
'quarantine' (even it is set to 'no collect' !!), the virus mails
  will be sent to the configured email address - GUI: ..."Copy 
Spam to these recipients regardless of collection mode."...
  this not a changed behavior - but keep this in mind!

- it was very hard for an admin to find out why a specific spam mail was 
not forwarded to sendAllSpam
  if SessionLog is set to verbose, the reasons are now logged to the 
maillog.txt

- some connections of already finished spam messages were running in to a 
SMTPTimeout - resulting in a very high count of SocketCalls
  for the connection

- using postfix as local backend server, it was possible that assp sent 
orphaned data to postfix, which caused postfix to respond with
  '502 5.x.x syntax error' replies
  this was happen, if assp ignored the socket read-error 'EAGAIN - 
resource temporary not available' in some very special cases

- mails from gmail.com or googlemail.com users who sent automatic 
generated mails to your assp, were rejected/scored by DoNoFromSelect
 
  a good example for this case are google-calendar invitations: the 
envelope sender is ...@calendar-server.bounces.google.com
  the from address is the right ..@gmail.com or ...@googlemail.com user 
address - the sender header addess is a ...@google.com address
  the missmatch of the domain names caused assp to score and/or reject the 
mail
  DoNoFromSelect now processes gmail.com, googlemail.com and google.com as 
equal domains (internaly all domains are set to gmail.com for the 
  DoNoFrom check)
 
- the statistic was not counted, if a DKIMidentityWLmatch or 
DKIMidentityNPmatch was found
 

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22313

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22313:

- if any of DKIMWLAddresses or DKIMNPAddresses was used and a mail 
contained more than one DKIM signature (eg. for different identities)
  only the first (header - top to bottom) valid DKIM identity was checked 
against both parameters
  now all found valid DKIM identities are checked agains DKIMWLAddresses 
and/or DKIMNPAddresses.


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22310

[Thomas . Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22310:

- on non english linux installations some times the connection retry was 
not working


changed:

- in 'ccSpamInDomain' it is now possible to use the USERNAME literal
  The literal USERNAME is replaced by the user part of the recipient. 



- the minimum version of the module 'Schedule::Cron' is changed to 1.03

- if the "NWLI" directive is used in weighted regular expressions, the 
skipping reason is now shown in the maillog for real mail processing,
  (not in the analyzer)


added:

- URL's encoded in base64 using the ".atob" HTML statement are now 
detected, decoded and checked in URIBL

- the ASSP_AFC plugin (5.44) is able to detect native integrated Base64 
encoded as well as javascript code in "text/html" and "image/svg+xml"
  attachments, if "exe-bin" is configured to be detected.
  The native Base64 parts are decoded and analyzed like every other 
attachment.


- the ASSP_AFC plugin (5.44) now also supports the following two blocking 
exceptions
 :JSHTML - HTML file with JavaScript or mouse driven HTML events (like: 
onmouseover, onmouseout, onfocus, onblure ...)
 :JSSVG - SVG images with JavaScript or mouse driven HTML events (like: 
onmouseover, onmouseout, onfocus, onblure ...)


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Allowing certain javascript in HTML to pass

[Thomas Eckardt]

>it appears that it is showing the sha256 of the .html file 

yes, the javascript is in the html file - - the fault is "javascript used 
in html", not the base64 encoded javascript part

>That means that the sha256 that shows in the log is different each time 
and can't be use for the exception.

yes, like expected , if the content is change the hash will change

>vs just the portion of javascript that is being detected.

this is an image/gif - what should be bad with this file ???

changelog : ... The native Base64 parts are decoded and analyzed like 
every other attachment. 

>I know I don't want to use a UserAttach exception for the sending email 
address,

no luck

>Is there a way that I can allow the javascript code (which is constant 
and in an ever changing html file) through using sha256 or another method, 
but still block all other html files with javascript embedded?

define the :CSC exception for sender and recipient
and
write your own code to detect javascript and call this code using 
$ASSP_AFC::checkExeExternal or $ASSP_AFC::checkExeExternalForce (both 
documented in the ASSP_AFC code and called in sub isAnExe)

or something like : npRe in combination with BlockNPExes

>quick question - before I dig deeper, did the previous AFC plugin not 
block javascript in HTML at all?  

only for some special cases

>before I dig deeper

I don't want to - but possibly other list members.

Thomas




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  31.10.2022 19:20
Betreff:Re: [Assp-test] Allowing certain javascript in HTML to 
pass



quick question - before I dig deeper, did the previous AFC plugin not 
block javascript in HTML at all?  

On Mon, Oct 31, 2022 at 10:21 AM K Post  wrote:
The new AFC is blocking a nightly report that comes in HTML format with 
javascript in it -- as I would expect, but before his new AFC, they were 
erroneously slipping through.

I don't know why these reports weren't being blocked before, it's basic 
HTML with a short block of javascript at the end.  Of note, the javascript 
starts like this and has a base64 image in its code - something that the 
new AFC addresses:

[Assp-test] ASSP_AFC 5.42 available at SVN

[Thomas Eckardt]

Hi all,

ASSP_AFC.pm 5.42 is released on the SVN repo.

The last days have shown an extensive usage of very smart attacks, which 
are based on native integrated code in SVG images and HTML pages.
This release is able to detect native integrated Base64 encoded as well as 
javascript code in "text/html" and "image/svg+xml" attachments, if 
"exe-bin" is configured to be detected.
The native Base64 parts are decoded and analyzed like every other 
attachment.

Thomas

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22293

[Thomas . Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22293:

- if BlockRepForwHost was configured and any of EmailBlockReportDomain, 
EmailBlockReport or BlockRepForwHost was changed, queued forwarding 
requests failed
  in case any of these vales is changed, the BlockReport-Forwarding queue 
is cleared now

changed:

- If delaying is enabled and DelayUseNetblocks is set to 'On' and the perl 
module 'NetAddr::IP::Lite' is installed and enabled, assp resolves the SPF 
record for the domain used in the 
  envelope sender address. If the connected IP-address is valid (in terms 
of the SPF-record) all IP-adresses and ranges from the SPF-record are 
hashed and the hash is used
  (instead of the the connected IP-address) for delaying.
  This new behavior makes sure, that another valid IP-address, that tries 
to deliver the same mail after the first valid address was delayed, will 
be not delayed.
  If no SPF-record is available, the SPF-record is invalid or the 
connected IP-address is not valid - the connected IP-address will be used 
for delaying (old behavior).
  Because it is expected that a system will try to connect again after it 
was delayed, the SPF-records are cached inside assp.
  The records are stored in assp/tmpDB/files/SPFRecCache.sav and refreshed 
by the MaintThread, if any TTL gets outdated. 

 

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22280

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22280:

- after an upgrade of the perl module Schedule::Cron to version 1.03 the 
assp scheduler was no longer working

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22279

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22279:

- griplist uploads and downloads were no longer working, because the 
sourceforge http server no longer accepts plain http transfer
  NOTICE: ALL older versions of assp will fail to upload to the griplist 
server!

- statistc uploads were no longer working, because assp used http - now 
https is used by assp
  NOTICE: ALL older versions of assp will fail to upload to the stats 
server!

- after talking to the sourceforge support team, an exception is made for 
assp the assp project at the sourgeforge web-server to accept plain https 
connections for a short
  range of time. It is strongly recommended to upgrade your assp 
installation to the latest version!
  If the exception is canceled at any time, all older versions of assp 
will be unable to use griplist and stats.
 
- in some cases assp failed to decode empty MIME-encoded content 
(=?UTF-8?Q??=) correctly

changed:

- assp generates now 2048 bit RSA keys (instead of 1024 bit) if no 
SSL-keys/certs are found at startup

- all sourgeforge.net related URL's are changed to use https instead of 
http


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22252

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22252:

- if line continuation '\' was used in a regular expression file, the 
regex was no longer working like expected

- if line continuation '\' was used in a regular expression file, the 
analyzer has'nt shown the matching file and line

Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Line Continuation in config files

[Thomas Eckardt]

The line continuation is supported in every file.

How ever, I've done a positioning mistake in the used regex

current: s/\\(?
An: "ASSP development mailing list" 
Datum:  08.09.2022 21:14
Betreff:[Assp-test] Line Continuation in config files



Back in November 2011, I saw:
fixed in assp 2.6.6 *SPAM-Evaporator* build 21317:
- files used in configuration parameters are now 
supporting line continuation by adding a backslash '\' at the end of 
a line

I'm just trying this now, but it doesn't seem to be working.   

When you say "files used in configuration parameters" do you only mean 
files that use the # include syntax, or should this work for things like 
bomgHeaderRe where we have file:/files/bombHeaderRe.txt in the GUI?

If line continuation is only supported in # include files, could you 
extend the functionality to work in files that are directly referenced by 
the GUI (file:files/ext.txt)?

Here's what I'm experiencing:

In my bombHeaderRe file, which is directly referenced in the gui as 
file:files/bomgHeaderRe.txt, I have

~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))(?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10

all on a single line.  That scores -10 to any message where to: 
senderdom...@ourdomain.com and from: anything@*.SenderDomain.com appear in 
the header, in any order.  (thanks for all the help building this, it's 
been incredibly beneficial)

I just tried splitting that regex into multiple lines by adding a \ at the 
end of lines.   I'm not putting a space

~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))\
(?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n\
(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10

Saving the file does not trigger an error in the GUI, but the analyze GUI 
no longer shows matches for the same email that does when the regex is on 
a single line.  


Thanks![Anhang "att948xp.txt" gelöscht von Thomas Eckardt/eck] [Anhang 
"attmyh5k.txt" gelöscht von Thomas Eckardt/eck] 


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22251

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22251:

- some unexpected log lines about unresolveable IP-addresses were shown by 
assp (since build 22200)

- the attachment 'NoCheckIf' rule was not working, if the SPF-check or the 
DKIM-check was skipped because of any condition (noprocessing, 
whitelisting, ...)



changed:

- until know soft-hyphens ( 0xAD , U+00AD) were replaced by normal 
hyphens (- , 0x2D , U+002D) for all text related tests and operations in 
assp -
  from now on, soft-hyphens are removed from all text parts


Thomas
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] soft hyphen fooling Bayesian analysis

[Thomas Eckardt]

If unicode normalization NFKC does'nt fulfill your requirement, you may 
enable 'DoTransliterate' - by accepting some performance penalties.

The "Unicode Technical Standard #39" http://www.unicode.org/reports/tr39/ 
will give you some more information and 
https://www.unicode.org/Public/security/revision-05/intentional.txt shows 
a nice table for cyrillic and greek.
If someone expects an ASCII mail, those translations may somehow help. But 
in all other cases (100% cyrillic/greek/), such a character 
replacement is contra-productive (for example: not all cyrillic letters 
have a valid latin replacement).

> potentially treat look-alike characters as the latin character for 
bayesian purposes

The HMM and Bayesian engines are using heuristic mechanism. Trying to 
treat single characters as latin (or anything else) will not worth the 
effort. Over a short periode of time, both engines will have learned also 
obscured words (word combinations).


Thomas




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  06.09.2022 21:31
Betreff:Re: [Assp-test] soft hyphen fooling Bayesian analysis



Eager to see what you come up with in terms of ignoring the soft hyphen.  
 

 Your <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>> regex is clear, and I 
understand using that for scoring purposes, but I'm looking for a way to 
potentially treat look-alike characters as the latin character for 
bayesian purposes and/or to catch commonly obscured words (like 
GeekSquad).  Is it okay if I reply further in my  August 1 post here to 
keep that in the same thread?

On Tue, Sep 6, 2022 at 2:06 PM Thomas Eckardt  
wrote:
>HTML::strip 

html parsing to get text parts has nothing to do with html de(en)coding 


>iso-8559-1 
ASSP processes all content as UTF-8 


> 
ASSP is aware about this - and replaces soft-hyphens with hard-hyphens - 
and multiple concurrent hard-hyphens with a single one 
How ever - the option to remove the soft-hyphens instead, sounds somehow 
better. Tests are still running. 

>My thinking is that if it doesn't display. 
ASSP does'nt know if something displayed or not (and will never know it) 


>I suspect that other characters will be abused in the same way 
 as well as several BIG5, numerical and other unicode characters are 
already special handled by assp. Other CTL-chars are ignored by assp. 
Everything is converted to UTF8, unicode normalized (including grapheme 
clusters), stemmed and simplyfied. 


>This kind of obfuscation goes hand in hand with my previous questions 
about considering some non-Latin characters that look like Latin 
characters as those Latin alphabet characters.  

With some unicode knowledge, some help from the analyzer and some regex 
knowledge - such things are easy to find 
for example : <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>> 
finds a sequence where cyrillic (a p b ) are used in words - commonly 
used by spammers 

Thomas 



Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:06.09.2022 16:16 
Betreff:[Assp-test] soft hyphen fooling Bayesian analysis 




Is there a way to improve the way that ASSP parses certain special, 
non-printing, characters?  I'm having trouble with spam emails that have 
their body heavily obfuscated with "soft hyphens" slipping through.  They 
all seem to have multipart bodies, first with an iso-8559-1 text part with 
=AD interterspersed in words and then an html part with  all over the 
place.  These are the "soft hyphen," a hyphen that only prints if it is 
needed to break the word to the next line.  It's clever.  The user doesn't 
see the character, but ASSP thinks it's a word boundary.   

The part first part 
Content-Type: text/plain; charset="iso-8859-1" 
Content-Transfer-Encoding: quoted-printable 
will be plain text, and have have spammy words with =AD inserted in the 
middle of them, for example, "This is a sentence with spammy phrase." 
could be written something like  
This is a sentence with sp=ADammy p=ADhr=ADase. 

The next mime part is the html, which does the same thing, but uses  
(html for soft hyphen) mid-word.  So, something like: 
This is a sentence with spammy phrase in it 

The whole body of the message is filled with these soft hyphens anywhere 
that there's spammy words/phrases, and in many cases, there are soft 
hyphens every couple of letters across the entire body.  When I do an 
analysis, it appears that the soft hyphen tricks ASSP into thinking that 
each part of the word is a separate word, so for spammy 
phrase, it thinks the words are 
sp ammy p hr ase 

I am using HTML::strip.  Would TreeBuilder work better?  I'm concerned 
about performance there. 

Is there a way (and is it a good idea) to somehow instruct ASSP to treat 
certain html special characters as 

Re: [Assp-test] soft hyphen fooling Bayesian analysis

[Thomas Eckardt]

>HTML::strip

html parsing to get text parts has nothing to do with html de(en)coding


>iso-8559-1
ASSP processes all content as UTF-8


>
ASSP is aware about this - and replaces soft-hyphens with hard-hyphens - 
and multiple concurrent hard-hyphens with a single one
How ever - the option to remove the soft-hyphens instead, sounds somehow 
better. Tests are still running.

>My thinking is that if it doesn't display.
ASSP does'nt know if something displayed or not (and will never know it)


>I suspect that other characters will be abused in the same way
 as well as several BIG5, numerical and other unicode characters are 
already special handled by assp. Other CTL-chars are ignored by assp.
Everything is converted to UTF8, unicode normalized (including grapheme 
clusters), stemmed and simplyfied.


>This kind of obfuscation goes hand in hand with my previous questions 
about considering some non-Latin characters that look like Latin 
characters as those Latin alphabet characters. 

With some unicode knowledge, some help from the analyzer and some regex 
knowledge - such things are easy to find
for example : <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>>
finds a sequence where cyrillic (a p b ) are used in words - commonly 
used by spammers

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  06.09.2022 16:16
Betreff:[Assp-test] soft hyphen fooling Bayesian analysis




Is there a way to improve the way that ASSP parses certain special, 
non-printing, characters?  I'm having trouble with spam emails that have 
their body heavily obfuscated with "soft hyphens" slipping through.  They 
all seem to have multipart bodies, first with an iso-8559-1 text part with 
=AD interterspersed in words and then an html part with  all over the 
place.  These are the "soft hyphen," a hyphen that only prints if it is 
needed to break the word to the next line.  It's clever.  The user doesn't 
see the character, but ASSP thinks it's a word boundary.  

The part first part
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
will be plain text, and have have spammy words with =AD inserted in the 
middle of them, for example, "This is a sentence with spammy phrase." 
could be written something like 
This is a sentence with sp=ADammy p=ADhr=ADase.

The next mime part is the html, which does the same thing, but uses  
(html for soft hyphen) mid-word.  So, something like:
This is a sentence with spammy phrase in it

The whole body of the message is filled with these soft hyphens anywhere 
that there's spammy words/phrases, and in many cases, there are soft 
hyphens every couple of letters across the entire body.  When I do an 
analysis, it appears that the soft hyphen tricks ASSP into thinking that 
each part of the word is a separate word, so for spammy 
phrase, it thinks the words are
sp ammy p hr ase

I am using HTML::strip.  Would TreeBuilder work better?  I'm concerned 
about performance there.

Is there a way (and is it a good idea) to somehow instruct ASSP to treat 
certain html special characters as ones to ignore, and others to be 
treated as a word separator?  My thinking is that if it doesn't display, 
then it should be ignored when doing bayesian / HMM evaluation.

https://cs.stanford.edu/people/miles/iso8859.html has a bunch of Control 
Characters and Special Characters that don't print - or in the case of the 
soft hyphen, only print when the contained word is at the end of a line.  
I suspect that other characters will be abused in the same way.

This kind of obfuscation goes hand in hand with my previous questions 
about considering some non-Latin characters that look like Latin 
characters as those Latin alphabet characters. 

Thanks





[Anhang "attz351u.txt" gelöscht von Thomas Eckardt/eck] [Anhang 
"att8gq15.txt" gelöscht von Thomas Eckardt/eck] 


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Warnings for unable to resolve IP

[Thomas Eckardt]

Yes, build 22200 contains a fix for the FQDN (relay host) detection. A log 
line was added to the changed 'resolve-code' - but I've overlooked, that 
this code is also used as fallback in the MX/MXA check, in case nothing is 
resolved using the configured DNS.
The next version will prevent these loglines, if the code is called from 
the MX/MXA check.

Thank you for reporting.

Thomas


DISCLAIMER: 
*** 
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed. 
This email was multiple times scanned for viruses. There should be no 
known virus in this email! 
*** 




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  09.08.2022 17:55
Betreff:[Assp-test] Warnings for unable to resolve IP



Has anything changed with recent dev versions of ASSP in terms of 
warnings in the logs for being unable to lookup an IP?

For example:
warning: can't resolve the IP-address for the destination 
stonewallkitchenvip.com using the configured DNS-servers

Nothing's wrong with assp, stonewallkitchenvip.com doesn't exist, but I'm 
getting a lot of warnings in the logs for misconfigured (or non existent) 
domains since updating to the latest ASSP and I don't think I've had that 
before.  I can stop the alert emails based on the warning, I just wanted 
to see if these are new / different warnings that ASSP is doing now or if 
there's been an uptick in bulk sending services doing things wrong.

Thanks
[Anhang "attm5aih.txt" gelöscht von Thomas Eckardt/eck] [Anhang 
"att78ilr.txt" gelöscht von Thomas Eckardt/eck] 


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22209

[Thomas . Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22209:

- If an EmailAdmin requested a resend of a blocked mail (for another 
person) and the blocked mail was originally sent to multiple envelope 
recipients,
  the mail was only resent to the first original envelope recipient, 
because only the first found 'X-Assp-Intended-For:' header was parsed
  for recipient addresses by assp.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] ASSP_AFC version 5.41

[Thomas Eckardt]

I released ASSP_AFC.pm version 5.41 at SVN.

If a mail was blocked because of a bad attachment and an admin requested a 
resend without using the 'noscan' tag or without defining the 
'X-Assp-ForceResend:' header,
assp has possibly moved and renamed the original file (most times in to 
'discarded') - the trailing number in the file name was changed.
If at a later time anyone requested a resend for this file again (e.g. 
multiple original recipients were blocked) - assp was unable to find the 
file , which caused the resend to fail.



Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] ASSP_AFC version 5.40

[Thomas Eckardt]

I released ASSP_AFC.pm version 5.40 at SVN.
It fixes a problem were the attachment parser was running in to a 
processing timeout, if an extracted attachment was a corrupt  .eml file.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[Thomas Eckardt]

>I'd be shocked if more than 10 people out of the couple thousand 
remaining ASSP users (??) has an CorrectASSPcfg functions in a regex.

May be some perl experts, who wants to extend the regex features in assp. 
But IMHO this is not required and I never saw a related question. Even I 
don't do such things in a regex. And never the less: 'AllowCodeInRegex' is 
disabled per default for security reasons!
https://perldoc.perl.org/perlretut#A-bit-of-magic:-executing-Perl-code-in-a-regular-expression
The goal of the lib/CorrectASSPcfg.pm is to set configuration parameters, 
to implement custom code and to provide some callbacks ( I already spoke 
about)  for assp.pl.

Perl experts will know more ways how to implement custom code in to assp.- 
without modifying the original perl script.

>but wasn't aware of the *FAIL syntax.

https://perldoc.perl.org/perlretut#Backtracking-control-verbs

It's a good idea to have https://perldoc.perl.org/perlretut in mind (or at 
hands) if you want to do perl regular expressions!


>This is TERRIFIC. Terrific, terrific, terrific.

Possibly yes. BUT SURELY it can be dangerous! You have to enable 
'$AllowCodeInRegex' , which applies to all custom regular expressions. 
If someone (a hacker) is smart enough to let assp fill logs, headers or 
bodys with bad content - it can be possible that assp executes malicious 
code (for example like log4j ). Not only at the time a mail is processed, 
this can be happen at any later time (analyze, blockreport, notify, 
ham/spam report, rebuildspamdb, ...)
I'm sure this is not possible if 'AllowCodeInRegex' is disabled!

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  16.07.2022 13:32
Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on 
DMARC rejects



This is TERRIFIC. Terrific, terrific, terrific.
I've done a lot of regex work in my days mostly in php, vb, and linux 
batch scripts, but wasn't aware of the *FAIL syntax.  I was thinking a 
return of 1 or 0, not no return.  That makes much more sense.

And thanks for the continued examples.  I'd be shocked if more than 10 
people out of the couple thousand remaining ASSP users (??) has an 
CorrectASSPcfg functions in a regex.

Three small charities that used to use my charity's ASSP installation for 
email filtering have gone direct to Office365 and removed ASSP from 
the equation in the last 2 years.  I'm still in touch with their staff.  
While the like 365's features, they can't stand the spam filtering. You've 
spoiled them with never having to sort through a junk folder!!




On Fri, Jul 15, 2022 at 6:37 AM Thomas Eckardt  wrote:
Ken - learn perl regular expressions! 

>Why is the *FAIL bit in your example of  

- the (*FAIL) or (*F) statement makes a regex fail, even a match was 
found. 

Because the return value (e.g. setting $_) from a code execution in a perl 
regex does not modify the 'match found/no match found' flag of the regex. 
But the $_ can be used in a conditional regex to tell the regex engine 
what to do (in which case). 
There are multiple ways to do it. And possibly there are better ways - but 
this one I found nice. 

(\@.+\.docusign\.net|next domain|next 
domain|...|...)(?(?{::myWantedDKIMCheck($fh,$+)})|(*FAIL)) 

explanation: 
( the matching strings/domains, match captured in $+) # if failed, the 
next parts of the regex are ignored and the regex fails - if matched 
'match found' is set by the regex engine 
 (?( # start of a 
conditional (yes|no) regex (?(cond)yes|no) 
?{  # start of the 
code to be executed ?{code} 
  
::myWantedDKIMCheck # call this sub 
  
  ($fh,$+) # provide the filehandle and the last match 
result to the sub 
  
  }) # end of the code and the condition - the 
return value of the sub is the conditional result 
  
| # the 'yes' part (before the pipe [empty]) - 
if the code returned 1, nothing is to do, regex keeps 'match found' 
  
 (*FAIL) # the 'no' part (after the pipe) - if 
the sub returned 0 or undef make the regex fail (no match found) 
  
) # end of the conditional regex 


> seems to return if there's no DKIM  (return unless $this->{isDKIM};) 
wouldn't that not match the regex, so the 60 score wouldn't be applied? 

right! 
returns undef in case there is no DKIM-signature found - which makes the 
regex 

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22200

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22200:

- if 'runAsUser' was used on nix systems, it was possible that the file 
assp/tmpDB/files/SPFRecCache.sav was saved using a wrong owner (root) -
  which may be caused a permission error at the next start of assp
 
- if setFilePermOnStart was set - the permission for several files and 
folders were set twice at the next start

- if a FQDN was defined for relayHost, which resolved to multiple 
IP-addresses - and relayAuthUser/relayAuthPass (or AUTHrelayTable) was 
configured
  it was possible, that assp has'nt detected the connected host for 
authentication and skipped the AUTH command (outgoing mails were rejected 
by the relay host)


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[Thomas Eckardt]

 and promise!  I hope it doesn't sound pathetic, but that's 
exciting!

A couple more questions if (or when) you have the time and energy for 
this:


Why is the *FAIL bit in your example of 
~<<<(\@.+\.docusign\.net|next domain|next 
domain|...|...)(?(?{::myWantedDKIMCheck($fh,$+)})|(*FAIL))>>>~=>60
I'm concerned about only matching (docusign.\net|otherdomains)(.*FAIL)
I'm assuming you intended to have a period before the * Won't that 
match any header like:

from: whate...@docusign.net
subject: failure to complete submission

 
The function example:
sub CorrectASSPcfg::myWantedDKIMCheck {
my ($fh,$match) = @_;
my $this = ($fh && exists($main::Con{$fh})) ? $main::Con{$fh} :'';
return unless $this;
return unless $this->{isDKIM};
return 1 if $this->{dkimresult} eq 'pass';
return 1 if $this->{dkimverified} eq 'verified-OK';
my $re = qr/
domain1\.org
  | \.domain2\.org
  | user[^@]+?\@.+?\.domain3\.org  
/xis;
return ($match !~ /$re/);
}
 seems to return if there's no DKIM  (return unless $this->{isDKIM};) 
wouldn't that not match the regex, so the 60 score wouldn't be applied?  
Part of my goal is to require DKIM signature for certain domain names, not 
only requiring valid DKIM.

Thanks again
Ken




On Sat, Jul 9, 2022 at 4:53 AM Thomas Eckardt  
wrote:
I'm sorry but the example 

\@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60 

should be better 

~<<<(\@.+\.docusign\.net|next domain|next 
domain|...|...)(?(?{::myWantedDKIMCheck($fh,$+)})|(*FAIL))>>>~=>60



The first one does not fail if CorrectASSPcfg::myWantedDKIMCheck returns 
0, The second provides $fh and the matched string to the sub 
CorrectASSPcfg::myWantedDKIMCheck.

short example for CorrectASSPcfg::myWantedDKIMCheck 

sub CorrectASSPcfg::myWantedDKIMCheck { 
my ($fh,$match) = @_; 
my $this = ($fh && exists($main::Con{$fh})) ? $main::Con{$fh} :''; 
return unless $this; 
return unless $this->{isDKIM}; 
return 1 if $this->{dkimresult} eq 'pass'; 
return 1 if $this->{dkimverified} eq 'verified-OK'; 
my $re = qr/ 
domain1\.org 
  | \.domain2\.org 
  | user[^@]+?\@.+?\.domain3\.org   
/xis; 
return ($match !~ /$re/); 
} 


Thomas 


Von:"Thomas Eckardt"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:08.07.2022 16:53 
Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on 
DMARC rejects 



If such a feature would be implemented, it will result in havy coding. 

>I want to outright block any message from @*.docusign.net that isn't 
signed or that has an invalid signature.  I don't care if it's from a 
whitelisted email address, from an IP that's in the SPF record, and with a 
message body that is 100% great.

You want not only to make the test domain based strict, you want to ignore 
flags like 'whitelisted' - that's ok - but if I would start to allow any 
flag exceptions, other users may want to have other or more flag 
exception. 

- noprocessing 
- whitelisted 
- spamlover 
- domain based scoring values 
- SMIME/PGP signed 
. 

Yes, a great feature - but who would need it? 

The best way would be to create a level 1 plugin for this purpose. There 
you can check the dkim result, flags, ip's ... what ever you want - and 
based on your logic, you can block or pass the mail. 

But knowing (and thinking like) assp, will open other ways (solution 
workarounds) - for example. 

we assume the DKIM check is set to scoring - and the scoring value is 20 
points below the penalty limit. 

If a DKIM signature is invalid - assp scores. 
If the domain has ever sent a mail with a valid DKIM signature before (a 
DKIMCache entry is found), assp scores for DKIM if a mail  without a DKIM 
signature from this domain is received. 

Now, if there was not added any other score (the mail is 100% ok, except 
DKIM) the mail will pass because the penalty limit is not reached. But you 
want to block the mail if the sender matches @*.docusign.net 

sender??? ... matches???... - assp has weighted regular expressions - 
like: bombSenderRe - where you can add or remove scoring points 
if you set there 
\@.+\.docusign\.net=>20 

all mails from those domains will get a penalty of 20 points, which is 
harmless if there is everything else ok with the mail 
if dkim fails, the penalty limit will be reached and the mail will be 
blocked 
this can be finetuned using :>NWLI 

You are also able to implement code in to the regex (for example to check 
for the DKIM result). This is much less complicated than writing a plugin. 

\@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60 
"score with 60 if the sender matches and the sub 
CorrectASSPcfg::myWantedDKIMCheck returned 1" 

Both examples should only show, that there are more ways to get wanted 
results in assp. If someone solve

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[Thomas Eckardt]

I'm sorry but the example

\@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60

should be better

~<<<(\@.+\.docusign\.net|next domain|next 
domain|...|...)(?(?{::myWantedDKIMCheck($fh,$+)})|(*FAIL))>>>~=>60


The first one does not fail if CorrectASSPcfg::myWantedDKIMCheck returns 
0, The second provides $fh and the matched string to the sub 
CorrectASSPcfg::myWantedDKIMCheck.

short example for CorrectASSPcfg::myWantedDKIMCheck

sub CorrectASSPcfg::myWantedDKIMCheck {
my ($fh,$match) = @_;
my $this = ($fh && exists($main::Con{$fh})) ? $main::Con{$fh} :'';
return unless $this;
return unless $this->{isDKIM};
return 1 if $this->{dkimresult} eq 'pass';
return 1 if $this->{dkimverified} eq 'verified-OK'; 
my $re = qr/
domain1\.org
  | \.domain2\.org
  | user[^@]+?\@.+?\.domain3\.org 
/xis;
return ($match !~ /$re/);
}


Thomas


Von:"Thomas Eckardt" 
An: "ASSP development mailing list" 
Datum:  08.07.2022 16:53
Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on 
DMARC rejects



If such a feature would be implemented, it will result in havy coding. 

>I want to outright block any message from @*.docusign.net that isn't 
signed or that has an invalid signature.  I don't care if it's from a 
whitelisted email address, from an IP that's in the SPF record, and with a 
message body that is 100% great.

You want not only to make the test domain based strict, you want to ignore 
flags like 'whitelisted' - that's ok - but if I would start to allow any 
flag exceptions, other users may want to have other or more flag 
exception. 

- noprocessing 
- whitelisted 
- spamlover 
- domain based scoring values 
- SMIME/PGP signed 
. 

Yes, a great feature - but who would need it? 

The best way would be to create a level 1 plugin for this purpose. There 
you can check the dkim result, flags, ip's ... what ever you want - and 
based on your logic, you can block or pass the mail. 

But knowing (and thinking like) assp, will open other ways (solution 
workarounds) - for example. 

we assume the DKIM check is set to scoring - and the scoring value is 20 
points below the penalty limit. 

If a DKIM signature is invalid - assp scores. 
If the domain has ever sent a mail with a valid DKIM signature before (a 
DKIMCache entry is found), assp scores for DKIM if a mail  without a DKIM 
signature from this domain is received. 

Now, if there was not added any other score (the mail is 100% ok, except 
DKIM) the mail will pass because the penalty limit is not reached. But you 
want to block the mail if the sender matches @*.docusign.net 

sender??? ... matches???... - assp has weighted regular expressions - 
like: bombSenderRe - where you can add or remove scoring points 
if you set there 
\@.+\.docusign\.net=>20 

all mails from those domains will get a penalty of 20 points, which is 
harmless if there is everything else ok with the mail 
if dkim fails, the penalty limit will be reached and the mail will be 
blocked 
this can be finetuned using :>NWLI 

You are also able to implement code in to the regex (for example to check 
for the DKIM result). This is much less complicated than writing a plugin. 

\@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60 
"score with 60 if the sender matches and the sub 
CorrectASSPcfg::myWantedDKIMCheck returned 1" 

Both examples should only show, that there are more ways to get wanted 
results in assp. If someone solved a similar problem using another way, it 
would be nice to hear, how this was done. 


Thomas 




Von:"K Post"  
An:"ASSP development mailing list" 
 
Datum:07.07.2022 15:56 
Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on 
DMARC rejects 



All of your points are clear, and the explanation is greatly appreciated.  
I now understand why it may be unwise to generally honor reject DMARC 
policy if we've overridden spf/dkim policy once we start manipulating 
results with ASSP.  That makes sense. 

I still feel like a blockStrictDKIMRe type of new feature, where a failed 
OR missing dkim signature where the message matches the regex would be 
strictly blocked (just like we can do with blockstrictSPFRe for spf 
failures) would be helpful. 

For example (hopefully this is more illustrative of the desire), I want to 
outright block any message from @*.docusign.net that isn't signed or that 
has an invalid signature.  I don't care if it's from a whitelisted email 
address, from an IP that's in the SPF record, and with a message body that 
is 100% great.  If there's no DKIM signature or an invalid one for a 
message that matches the regex, reject the message (just like their DMARC 
policy says to do). 

Is there another way with current ASSP features to accomplish this only if 
a message matches this proposed regex? 

Ken 


On Fri, Jun 17, 2022 at 4:35 AM Thomas E

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[Thomas Eckardt]

If such a feature would be implemented, it will result in havy coding.

>I want to outright block any message from @*.docusign.net that isn't 
signed or that has an invalid signature.  I don't care if it's from a 
whitelisted email address, from an IP that's in the SPF record, and with a 
message body that is 100% great.

You want not only to make the test domain based strict, you want to ignore 
flags like 'whitelisted' - that's ok - but if I would start to allow any 
flag exceptions, other users may want to have other or more flag 
exception.

- noprocessing
- whitelisted
- spamlover
- domain based scoring values
- SMIME/PGP signed
.

Yes, a great feature - but who would need it?

The best way would be to create a level 1 plugin for this purpose. There 
you can check the dkim result, flags, ip's ... what ever you want - and 
based on your logic, you can block or pass the mail.

But knowing (and thinking like) assp, will open other ways (solution 
workarounds) - for example.

we assume the DKIM check is set to scoring - and the scoring value is 20 
points below the penalty limit.

If a DKIM signature is invalid - assp scores.
If the domain has ever sent a mail with a valid DKIM signature before (a 
DKIMCache entry is found), assp scores for DKIM if a mail  without a DKIM 
signature from this domain is received.

Now, if there was not added any other score (the mail is 100% ok, except 
DKIM) the mail will pass because the penalty limit is not reached. But you 
want to block the mail if the sender matches @*.docusign.net

sender??? ... matches???... - assp has weighted regular expressions - 
like: bombSenderRe - where you can add or remove scoring points
if you set there
\@.+\.docusign\.net=>20

all mails from those domains will get a penalty of 20 points, which is 
harmless if there is everything else ok with the mail
if dkim fails, the penalty limit will be reached and the mail will be 
blocked
this can be finetuned using :>NWLI

You are also able to implement code in to the regex (for example to check 
for the DKIM result). This is much less complicated than writing a plugin.
\@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60
"score with 60 if the sender matches and the sub 
CorrectASSPcfg::myWantedDKIMCheck returned 1"

Both examples should only show, that there are more ways to get wanted 
results in assp. If someone solved a similar problem using another way, it 
would be nice to hear, how this was done.


Thomas




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  07.07.2022 15:56
Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on 
DMARC rejects



All of your points are clear, and the explanation is greatly appreciated.  
 I now understand why it may be unwise to generally honor reject DMARC 
policy if we've overridden spf/dkim policy once we start manipulating 
results with ASSP.  That makes sense.

I still feel like a blockStrictDKIMRe type of new feature, where a failed 
OR missing dkim signature where the message matches the regex would be 
strictly blocked (just like we can do with blockstrictSPFRe for spf 
failures) would be helpful.   

For example (hopefully this is more illustrative of the desire), I want to 
outright block any message from @*.docusign.net that isn't signed or that 
has an invalid signature.  I don't care if it's from a whitelisted email 
address, from an IP that's in the SPF record, and with a message body that 
is 100% great.  If there's no DKIM signature or an invalid one for a 
message that matches the regex, reject the message (just like their DMARC 
policy says to do).  

Is there another way with current ASSP features to accomplish this only if 
a message matches this proposed regex?  

Ken


On Fri, Jun 17, 2022 at 4:35 AM Thomas Eckardt  wrote:
>Would you please consider adding a feature to do the same for a failed 
DKIM signature?

NO! 

Contrary to SPF, a DKIM signature has only two options : OK and FAIL - 
Based on the signature it self or based on a trusted forwarders 
authentication result (ARC).
A DKIM signature has to be valid every time for any of the above reasons. 

> I score failed spf and score failed dkim, so DoDMARC is only scoring 
even though p=reject. 

What else makes sense? 
If SPF is scored and DKIM is scored and DMARC is score - AND the resulting 
score does'nt block the mail at the pealtybox, your settings are wrong! 


>If DMARC says p=reject, why shouldn't assp outright honor that, 
regardless of if we have spf / dkim failures set to only score? 

SPF has too many options to change/override the original result in assp 
(more or less strict, overwrite, skip ), some these options also 
exists for DKIM. 
If we ignore/change/override   sender policies for SPF and DKIM, it is 
not wise to honor the reject DMARC policy strictly. 

Thomas 




Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lis

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22187

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22187:

- it was possible to define a colon (:) in 'proxyuser' - this is no longer 
allowed - as stated in rfc7235

- the upload of Stats and Griplist to the sourceforge servers failed, if 
'proxyserver' and 'proxyuser' and 'proxypass' were configured

- the connection to an IPv6 backend-server (MTA) failed, if the client 
connected to a plain (not SSL) assp-IPv6-listener and _INBOUND_:Port
  was defined for the related destination and a "destination routing 
table" for the connected local V6-IP was not existing 

- if the root password was forgotten and 'webAdminPassword' was set in 
assp.cfg to a value either starting with '45' or with a length of 13 
characters,
  weblogin was no longer possible for root, even the right was used in the 
GUI
  if a new root password is set in the assp.cfg, the case: it is starting 
with 45 and it is 13 characters long, will disable root login

  NOTICE: if you (change) set the value for 'webAdminPassword' manually in 
the assp.cfg while assp is not running
  YOU WILL LOSE ALL 
 - encryped configuration parameters
 - encrypted configuration files and included files
 - config synchronization contexts
 - defined assp GUI-users
 - licenses granted exclusively for this assp instance and the feature 
related data
 - global penaltybox registrations



Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[Thomas Eckardt]

>Would you please consider adding a feature to do the same for a failed 
DKIM signature?

NO!

Contrary to SPF, a DKIM signature has only two options : OK and FAIL - 
Based on the signature it self or based on a trusted forwarders 
authentication result (ARC).
A DKIM signature has to be valid every time for any of the above reasons.

> I score failed spf and score failed dkim, so DoDMARC is only scoring 
even though p=reject.

What else makes sense?
If SPF is scored and DKIM is scored and DMARC is score - AND the resulting 
score does'nt block the mail at the pealtybox, your settings are wrong!


>If DMARC says p=reject, why shouldn't assp outright honor that, 
regardless of if we have spf / dkim failures set to only score?

SPF has too many options to change/override the original result in assp 
(more or less strict, overwrite, skip ), some these options also 
exists for DKIM.
If we ignore/change/override   sender policies for SPF and DKIM, it is 
not wise to honor the reject DMARC policy strictly.

Thomas




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  16.06.2022 19:28
Betreff:[Assp-test] blockStrictDKIMRe -- also thoughts on DMARC 
rejects



The ability to block failed SPF, instead of just scoring them, for 
delect regex matches has been a terrific feature of ASSP for a long time.  
  (Block SPF Processing Regex* (blockstrictSPFRe) )   Would you please 
consider adding a feature to do the same for a failed DKIM signature?  
Outright blocking of a matching message that fails DKIM, regardless of the 
domain's DMARC settings.   -- maybe that's not necessary if DoDMARC will 
honor =reject, see more below.

Reasoning:
I already score failed DKIM signatures, but I can't set that score too 
high because so many organizations still send messages through 3rd parties 
with invalid DKIM signatures.  It really is incredible how many I see.  
But for frequently abused sender addresses (docusign for example), who are 
often spoofed but send otherwise unspammy content, I want to outright 
block if the DKIM signature fails.  blockStrictSPFRe usually works because 
these bad DKIM sigs are on mails that also violate SPF rules, still though 
it would be helpful if I could also just say "if a specific regex is 
matched on an email with an invalid DKIM, reject the message"

RELATED: DMARC p=reject should always reject if failed
Docusign.net has a dmarc rule of p=reject.  I want to honor that.  The 
last scam that came in from them failed SPF and failed DKIM validation, 
but the message was from a whitelisted address..  DoDMARC says that the 
blocking will be the "most less aggressive" (least aggressive) and the 
published DMARC record.  I score failed spf and score failed dkim, so 
DoDMARC is only scoring even though p=reject.

Enable DMARC Check (DoDMARC)
If enabled and ValidateSPF and DoDKIM are enabled and the sending domain 
has published a DMARC-record/policy, assp will act on the mail according 
to the senders DMARC-policy using the results of the SPF and DKIM check 
and validating the SPF/DKIM address/domain Identifier Alignment rules 
(RFC7489 section 3). It is safe to leave this feature ON, it will not 
produce false positives! The blocking mode (block, monitor, score, 
testmode) is adapted from the most less aggressive setting of ValidateSPF 
and DoDKIM - and the published DMARC record 
([p][sp]=[reject][quarantine]). Scoring is done using dmarcValencePB.
 
If DMARC says p=reject, why shouldn't assp outright honor that, regardless 
of if we have spf / dkim failures set to only score?

Thanks
Ken

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blocking new MS doc vunerability (URI attack vector)

[Thomas Eckardt]

document relationships are too commonly used to block them

if you really need to block them, use the implemented calls to 

our $checkExeExternal;  # custom subroutine to check executables external 
(eg. lib/CorrectASSPcfg.pm) - 
$ASSP_AFC::checkExeExternal->($self,\$sk,\$buff,$raf,\$pdf) if the 
internal check has not found an executable
# self - the ASSP_AFC object for this mail
  # the following paramters are refences to 
scalars
# sk - active skip tags at runtime
# buff - up to first 64 binary bytes of the 
attachment
# raf - complete binary content of the 
attachment
# pdf - decoded binary PDF content, if the 
attachment is a PDF , otherwise undef

OR

our $checkExeExternalForce; # same as $checkExeExternal - but called 
weather the internal check has found an executable or not - 
$ASSP_AFC::checkExeExternalForce->($self,\$sk,\$buff,$raf,\$pdf,\$type)
  # 
  # type - contains the previous detected 
executable type description or undef


in ASSP_AFC


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22165

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22165:

- all plugins failed to detect assp.pl versions with two digit version 
numbers
  all plugins are updated

- assp on perl 5.36.0 showed unexpected warnings at compile time

- SSL renegotiations at a tranparent proxy connection caused SMTP-timeout 
on some systems



changed:

- perl version 5.36.x is now supported and shown as recommended perl 
version for assp

- to prevent inconsitent version checks, assp.pl provides the 
version-check code for all plugins

- 'maxSSLRenegotiations' is no longer checked for transparent proxy 
connections


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blocking new MS doc vunerability (URI attack vector)

[Thomas Eckardt]

This is not possible because:


Note that the suspicious scheme ("ms-msdt:/") is not present in the 
document. It's present in the first stage payload that will be downloaded 
by Office. 

and

The document contains an external reference pointing to a malicious URL:


If the malicious URL is known, it can be detected by assp using URIBL.
Keep in mind that those malicious URL's can be generated and changed very 
quickly!

>Hopefully clamav will eventually catch it,

I don't think this is possible for every case. Also traditional AV 
scanners need to know all used malicious URL's. Only a behavior analysis 
of the document will be able to detect the malicious download and 
playload.


Solutions for CVE-2022-30190 are provided by Microsoft:

https://msrc-blog.microsoft.com/2022/05/30/guidance-for-cve-2022-30190-microsoft-support-diagnostic-tool-vulnerability/

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  31.05.2022 20:14
Betreff:[Assp-test] blocking new MS doc vunerability (URI attack 
vector)



Hello Thomas,

Any way for ASSP to block this kind of thing?

https://isc.sans.edu/forums/diary/New+Microsoft+Office+Attack+Vector+via+msmsdt+Protocol+Scheme+CVE202230190/28694

Hopefully clamav will eventually catch it, but be nice great to be able 
strip documents off using AFC if they contain the URI protocol, just like 
we do for VBA code, etc.

Thanks___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] perl / assp and openssl v3.0.x

[Thomas Eckardt]

Hi all,


If anyhow possible you should consider to not switch openssl to version 
3.0.x. There are currently too many openssl related perl modules at CPAN, 
which can't be compiled using openssl v3.
ASSP seems not to be affected by any of these incompatible modules!


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] perl version switched to 5.36

[Thomas Eckardt]

Hi all,

the development of assp V2 is switched to perl version 5.36.0.

The latest available assp version 2.6.8 build 22125 still shows a hint at 
startup that perl 5.36 is too new - but works.

The next build will accept perl 5.36.x and will fix some minor 5.36 
warnings.


There is no need to upgrade perl to version 5.36 quickly, but this task 
should be written on your TODO list.


A strawberry-perl (for windows) based distro for perl 5.36.0 is available 
at SF - strawberry-perl-5.36.0.1-64bit-relocateable_4-assp.7z. This 
archive contains also all libs and executables required for assp.


Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] bombHeaderRe matching every email

[Thomas Eckardt]

>However, if you add scoring to it:

>(?:^|\n)from:\s*_+=>1.5

>ASSP rejects it as invalid Regex.

THIS IS EXPECTED!!!

copied from the bottom of my post

>> in your bombHeaderRe the line should be:
>> 
>> ~(?:^|\n)from:\s*_+~=>60
>> 
>> the tilds are required in assp because of the used pipe (|) in the 
regex

copied from the GUI-help/manual

Fields marked with two asterisk (**) contains regular expressions (regex) 
and accept a second weight value. Every weighted regex that contains at 
least one '|' has to begin and end with a '~' - inside such regexes it is 
not allowed to use a tilde '~', even it is escaped - for example: 
~abc\~|def~=>23 or ~abc~|def~=>23 - instead use the octal (\126) or hex 
(\x7E) notation , for example ~abc\126|def~=>23 or ~abc\x7E|def~=>23 . 
Every weighted regex has to be followed by '=>' and the weight value. For 
example: Phishing\.=>1.45|~Heuristics|Email~=>50 or 
~(Email|HTML|Sanesecurity)\.(Phishing|Spear|(Spam|Scam)[a-z0-9]?)\.~=>4.6|Spam=>1.1|~Spear|Scam~=>2.1
 
. The multiplication result of the weight and the penaltybox valence value 
will be used for scoring, if the absolute value of weight is less or equal 
6. Otherwise the value of weight is used for scoring. It is possible to 
define negative values to reduce the resulting message score.

>I did not realize that it used the /s regex switch - that make sense.

in
assp/files/optRE/
all the files used for bombs and regular expressions are starting with
(?^:(?^u:(?is:

What else makes sense? For example:  header-values can be broken in to new 
lines at any character, even directly after the collon of the header-tag!
For this reason the /s switch is used for configurable regular expressions 
(like spam bombs and others - Re)  in assp since they can be 
defined!

I had a look in to old code bases and changelogs.
The oldest assp.pl version I found is 1.1.0 from 04.08.2004 09:06 - it 
uses the /s switch as well!

in the 1.1.0 changelog I found:

2003-09-26 -- Release 1.0.3
   -- Fixed bug with blank spam forwards
   -- reorganized menu slightly
   -- added extensions to block executables feature
   -- added feature to block spam bombs
   -- added feature to disable greylist upload
   -- increased greylist scan to look at last 3 days


Thomas




Von:"Scott MacLean" 
An: "ASSP Development Mailing List" 
Datum:  09.06.2022 16:54
Betreff:Re: [Assp-test] bombHeaderRe matching every email



Thank you Thomas,

I did not realize that it used the /s regex switch - that make sense. I 
will need to go edit all of my BombRE's.

Your suggestion did work:

(?:^|\n)from:\s*_+

However, if you add scoring to it:

(?:^|\n)from:\s*_+=>1.5

ASSP rejects it as invalid Regex.

On 6/9/2022 5:05, Thomas Eckardt wrote:
>>I know regex fairly well
> 
> hmm.
> 
> 
> from\:.*\_
> 
> looks very bad - it is read like:
> 
> look for
> from:
> followed by anything any long (or nothing)
> followed by
> _
> 
> in the complete header
> keep in mind: all bombRE's are using the /s regex switch (ignoring CR 
> and LF)
> 
> So, if there is an *underscore* anywhere after *from:* in the mail 
> header, the regex will match:
> 
> use
> 
> (?:^|\n)from:\s*_+
> 
> instead (collon and underscore don't need to be escaped here . but can 
be)
> 
> is read like:
> 
> look for
> at the start or after each newline
> from:
> followed by any count of CR,LF,SPACE,TAB (or nothing)
> followed by any count (but at least one) of
> _
> 
> in the complete header
> 
> in your bombHeaderRe the line should be:
> 
> ~(?:^|\n)from:\s*_+~=>60
> 
> the tilds are required in assp because of the used pipe (|) in the regex
> 
> 
> Thomas
> 
> 
> 
> 
> Von: "Scott MacLean" 
> An: "K Post" , "ASSP Development Mailing List" 
> 
> Datum: 07.06.2022 19:22
> Betreff: Re: [Assp-test] bombHeaderRe matching every email
> 
> 
> 
> 
> No, I did not. I know regex fairly well, and this to me looks like a bug
> or otherwise unintentional operation. I've commented out these lines in
> my BombHeader for now.
> 
> On 6/7/2022 10:58, K Post wrote:
>> Hi Scott,
>> Did you ever figure this out?
>> I'm no regex wiz like Thomas is, but what you have appears pretty 
simple
>> to me -- and I don't see anything wrong with it...
>> I tried
>> 
>> from\:.*\_
>> 
>> in testRE and see it matching everything too.  I don't understand  why.
>> I know this doesn't help you with why this is happening, but figured
>> that it would at least help to hear that you're not the on

Re: [Assp-test] bombHeaderRe matching every email

[Thomas Eckardt]

>I know regex fairly well

hmm.


from\:.*\_

looks very bad - it is read like:

look for 
from:
followed by anything any long (or nothing)
followed by
_

in the complete header
keep in mind: all bombRE's are using the /s regex switch (ignoring CR and 
LF) 

So, if there is an underscore anywhere after from: in the mail header, the 
regex will match:

use

(?:^|\n)from:\s*_+

instead (collon and underscore don't need to be escaped here . but can be)

is read like:

look for 
at the start or after each newline
from:
followed by any count of CR,LF,SPACE,TAB (or nothing)
followed by any count (but at least one) of 
_ 

in the complete header

in your bombHeaderRe the line should be:

~(?:^|\n)from:\s*_+~=>60

the tilds are required in assp because of the used pipe (|) in the regex


Thomas




Von:"Scott MacLean" 
An: "K Post" , "ASSP Development Mailing List" 

Datum:  07.06.2022 19:22
Betreff:Re: [Assp-test] bombHeaderRe matching every email



No, I did not. I know regex fairly well, and this to me looks like a bug 
or otherwise unintentional operation. I've commented out these lines in 
my BombHeader for now.

On 6/7/2022 10:58, K Post wrote:
> Hi Scott,
> Did you ever figure this out?
> I'm no regex wiz like Thomas is, but what you have appears pretty simple 

> to me -- and I don't see anything wrong with it...
> I tried
> 
> from\:.*\_
> 
> in testRE and see it matching everything too.  I don't understand why. 
> I know this doesn't help you with why this is happening, but figured 
> that it would at least help to hear that you're not the only one whose 
> system generates that result.
> 
> 
> 
> On Wed, Jun 1, 2022 at 5:32 PM Scott MacLean  > wrote:
> 
> I've been seeing a bunch of spam getting through my filter recently,
> and
> they all have the same thing in common: an underscore at the 
beginning
> of the "From" and/or "Subject" lines. This should be really easy to
> pick
> up with bombHeaderRe, but something's not working.
> 
> Here's an example of the spam I'm seeing:
> 
> From:_Male Health
>  >
> Subject:_Size matters and we can help
> 
> Sometimes there is a space in between the colon and the underscore,
> usually there is not.
> 
> Here is the regex I added to my bombHeaderRe:
> 
> From\:.*\_=>60
> Subject\:.*\_=>60
> 
> However, I quickly realized that this was tagging EVERY email coming
> through the server! For instance, here's an email:
> 
> From: Readly mailto:rea...@news.readly.com
>>
> 
> And looking at mail analysis, it's being caught by this regex, even
> though there is no underscore:
> 
> BombHeader RE: 'highest match: "(matchlength:84) From: Readly
>  matching bombHeaderRe(file:files/bombheaderre.txt[line 188]):
> 'From\:.*_'
> 
> Any idea what's going wrong and causing this?
> 
> 
> 
> 
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net <
mailto:Assp-test@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/assp-test
> 
> 



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22125

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22125:

- TLSv1.3 connections from assp to a backend-server were running in to a 
SMTP-timeout. This was caused by an unhandled second session-ID 
transmission in TLSv1.3.

 
changed:

- 'forceTLSIP' can now be configured for selected sender addresses/domains

- per default assp generates the 'X-Original-Authentication-Results' 
header line
  if the hidden config-variable 'genXOrigAuthResHeader' is set to zero, 
assp will generate the 'Original-Authentication-Results' header line 
instead.

- the connection-timeout-debug (ConTimeOutDebug) output is enhanced
 

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22080

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22080:


- if 'myGreeting' was configured as multiline greeting, assp has prepended 
'220 ' even the first line was starting with '220-'



changed:

The literal 'LASTCOMMAND' will be replaced by the last used SMTP-command 
in every SMTP error reply.
The literal 'MAILFROM' will be replaced by received envelope sender in 
every SMTP error reply.
The literal 'RECEIVEDHELO' will be replaced by the received HELO/EHLO 
string in every SMTP error reply.


added:

'forceTLSIP','Force these IP's to use TLS*'
  Enter IP's that you want to be enforced to use SSL/TLS, separated by 
pipes (|).
  DoTLS needs to be set to "do TLS" to make this feature working!
  If a host or client uses the MAIL FROM: command without it used STARTTLS 
before or STARTTLS has failed or it is not connected to a SSL-listener 
  (the connection is not transport layer secured), the permanent 
SMTP-error code
  502  connected by 'IPCONNECTED' - 'RECEIVEDHELO'. The used 
command 'LASTCOMMAND: ' is still not supported, because the 
connection is NOT secured by an encryption layer (TLS) - please use 
STARTTLS first FORCEEXPLAIN
  will be sent by assp and the connection will be dropped.
  IP's listed in noTLSIP , private IP-ranges , IP's in SSL-failed-Cache 
and IP's connected to a NoTLSlistenPorts are excluded from being forced by 
this feature.
  To force all IP's, enter 0.0.0.0/0|0::0/0 .
  Mails to BounceSenders are also excluded from being forced by this 
feature! So TLSRPTv1 reports and other notifications are delivered, even 
TLS/SSL is in an invalid state.
  If a connection is dropped by this feature, the connected IP will get no 
penalty (score)!
 
  If this feature is enabled for all connecting IP's, it is highly 
recommended to configure MTA-STS (SMTP MTA Strict Transport Security - RFC 
8461) or the more secure
  DANE (DNS-Based Authentication of Named Entities - RFC 6698, 7671)(SMTP 
Security via Opportunistic DNS-Based Authentication of Named Entities 
(DANE) Transport Layer Security (TLS) - RFC 7672)
  for your hosted domains!
  Notice: MTA-STS and DANE require both the SSL_version TLSv1_2 and/or 
TLSv1_3. 

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22075

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22075:

- if a DNS-TXT record contained more than one entry (multi line), only the 
first entry was read
  some SPF-records were not processed correctly for this reason
  this is fixed
 
 
changed:

- the GUI dialog "working with IP-addresses" supports now the calculation 
of resulting IP-networks, if an IP-address/range is removed from or added 
to an IP-range
  example: 192.168.0.0/16-192.168.1.0/24  or 192.168.0.0/24+192.168.1.0/24

- the 'SPF:' SPF-record definition for Groups and IP-lists is enhanced
  it is now possible to exclude SPF-include, SPF-redirect or 
IP-addresses/ranges from resolved SPF-records
  example: SPF:amazon.com -amazonses.com
  please read the general GUI-help or the Groups-GUI-help for the detailed 
explanation



Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Auto plugin update

[Thomas Eckardt]

Yes, only GPB registered assp instances will update plugins and library 
files.

Thomas





Von:"Daniel Miller via Assp-test" 
An: "ASSP development mailing list" 
Kopie:  "Daniel Miller" 
Datum:  14.03.2022 22:08
Betreff:[Assp-test] Auto plugin update



Is a subscription to the global penalty box service required for 
auto-updates of the ASSP plugins?

--
Daniel___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Empty folder created on startup

[Thomas Eckardt]

Looks like your assp.cfg contains an invalid encrypted folder name.

Thomas





Von:"Daniel Miller via Assp-test" 
An: "ASSP development mailing list" 
Kopie:  "Daniel Miller" 
Datum:  09.03.2022 03:26
Betreff:[Assp-test] Empty folder created on startup



At some point soon after ASSP starts a folder is created in the main assp 
folder. It's named "dbe254..." looks like it's over 60 characters long. 
It's empty - I never see anything in it. I can remove the folder but it's 
always re-created. What is this?

--
Daniel___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] assp development switched to perl 5.34.0

[Thomas Eckardt]

>How'd you get this done it's released at strawberryperl.com?

I've build it using a modified version of Perl::Dist::Strawberry.

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  08.03.2022 21:14
Betreff:Re: [Assp-test] assp development switched to perl 5.34.0



Nicely done!  How'd you get this done it's released at strawberryperl.com?

On Tue, Mar 8, 2022 at 10:18 AM Thomas Eckardt  wrote:
Hi all, 

the assp development is switched to perl 5.34.0 

strawberry perl 5.34.0 (for win_x64) is available at the sourceforge assp 
download repository 

ASSP V2 multithreading/ASSP V2 module 
installation/strawberry-perl-5.34.0.1-64bit-relocateable_4-assp.7z 


Thomas 


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] assp development switched to perl 5.34.0

[Thomas Eckardt]

Hi all,

the assp development is switched to perl 5.34.0

strawberry perl 5.34.0 (for win_x64) is available at the sourceforge assp 
download repository

ASSP V2 multithreading/ASSP V2 module 
installation/strawberry-perl-5.34.0.1-64bit-relocateable_4-assp.7z


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22063

[Thomas Eckardt]

Hi all,


fixed in assp 2.6.8 *SPAM-Evaporator* build 22063:


added:

- The XOAUTH2 authentication mechanism is implemented (IN and OUT/relay). 
A SSL/TLS protected connection is required in every case for XOAUTH2 - 
independend from the setting of AUTHrequireTLS.
  The help text for AUTHrequireTLS, relayAuthUser and relayAuthPass is 
changed.
  To provide the XOAUTH2 authentication mechanism, the assp library module 
Authen::SASL::Perl::XOAUTH2 is required 
(assp/lib/Authen/SASL/Perl/XOAUTH2.pm). 

changed:

- transparentRecipients: 
   - moving to a transparent proxy connection requires now that all 
envelope recipients are matching transparentRecipients (not only one)
   - moving to a transparent proxy connection is now done after the DATA 
command is received - no longer when the RCPT TO: is received
  These changes are done for security reasons. There is no longer a chance 
to abuse a transparent connection for NON-transparentRecipients or not 
local recipients!


- the default value of 'MaxAllowedDups' is changed from 50 to 0 - the help 
text is changed

- the (minimum) recommended version for the perl module Email::MIME is 
changed from 1.946 to 1.950

- SignalLog is improved: if an unexpected signal is detected (like SEGV) 
and SignalLog is set to verbose, the complete perl caller stack is now 
written to the file debugSignal.txt



Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.8 *SPAM-Evaporator* build 22058

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.8 *SPAM-Evaporator* build 22058:

- because of a MIME decoding BUG, reported mails with very long subjects 
were not processed correctly

- if a blocked mail eml-file was moved from the spam folder to the 
discarded folder, it was sometimes no longer found by the BlockReport 
feature - and so no resendlinks were provided in the blockreports


added:

'send250toIP','Send 250 OK to this list of IP-addresses*',
 List of connecting IP-addresses which will get the reply '250 OK' instead 
of SMTP error codes ('5xx a.b.c') - see send250OK .
 This is a usefull setting, if a blocked sending host got a 5xx reply and 
does not follow the SMTP-RFC's (stop and send a NDR). Instead the host 
permanently tries to send the same mail again and again.
 Such blocked mails are internaly processed like any other SPAM mail, but 
the sender will not get informed about, that the mail was not delivered to 
the final recipient!


changed:

- if SPF: lists are included in to IP-address-list, the resolving of the 
SPF-records in now done in backgound and the results are cached for the 
lowest received TTL

- 'bombDataRe' (contrary to all other bomb-RE's) was running against the 
HTML undecoded body content only -
  now the HTML decoded and HTML undecoded body is checked (only the HTML 
line endings (=CRLF) are removed from the HTML undecoded body)

- the regular expression optimization is now disabled for a regex 
configuration parameter, if weights are used for it
  this way the definiton order for the regular expressions and their 
weights is keeped

- the processing speed for IP-address regular expressions is improved 

- ASSP_FC version 5.38 is released. The virus detection is enhanced to 
detect: 
https://www.bleepingcomputer.com/news/security/malicious-csv-text-files-used-to-install-bazarbackdoor-malware/


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 22019

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 22019:

- If the used perl version was compiled without linking to libcypto, the 
perl 'crypt' command was without function. This caused the assp internal 
encryption engine to fail
  and all encrypted configuration values and files were unuseable. The 
password for the root user was not stored.
  Now, if such a perl version is found by assp, it will try to load the 
module Crypt::UnixCrypt, which has the same function like the perl 
internal crypt command. If this module
  can't be loaded, assp will die and shows a related hint at the command 
line.

- ASSP contains code to handle unexpected SEGV signal errors. The past has 
shown, that recovering assp to a normal state after a SEGV occured is 
impossible.
  Most times the maillog.txt was filled with thousands or even million of 
error lines.
  For this reason, assp will now try a restart, if a SEGV happens - if a 
restart is not possible, the assp process will be ended.

- If a query string in RBL-, RWL- and URIBL-queries was longer than 62 
byte, the query was not processed by assp. The length of such a query 
string is now limited to 253 byte.
  The length of the labels in a domain string are limited to 63 byte.


changed:

- It was possible for years now (but undocumented) to provide api keys for 
RBLServiceProvider and URIBLServiceProvider.
  The documentation for both parameters is extended.
  ...  It can be possible, that you need to provide a privat key or ID in 
the query string for a URIBL Service Provider - like: 
your-key.query-data.uribl-provider.org
   In this case, define the URIBL Service Provider like: 
your-key.$DATA$.uribl-provider.org
   The string $DATA$ will be replaced by the queried data in each 
request.


- A new function is implemented in to all IP-address lists. It is now 
possible to include all IP's of a SPF-record of a domain in to IP-address 
lists.
  The help text is extended:
   For several IP-address lists in assp, it can be advantageous to 
include all IP's (and ranges) listed in the SPF-record of a specific 
domain (for example in noPB, noHelo, whiteListedIPs, ...).
   To provide this, simply write SPF: in front of the domain name in a 
list entry - like 182.82.10.0/24|SPF:amazon.com|2201:1::1 .
   In this example assp will replace the term SPF:amazon.com with the 
list of all IP's and resolved IP's defined in the SPF-record of 
amazon.com.
   This will also work for IP lists in a group definition. Assignments 
made to such an entry - like SPF:amazon.com=>[usergroup] will be added to 
each resolved SPF-IP-address.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS?

[Thomas Eckardt]

enable 'DebugSPF'

Thomas





Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  01.01.2022 16:01
Betreff:Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS?



Nice idea Thomas, but I'm afraid this does not fit my setup. ASSP is just 
a
VM on the internal network. One IP, one default gateway. It is the 
internet
router that is dual-homed. 

In the meantime I did a tcpdump and found proof for the "funny" things I
described before:
If I do a DNS query on the command line with "dig", I see a network packet
going out and an answer packet coming back.
If I watch the tcpdump while ASSP does its regular query for DNS checking
(once a minute) I see a network packet going out, but no answer packet
coming back:

Manual DNS query:
12:53:34.550204 IP 192.168.3.201.48836 > 80.69.100.198.53: 19486+ [1au] A?
sourceforge.net. (56)
12:53:34.570496 IP 80.69.100.198.53 > 192.168.3.201.48836: 19486 1/0/0 A
204.68.111.105 (64)

Automatic query by ASSP:
12:53:45.181187 IP 192.168.3.201.53872 > 80.69.100.198.53: 50213+ A?
sourceforge.net. (33)
12:54:45.223403 IP 192.168.3.201.39858 > 80.69.100.198.53: 4629+ A?
sourceforge.net. (33)

Does anyone have an idea what the difference between the two queries might
be?

Thanks 
Dirk

Von: Thomas Eckardt  
Gesendet: Samstag, 1. Januar 2022 08:05
An: ASSP development mailing list 
Betreff: Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS?

>(two internet uplinks) 
It may be possible that you need to setuo 'dnsLocalIPAddress' on a
multihomed system! 

Thomas 







Von:"Dirk Kulmsee" <mailto:d.kulm...@netgroup.de> 
An:"'ASSP development mailing list'"
<mailto:assp-test@lists.sourceforge.net> 
Datum:31.12.2021 19:49 
Betreff:Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS? 




I think I need to cancel this thread. I can not confirm that the problem 
is
with ASSP. We have a dual-homed system here (two internet uplinks) and the
problem appears to be _somehow_ related to that. I still need to find the
root cause, but currently I think it is not ASSP.

Sorry for the noise.
Regards
Dirk

-Ursprüngliche Nachricht-
Von: Dirk Kulmsee <mailto:d.kulm...@netgroup.de> 
Gesendet: Freitag, 31. Dezember 2021 13:25
An: mailto:assp-test@lists.sourceforge.net
Betreff: [Assp-test] Timeout for 3rd DNS?

Hi everybody,

I'm currently on ASSP 2.6.6. 21351, Linux, Perl 5.32. It looks like I have 
a
problem with the settings for "DNSServers". 

In the log it always says, that the DNS I put third in DNSServers timed 
out,
even when I raise the timeout value, but when I do a DNS query from the
console, everything is fine.

Example 1:
DNSServers: 192.168.3.100|217.237.149.205|80.69.100.198=>sourceforge.net

Dec 31 12:49:39 localhost assp.pl[3738680]: [Worker_1] Info: Name 
Server
80.69.100.198: ResponseTime = 2003 ms for sourceforge.net
Dec 31 12:49:39 localhost assp.pl[3738680]: [Worker_1] Warning: Name
Server 80.69.100.198: does not respond or timed out

;; ANSWER SECTION:
sourceforge.net.159 IN  A   204..68.111.105
;; Query time: 16 msec
;; SERVER: 80.69.100.198#53(80.69.100.198)

Example 2:
DNSServers: 192.168.3.100|80.69.100.198|217.237.149.205=>sourceforge.net

Dec 31 12:58:46 localhost assp.pl[290164]: [Worker_1] Info: Name 
Server
217.237.149.205: ResponseTime = 2003 ms for sourceforge.net
Dec 31 12:58:46 localhost assp.pl[290164]: [Worker_1] Warning: Name
Server 217.237.149.205: does not respond or timed out

;; ANSWER SECTION:
sourceforge.net.146 IN  A   204..68.111.105
;; Query time: 20 msec
;; SERVER: 217.237.149.205#53(217.237.149.205)

It's not a critical problem for me, because the internal DNS I put in 
first
place uses the other two as forwarders anyway. Still this does not look
right.

Best regards
Dirk



___
Assp-test mailing list
mailto:Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



___
Assp-test mailing list
mailto:Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test






DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known
virus in this email!
***



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test






DISCLAIMER:
***
This email and any files transmitted with it may be c

Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS?

[Thomas Eckardt]

>(two internet uplinks)
It may be possible that you need to setuo 'dnsLocalIPAddress' on a 
multihomed system!

Thomas







Von:"Dirk Kulmsee" 
An: "'ASSP development mailing list'" 

Datum:  31.12.2021 19:49
Betreff:Re: [Assp-test] [CANCELLED] Timeout for 3rd DNS?



I think I need to cancel this thread. I can not confirm that the problem 
is
with ASSP. We have a dual-homed system here (two internet uplinks) and the
problem appears to be _somehow_ related to that. I still need to find the
root cause, but currently I think it is not ASSP.

Sorry for the noise.
Regards
Dirk

-Ursprüngliche Nachricht-
Von: Dirk Kulmsee  
Gesendet: Freitag, 31. Dezember 2021 13:25
An: assp-test@lists.sourceforge.net
Betreff: [Assp-test] Timeout for 3rd DNS?

Hi everybody,

I'm currently on ASSP 2.6.6. 21351, Linux, Perl 5.32. It looks like I have 
a
problem with the settings for "DNSServers". 

In the log it always says, that the DNS I put third in DNSServers timed 
out,
even when I raise the timeout value, but when I do a DNS query from the
console, everything is fine.

Example 1:
DNSServers: 192.168.3.100|217.237.149.205|80.69.100.198=>sourceforge.net

Dec 31 12:49:39 localhost assp.pl[3738680]: [Worker_1] Info: Name 
Server
80.69.100.198: ResponseTime = 2003 ms for sourceforge.net
Dec 31 12:49:39 localhost assp.pl[3738680]: [Worker_1] Warning: Name
Server 80.69.100.198: does not respond or timed out

;; ANSWER SECTION:
sourceforge.net.159 IN  A   204.68.111.105
;; Query time: 16 msec
;; SERVER: 80.69.100.198#53(80.69.100.198)

Example 2:
DNSServers: 192.168.3.100|80.69.100.198|217.237.149.205=>sourceforge.net

Dec 31 12:58:46 localhost assp.pl[290164]: [Worker_1] Info: Name 
Server
217.237.149.205: ResponseTime = 2003 ms for sourceforge.net
Dec 31 12:58:46 localhost assp.pl[290164]: [Worker_1] Warning: Name
Server 217.237.149.205: does not respond or timed out

;; ANSWER SECTION:
sourceforge.net.146 IN  A   204.68.111.105
;; Query time: 20 msec
;; SERVER: 217.237.149.205#53(217.237.149.205)

It's not a critical problem for me, because the internal DNS I put in 
first
place uses the other two as forwarders anyway. Still this does not look
right.

Best regards
Dirk



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test






DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21351

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21351:

- if 'onlyAUTHHeloRe' was used, a logline for a match was written after 
each SMTP command - now the match is shown only after the HELO/EHLO 
command was used


changed:

- ASSP_AFC.pm version 5.37 is now able to detect template injection in 
RTF-documents
  
https://www.proofpoint.com/uk/blog/threat-insight/injection-new-black-novel-rtf-template-inject-technique-poised-widespread


- the status of the perl module IO::Socket::INET6 was set to deprecated by 
the cpan maintainer
  assp.pl uses now IO::Socket::IP for IPv6 handling
  ASSP_DCC.pl 2.02 also uses IO::Socket::IP for IPv6 handling
  IO::Socket::INET6 is no longer installed by the assp perl module 
installer (2.10)


- the module Mail::SPF::Query (SPFv1) is removed from assp.pl (only 
Mail::SPF is used)
  Mail::SPF::Query is no longer installed by the assp perl module 
installer (2.10)
  'LocalPolicySPF' is removed from configuration, it was only used by 
Mail::SPF::Query (SPFv1)
  'SPF2' is removed from configuration - it is no longer required, 
Mail::SPF (SPFv2) is permanently used

 

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] onlyAUTHHeloRe gets checked repeatedly

[Thomas Eckardt]

Everything is OK. onlyAUTHHeloRe is checked as part of the internal 
'DisableAUTH' check. This has to be done multiple times, if the 
'DisableAUTH'-state was never reached for any reason.
For now, you can skip the logging using 'noLogLineRe'.

I'll see, if I can skip the logging - the check itself is required.

Thomas






Von:"Dirk Kulmsee" 
An: 
Datum:  25.11.2021 15:56
Betreff:[Assp-test] onlyAUTHHeloRe gets checked repeatedly



Hi everybody,
I am now running  ASSP 2.6.6 21328 on Linux with Perl 5.32. 
I have set NoAUTHListenPorts for port 25.
My listenPort2 is 587 and I have activated EnforceAUTH for the secondary 
listen port.
There is only one external system that should relay through me, so I set 
onlyAUTHHeloRe accordingly.
The whole setup works fine, but onlyAUTHHeloRe apparently is checked 
numerous times:
 
Nov 25 13:56:00 localhost assp.pl[880887]: [Worker_1] Worker_1 wakes up
Nov 25 13:56:00 localhost assp.pl[880887]: [Worker_1] Info: Worker_1 got 
connection from MainThread
Nov 25 13:56:00 localhost assp.pl[880887]: [Worker_1] IP 88.130.20.65 
matches debugIP - with 88.130.20.65/32
Nov 25 13:56:00 localhost assp.pl[880887]: [Worker_1] Connected: 
session:7F64D1D294D8 88.130.20.65:15079 > 192.168.101.242:587 > 
127.0.0.1:125
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] 88.130.20.65 info: 
got STARTTLS request from 88.130.20.65
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] [TLS-in] 
88.130.20.65 info: authentication - login is used
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] SWL-2012R2-DC 
matches SWL-2012R2-DC in onlyAUTHHeloRe
Nov 25 13:56:01 localhost assp.pl[880887]: m1-44961-10616 [Worker_1] 
[TLS-in] 88.130.20.65  info: found message size 
announcement: 27.93 kByte
Nov 25 13:56:01 localhost assp.pl[880887]: [Worker_1] u...@senderdomain.de 

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21328

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21328:


- If 'AddRWLHeader' was enabled and a RWLCache hit was found, no 
RWL-header was added to the mail.


changed:

- text/plain parts of a mail are now also cleanedup from (badly) added 
HTML-tags, which improves all text based features, because the text/plain 
parts are most times processed first

- ASSP_OCR.pm version 2.25 is released
  It implements a short time result cache, to prevent processing the same 
MIME-parts multiple times because the same mail was sent to multiple 
recipients.

- The clamav and the filescan feature are implementing a short time result 
cache, to prevent processing the same MIME-parts multiple times because 
the same mail was sent to multiple recipients.

- In rare cases it was possible to overload assp by sending a large mail 
to many recipients.
  This can now prevented by configuring the hidden parameter 
'maxSMTPipRelaySessions' - if used, it should be set one less than the 
configured number of SMTP-Workers (NumComWorkers)
 
# (number) limit the connection count per IP for relay - 0 and 
noMaxSMTPSessions disables the check
our $maxSMTPipRelaySessions = 0;

- the RWLCache was used without giving any cached results in the 
maillog.txt - this caused confusion, if the log wére analyzed
- the term 'whitelisted' is replaced by 'trusted' in RWL-headers and log 
lines - this caused confusion, if the mails was not whitelisted because of 
lower RWL-trust value

- The RBL/DNSBL check was skipped, if a RWL-trust value of 2 was reached. 
>From now, the RBL/DNSBL check is skipped, if a RWL-trust value of 2 is 
reached and RBLWL is not set.


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

lyzer for reports 
from Outlook, corpus cleanup for DKIM WL/NP matches.
21396 more changes because of discussions about Outlook reporting  (FYI  
forward as attachment from Outlook still doesn't result in correct analyze 
reports nor does multiple report attachments in a single email from 
Outlook work at all.)
21317 After my questions about the unusual request for help for a way to 
match username of the recipient to the sender we discovered the bug about 
unoptimized weighted bombs with a scoring parameter and the bug with 
definite statements
And over the years you've added useful features and fixed bugs because of 
my questions or requests which you originally dismissed as being misguided

There's a trend here. When I'm active on this forum, I discuss things that 
lead you to improve ASSP which benefits everyone.

If I had asked my question and then not responded to your short "no" or 
"have you thought about this" type of replies, would these changes have 
been made?  If I hadn't fully described the issue/question/challenge, how 
would you have known what I was talking about?

I will now step away from this form as requested for as long as I am able. 
I do hope that you are willing to entertain future questions/concerns once 
I return, if not for me, then for the rest of the quiet spam fighters on 
this list.

On Sun, Nov 14, 2021 at 5:59 AM Thomas Eckardt  wrote:
>How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list?  

how many? 1 at 5.11.2021 - weight bug 

most - where? -> forum , bug tracker , self testing, forced by attackers 

You may use the forum, where everyone is free to skip reading your endless 
posts and blogs. It takes simply too much time to pick up the 1 to 5% of 
helpful content and to be forced by you to answer also the rest. 


Thomas 





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:14.11.2021 00:14 
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation? 



I don't know what I've done to deserve that reply, but regardless, I'm 
sorry to have upset you.  I will take a long break from posting 
further here, but please do know that I'm appreciative of your continued 
support of this important program.  

Before I go, please entertain these thoughts:   

I hope that you're able to re-evaluate your request for me to go away.  
I've recommended more very good change requests to ASSP than ones that you 
consider to be bad.  I'm not able to implement them myself.  I'm not 
perfect, but your request for me to sign off of this list, which is a 
critical resource, is unfair. 

How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list?  How many bugs have been 
quashed because of things I've discovered?  How many improvements did you, 
and only you, make because of questions I've asked and because of feature 
requests I've made (recently and over the many years)? 

Are you angry because I'm (adminitedly) long winded?  Please understand 
that this is not out of disrespect, it's because I want to make sure that 
I'm being clear.  When I get a short answer, I try to continue the 
conversation.  This is a discussion list after all. 

Are you angry because I'm persistent?  My persistence is also not out of 
disrespect, it's because I'm inquisitive,  am by no means an expert in 
coding or the inner workings of spam detection, and have a burning desire 
to continue to see ASSP improve.  Often I ask a detailed question, and 
only get an answer back from you like "have you considered this?" or "no" 
without explanation.  Is it so bad that I ask why not?  I wait patiently 
for your replies, but do inquire more if my questions haven't been fully 
answered.  If you don't have the time or desire to entertain my questions, 
so be it, but please remember that most of what I ask has ultimately led 
to you eventually improving ASSP.  

Anyway, I don't expect and certainly don't require a reply here.  But 
please know that my intentions are pure, I'm charitable, patient, and a 
good person. It hurts deeply that you seem to think otherwise.  I don't 
have the experience nor the ability that you do, not even close, but I 
like to think that even if I can be frustrating that I'm ultimately bring 
some good to the ASSP world by offering suggestions and asking questions. 



On Sat, Nov 13, 2021 at 3:56 AM Thomas Eckardt  wrote: 
Ken , it would be nice if you consider to signoff this list or at least to 
no longer post here. 

Thank you. 

Thomas





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:12.11.2021 22:46 
Betreff:Re: [Assp-test] Concept Question: Scan 

Re: [Assp-test] No more update to the "http" repository ?

[Thomas Eckardt]

Depending on the version you use the download location is different. Links 
are provided in GUI-Stats page.

2.6.5 (odd minor number) - public release (SF http download)
2.6.6 (even minor number) - development release (SVN)

Thomas



Von:"Davide Yachaya" 
An: assp-test@lists.sourceforge.net
Datum:  15.11.2021 11:37
Betreff:[Assp-test] No more update to the "http" repository ?




Hi Thomas,


Sorry to bother you but it seems that the http repository on sourceforge 
for assp.pl is not updated (18 Aug 2021) so the last version is never 
downloaded from assp.

The “SVN”  repository is ok.

Did i miss some notice ?


Kind regards,

  Davide


 
--
Davide Yachaya
HyperGrid s.r.l.
V.le Golgi 63 - 27100 Pavia - ITALY   http://www.hypergrid.it
Tel:   +39-0382-528875  Fax:   +39-0382-049303 




___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

>I have no other way to communicate with you

I told you, to use the forum (the assp - forum, what else?). 
http://sourceforge.net/p/assp/forum/

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  14.11.2021 17:01
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?



I can not decypher what this means:
most - where? -> forum , bug tracker , self testing, forced by attackers
and it's my lack of clarity on your short replies which leads me to 
question further.  

I need to find a way to still be able to report my findings and ask my 
questions without being a bother.  The last thing I want to be is a 
burden, but I have no other way to communicate with you, as the sole 
developer on a project that has minimal user communication other than what 
you and I discuss.

While I wish it were easier for me to be more concise, my persistence and 
full description of issues and challenges has resulted in far more than 
the one change you referenced.  I've outlined some of them from the last 7 
versions below. 
1 of the changes in 21277 is because of my report.  Very slow startup of 
the rebuild process.
2+ of the changes in 21280 stemmed from my messages.  Too many open files 
in Windows, early bad SSL changes, catching invalid regex instead of ASSP 
crashing
21287 & 21290: your changes to griplist folder creation, changes/fixes to 
BereleyDB error logging, gui changes, and windows file descriptor changes 
are because of things I've brought up
21293: The NWLI changes are because of what I asked
7 of the 8 changes in 21302 are because of my reports, questions, 
requests, and suggestions.  Related to external file change times not 
being recorded in ASSP (long time bug), improvement in a single file 
changing causing all to be reloaded, changes to the analyzer for reports 
from Outlook, corpus cleanup for DKIM WL/NP matches.
21396 more changes because of discussions about Outlook reporting  (FYI  
forward as attachment from Outlook still doesn't result in correct analyze 
reports nor does multiple report attachments in a single email from 
Outlook work at all.)
21317 After my questions about the unusual request for help for a way to 
match username of the recipient to the sender we discovered the bug about 
unoptimized weighted bombs with a scoring parameter and the bug with 
definite statements
And over the years you've added useful features and fixed bugs because of 
my questions or requests which you originally dismissed as being misguided

There's a trend here. When I'm active on this forum, I discuss things that 
lead you to improve ASSP which benefits everyone.

If I had asked my question and then not responded to your short "no" or 
"have you thought about this" type of replies, would these changes have 
been made?  If I hadn't fully described the issue/question/challenge, how 
would you have known what I was talking about?

I will now step away from this form as requested for as long as I am able. 
I do hope that you are willing to entertain future questions/concerns once 
I return, if not for me, then for the rest of the quiet spam fighters on 
this list.

On Sun, Nov 14, 2021 at 5:59 AM Thomas Eckardt  wrote:
>How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list?  

how many? 1 at 5.11.2021 - weight bug 

most - where? -> forum , bug tracker , self testing, forced by attackers 

You may use the forum, where everyone is free to skip reading your endless 
posts and blogs. It takes simply too much time to pick up the 1 to 5% of 
helpful content and to be forced by you to answer also the rest. 


Thomas 





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:14.11.2021 00:14 
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation? 



I don't know what I've done to deserve that reply, but regardless, I'm 
sorry to have upset you.  I will take a long break from posting 
further here, but please do know that I'm appreciative of your continued 
support of this important program.  

Before I go, please entertain these thoughts:   

I hope that you're able to re-evaluate your request for me to go away.  
I've recommended more very good change requests to ASSP than ones that you 
consider to be bad.  I'm not able to implement them myself.  I'm not 
perfect, but your request for me to sign off of this list, which is a 
critical resource, is unfair. 

How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list?  How many bugs have been 
quashed because of things I've discovered?  How many improvements did you, 
and only you, make because of questions

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

>How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list? 

how many? 1 at 5.11.2021 - weight bug

most - where? -> forum , bug tracker , self testing, forced by attackers

You may use the forum, where everyone is free to skip reading your endless 
posts and blogs. It takes simply too much time to pick up the 1 to 5% of 
helpful content and to be forced by you to answer also the rest.


Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  14.11.2021 00:14
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?



I don't know what I've done to deserve that reply, but regardless, I'm 
sorry to have upset you.  I will take a long break from posting 
further here, but please do know that I'm appreciative of your continued 
support of this important program. 

Before I go, please entertain these thoughts:  

I hope that you're able to re-evaluate your request for me to go away.  
I've recommended more very good change requests to ASSP than ones that you 
consider to be bad.  I'm not able to implement them myself.  I'm not 
perfect, but your request for me to sign off of this list, which is a 
critical resource, is unfair.

How many of the changes in the last 10 or so versions of ASSP have been 
from the requests of anyone else on this list?  How many bugs have been 
quashed because of things I've discovered?  How many improvements did you, 
and only you, make because of questions I've asked and because of feature 
requests I've made (recently and over the many years)?

Are you angry because I'm (adminitedly) long winded?  Please understand 
that this is not out of disrespect, it's because I want to make sure that 
I'm being clear.  When I get a short answer, I try to continue the 
conversation.  This is a discussion list after all.

Are you angry because I'm persistent?  My persistence is also not out of 
disrespect, it's because I'm inquisitive,  am by no means an expert in 
coding or the inner workings of spam detection, and have a burning desire 
to continue to see ASSP improve.  Often I ask a detailed question, and 
only get an answer back from you like "have you considered this?" or "no" 
without explanation.  Is it so bad that I ask why not?  I wait patiently 
for your replies, but do inquire more if my questions haven't been fully 
answered.  If you don't have the time or desire to entertain my questions, 
so be it, but please remember that most of what I ask has ultimately led 
to you eventually improving ASSP. 

Anyway, I don't expect and certainly don't require a reply here.  But 
please know that my intentions are pure, I'm charitable, patient, and a 
good person. It hurts deeply that you seem to think otherwise.  I don't 
have the experience nor the ability that you do, not even close, but I 
like to think that even if I can be frustrating that I'm ultimately bring 
some good to the ASSP world by offering suggestions and asking questions.



On Sat, Nov 13, 2021 at 3:56 AM Thomas Eckardt  wrote:
Ken , it would be nice if you consider to signoff this list or at least to 
no longer post here. 

Thank you. 

Thomas





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:12.11.2021 22:46 
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation? 



First off, WOW.  Our rebuild times are in no way similar.   At first I 
thought it was you with fancy SSD's and lots of horsepower, but I'm seeing 
now that you have both useDB4Rebuild off and RebuildUseFileModel on.  The 
opposite of my settings.  I have useDB4Rebuild on and never enabled the 
RebuildUsedFileModel after initial attempts were failing (Early on with 
that feature).  useDB4Rebuild is the default and I was always worried 
about RAM when I started using ASSP 10+ years ago and never looked back.   


A long rebuild time doesn't bother me, but seeing how fast you can do one 
has got me back to needing to test the settings on my end again.  Thanks 
for that encouragement. 


I'm worried that going up to 50k maxbytes on my system seemed to cause a 
lot of false positives.  I don't understand how that's possible, but it's 
what happened.  I would have thought it was the other way around, too much 
spam getting through vs. too much legit being blocked.  Plus, I don't 
think that generally using that much for bayesian is necessary (or maybe 
it's even detrimental?)  Accuracy was very high for me at  6k and 10k, but 
I was missing the bombs.  


The question remains for me about the >CONCEPT< of optionally scanning 
more of a message at the time of attempted delivery for bombs.  ClamAV 
uses its own maximum size setting.  Why not also give us that optio

Re: [Assp-test] PenaltyExtreme not used?

[Thomas Eckardt]

Just checked this again.

worker_1 - AUTH failes and switched on DelayIP for the IP
worker_2 - DelayIP delays the IP

ExtremeIP is checked very much later.

Thomas





Von:"Dirk Kulmsee" 
An: 
Datum:  12.11.2021 14:47
Betreff:[Assp-test] PenaltyExtreme not used?



Hi all,
I'm currently running ASSP 2.6.6. (21306) on Linux with Perl 5.32.
I have set both DoPenaltyExtreme and DoPenaltyExtremeSMTP to "block". My 
ExtremePenaltyTheshold (PenaltyExtreme) is set to 1500.
In the log I see a candidate for extreme treatment, but the log lines do 
not mention the "Extreme" status. 
The following lines show two concurrent connections from the same IP, one 
ends up in damping (Worker_1), the second one gets delayed (Worker_2):

Nov 12 14:02:29 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
got connection request
Nov 12 14:02:29 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
freed by idle Worker_1 in 0.004 seconds and zero cycles - got (ok)
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Worker_1 wakes up
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: Worker_1 got 
connection from MainThread
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] IP 45.144.225.61 
matches debugIP - with 45.144.225.61/32
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: try to connect 
to server at 127.0.0.1:125
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: connected to 
server at 127.0.0.1:125
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Connected: 
session:7F0F3C318670 45.144.225.61:42832 > 192.168.101.242:25 > 
127.0.0.1:38320 > 127.0.0.1:125 , 1558-1560
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: sent DNS query 
for '45.144.225.61' type 'PTR' to nameserver 192.168.101.222 ID 22692
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: got valid DNS 
NON-DATA answer 'NXDOMAIN' from nameserver 192.168.101.222 ID 22692
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
injected '250-STARTTLS' offer in to EHLO reply
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
send '250-STARTTLS' - injected for 127.0.0.1
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
removed '250-STARTTLS' - it was already injected
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
[unsupported_AUTH] 45.144.225.61 AUTH not allowed
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: no skip 
condition detected for check: main::AUTHErrorsOK
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 Message-Score: added 60 (autValencePB) for too many (111) 
AUTH errors from 45.144.225.0, total score for this message is now 60
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 PB-IP-Score for '45.144.225.61' is 13740, added 60 for 
AUTHErrors
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 [SMTP Error] 502 AUTH not supported
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: start damping (58 s)
Nov 12 14:02:48 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
got connection request
Nov 12 14:02:48 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
freed by idle Worker_2 in 0.006 seconds and zero cycles - got (ok)
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] Worker_2 wakes up
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] Info: Worker_2 got 
connection from MainThread
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] IP 45.144.225.61 
matches debugIP - with 45.144.225.61/32
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] [SMTP Status] 451 
4.7.1 Please try again later
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] Delayed ip 
45.144.225.61, because PBBlack(13740) is higher than DelayIP(500)- last 
penalty reason was: AUTHErrors
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] Worker_2 will sleep 
now
Nov 12 14:03:29 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: damping - stolen 58 seconds
Nov 12 14:04:26 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: PB-IP-Score for '45.144.225.61' is 13740, added 60 in 
this session
Nov 12 14:04:26 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 disconnected: session:7F0F3C318670 45.144.225.61 - command 
list was 'EHLO,RSET,AUTH,QUIT' - used 4 SocketCalls - processing time 117 
seconds - damped 116 seconds
Nov 12 14:04:26 localhost assp.pl[446339]: [Worker_1] Worker_1 will sleep 
now

Why the different behaviour on these two connections? And shouldn't there 
be log lines, that refer to the IP score beyond PenaltyExtreme?
I'm curious what I did wrong this time 

Best regards
Dirk



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21317

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21317:

- If a line in a regular expression file was protected from regex 
optimization using the <<<...>>> pagma, a possibly defined weight 
(...=>ddd) was ignored and the default penalty points were used.

- If a predefinition of regular expressions like '(?(DEFINE)(?<..>...))' 
was used, assp has destroyed it sometimes to (?(?:DEFINE)...)


changed:

- files used in configuration parameters are now supporting line 
continuation by adding a backslash '\' at the end of a line

- SMTP-replies (554 ...Service denied ...) sent because a connection is 
terminated very early (before HELO) are now extended to 554 ...Service 
denied for IP x.x.x.x , ...
  where x.x.x.x contains the connected IP address. This provides better 
backtracking of early blocked connections.


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

Ken , it would be nice if you consider to signoff this list or at least to 
no longer post here.

Thank you.

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  12.11.2021 22:46
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?



First off, WOW.  Our rebuild times are in no way similar.   At first I 
thought it was you with fancy SSD's and lots of horsepower, but I'm seeing 
now that you have both useDB4Rebuild off and RebuildUseFileModel on.  The 
opposite of my settings.  I have useDB4Rebuild on and never enabled the 
RebuildUsedFileModel after initial attempts were failing (Early on with 
that feature).  useDB4Rebuild is the default and I was always worried 
about RAM when I started using ASSP 10+ years ago and never looked back.  

A long rebuild time doesn't bother me, but seeing how fast you can do one 
has got me back to needing to test the settings on my end again.  Thanks 
for that encouragement.


I'm worried that going up to 50k maxbytes on my system seemed to cause a 
lot of false positives.  I don't understand how that's possible, but it's 
what happened.  I would have thought it was the other way around, too much 
spam getting through vs. too much legit being blocked.  Plus, I don't 
think that generally using that much for bayesian is necessary (or maybe 
it's even detrimental?)  Accuracy was very high for me at  6k and 10k, but 
I was missing the bombs. 


The question remains for me about the >CONCEPT< of optionally scanning 
more of a message at the time of attempted delivery for bombs.  ClamAV 
uses its own maximum size setting.  Why not also give us that option for 
Bombs?  For the case I explained where bombs are late in the email body 
and likely other scenarios, don't you think it would be helpful to have a 
BombAddlBytes variable in the GUI? 

You know there's no way that I could ever code a plugin and that there's 
even less of a chance of this charity paying for one to be built!  I still 
have duct tape holding my desk chair together.  

Modifying getbody seems pretty straight forward.  Add a new variable 
called $bombdataref that would be used in place of $dataref for all bomb 
comparisons - similarly to the way that $clamavbytes is for the clamav 
stuff.  
my $bombdataref = $maxbytes + $BombAddlBytes : $BombAddlBytes : 0;
then, instead of if ( ! BombOK( $fh, $dataref ) ) { 
if ( ! BombOK( $fh, $bombdataref ) ) {
and the like everywhere that there's a bomb or script check in getbody

There would also need to be changes in analyze and anywhere else that the 
bomb checks are done.

I'm more than willing to try to modify ASSP as described above, give it a 
go, and report back.  It won't be easy for me to make the changes and have 
it work, but I'm game.  Before I do though, I'm concerned that you don't 
think that scanning more for bombs is a sound concept.  Or maybe you just 
don't think it's necessary?  I'm most interested in your opinion on that 
before I move forward.




On Fri, Nov 12, 2021 at 1:08 PM Thomas Eckardt  wrote:
Nov-12-21 04:00:20 RebuildSpamDB-thread rebuildspamdb-version 8.14 started 
in ASSP version 2.6.6(21314) 

Nov-12-21 04:00:20 detection of local disclaimers is enabled 

Nov-12-21 04:00:20 info: 'useDB4Rebuild' is NOT set to on - the rebuild 
spamdb process will possibly require a large amount of memory - but it 
will run very fast! 

Nov-12-21 04:00:20 RebuildSpamDB reloaded and uses the internal FileModel 
(with 39917 entries) to speedup processing 

Nov-12-21 04:00:20 RebuildSpamDB allocated 963.08 MByte of RAM to load the 
internal FileModel 

Nov-12-21 04:00:20 RebuildSpamDB will create a Hidden Markov Model 

Nov-12-21 04:00:20 RebuildSpamDB will include attachment-database-entries 
in to spamdb 

Nov-12-21 04:00:20 RebuildSpamDB will create unicode enabled databases 

Nov-12-21 04:00:20 RebuildSpamDB will process all words as Sequence of UAX 
#29 Grapheme Clusters 

Nov-12-21 04:00:20 RebuildSpamDB will normalize unicode characters 

Nov-12-21 04:00:20 RebuildSpamDB will use the ASSP_WordStem engine 

Nov-12-21 04:00:20 ---ASSP Settings--- 

Nov-12-21 04:00:20 RebuildSpamDB will create private spamdb entries for 
users email addresses and each local domain. 

Nov-12-21 04:00:20 Do Not Collect RedRe Messages: Enabled 
**Messages matching the RedRe will be removed from the corpus!** 

Nov-12-21 04:00:20 Use Subject as Maillog Names: True 
Nov-12-21 04:00:20 Maxbytes: 25,000 
Nov-12-21 04:00:20 Maxfiles: 31,000 
Nov-12-21 04:00:20 RebuildFileTimeLimit: 1 5 
Nov-12-21 04:00:20 RebuildFileTimeLimit: files will be moved away from the 
corpus if their processing takes longer than 5 second(s) 

processing ~40.000 corpus files in ~4 minutes 
building 15.500 spamdb.helo records in 2 seconds 
building 3.200.000 spamdb records in 25 seconds 
building 7.200.000 hmmdb records in 1:33 seconds 

complete pr

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

mbers (also available online) to capture these messages before 
they're delivered.  Simple.  If the message has one of these phone 
numbers, score it such that it'll get blocked.

The problem with many of these emails is that the phone number is way past 
the 3k mark, and past the 20k mark too.  The scammers have a bunch of HTML 
in the "confirmation" email, just like real stores tend to do.  I tried 
increasing MaxBytes up to 50kb, which easily caught messages with bombs 
later in the body, but that then seemed to cause a lot of false positives 
and obviously much longer rebuild process.  

If there could be a "continue canning for bombs for ___kb after maxbytes" 
setting, that would let bombs later in the body be detected.  I don't know 
what the downside to having such a feature would be.


Based on your reaction to my question, I'm obviously missing something 
important.
 




On Thu, Nov 11, 2021 at 1:38 AM Thomas Eckardt  wrote:
>Is there logic to having a separate MaxBytes setting like 
MaxBytesForBombs that's used only during message delivery?  That way, the 
entire message can be scanned for bombs, but the rebuild could use a lower 
number to better balance the differential between the average sized spam 
and average sized not-spam message. 

DID YOU EVER thougth about that ??? Or do you only write 
something to fillup the community mailing list? 

No - no way! 

Thomas 







Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:10.11.2021 20:22 
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation? 



After about 12 weeks of going from MaxBytes of 4k to MaxBytes of 50k, 've 
seen: 
1) Rebuild go from just over an hour (with 30k MaxFiles) to just over 2 
hours.  I'm fine with that, there's more to scan 
2) Bomb detections improve, as a lot of what's detected is beyond the 20k 
or 30k mark 
3) but, bayesian false positives going way up.  Lots of mail that would 
have (correctly) been delivered, is now getting too high of a score and is 
blocked. 

Surely #3 is specific to the types of messages my users are getting and I 
can tweak settings.  BUT, it makes me raise this question again: 
Is there logic to having a separate MaxBytes setting like MaxBytesForBombs 
that's used only during message delivery?  That way, the entire message 
can be scanned for bombs, but the rebuild could use a lower number to 
better balance the differential between the average sized spam and average 
sized not-spam message. 



On Mon, Nov 1, 2021 at 2:43 PM K Post  wrote: 
When looking at the "Use this HTML Parser" section on the GUI, I found 
this line: 
it is recommended to set MaxBytes to 5 (be carefull on heavy load 
systems - spam bomb regular expressions will take longer using 5!).\ 
I'm going to change my settings and see how bad the rebuild time is.  I've 
got enough processing power and RAM now, but the disks aren't SSD.  Just a 
4 disk Raid 1+0 traditional HDD setup.  We'll see... 

Since HTMl email accounts for a big percentage of all mail,  might it be a 
good idea to update/expand the guidance in the MaxBytes section of the 
GUI?



On Fri, Oct 29, 2021 at 8:40 PM K Post  wrote: 
Summary: 
Should/could any consideration be given to having ASSP scan the entire 
message at the time it is received for Bombs (only), while still using 
MaxBytes for Bayesian/HMM? 

We've been having some cleverly crafted messages slipping through all 
filters that would be easy to catch with Bombs if only the catchable 
content came before MaxBytes.  These messages are 20kb+, They have a scam 
phone number at the very end of the larger than MaxBytes messages.  I 
want/need to use bombs to catch the scam phone numbers. 

With MaxBytes set to 3000, which is useful for faster RebuildSpamDB, these 
BombDataRE matches just aren't being caught.  If I increase MaxBytes, my 
BombDataRE catches them, but then rebuildspamdb is (probably? see below) 
longer than it needs to be. 

So, is there any value in considering a MaxBytesAdditionalForBombs 
variable which would be added to MaxBytes and only used when scanning for 
bombs as messages arrive?   Would that kill performance??  Other 
downsides? 

We could still only look at MaxBytes for Bayesian/HMM since it's only 
MaxBytes used when building those databases. 

What do you think? 

And while we're talking MaxBytes: 
I've asked this before, is the guidance for 3kb for MaxBytes once there's 
a mature corpus still a valid recommendation?  With unlimited horsepower 
and ram, sure, why not, do 30kb or 100kb.  That's not my reality, so I 
want to see where to best allocate resources. If 3kb is still the 
guidance, even though the spam files I'm seeing have a median size around 
20kb, so be it.  I feel like when that guidance was written, html wasn't 
used a

Re: [Assp-test] PenaltyExtreme not used?

[Thomas Eckardt]

worker_1 is blocked because the AUTH check comes first and AUTH was used
worker_2 is blocked by PenaltyDelay - AUTH was not used

both are blocked before the PenaltyExtreme check is done

Thomas





Von:"Dirk Kulmsee" 
An: 
Datum:  12.11.2021 14:47
Betreff:[Assp-test] PenaltyExtreme not used?



Hi all,
I'm currently running ASSP 2.6.6. (21306) on Linux with Perl 5.32.
I have set both DoPenaltyExtreme and DoPenaltyExtremeSMTP to "block". My 
ExtremePenaltyTheshold (PenaltyExtreme) is set to 1500.
In the log I see a candidate for extreme treatment, but the log lines do 
not mention the "Extreme" status. 
The following lines show two concurrent connections from the same IP, one 
ends up in damping (Worker_1), the second one gets delayed (Worker_2):

Nov 12 14:02:29 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
got connection request
Nov 12 14:02:29 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
freed by idle Worker_1 in 0.004 seconds and zero cycles - got (ok)
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Worker_1 wakes up
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: Worker_1 got 
connection from MainThread
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] IP 45.144.225.61 
matches debugIP - with 45.144.225.61/32
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: try to connect 
to server at 127.0.0.1:125
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Info: connected to 
server at 127.0.0.1:125
Nov 12 14:02:29 localhost assp.pl[446339]: [Worker_1] Connected: 
session:7F0F3C318670 45.144.225.61:42832 > 192.168.101.242:25 > 
127.0.0.1:38320 > 127.0.0.1:125 , 1558-1560
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: sent DNS query 
for '45.144.225.61' type 'PTR' to nameserver 192.168.101.222 ID 22692
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: got valid DNS 
NON-DATA answer 'NXDOMAIN' from nameserver 192.168.101.222 ID 22692
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
injected '250-STARTTLS' offer in to EHLO reply
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
send '250-STARTTLS' - injected for 127.0.0.1
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] 45.144.225.61 info: 
removed '250-STARTTLS' - it was already injected
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
[unsupported_AUTH] 45.144.225.61 AUTH not allowed
Nov 12 14:02:30 localhost assp.pl[446339]: [Worker_1] Info: no skip 
condition detected for check: main::AUTHErrorsOK
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 Message-Score: added 60 (autValencePB) for too many (111) 
AUTH errors from 45.144.225.0, total score for this message is now 60
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 PB-IP-Score for '45.144.225.61' is 13740, added 60 for 
AUTHErrors
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 [SMTP Error] 502 AUTH not supported
Nov 12 14:02:30 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: start damping (58 s)
Nov 12 14:02:48 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
got connection request
Nov 12 14:02:48 localhost assp.pl[446339]: [Main_Thread] Info: Main_Thread 
freed by idle Worker_2 in 0.006 seconds and zero cycles - got (ok)
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] Worker_2 wakes up
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] Info: Worker_2 got 
connection from MainThread
Nov 12 14:02:48 localhost assp.pl[446339]: [Worker_2] IP 45.144.225.61 
matches debugIP - with 45.144.225.61/32
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] [SMTP Status] 451 
4.7.1 Please try again later
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] Delayed ip 
45.144.225.61, because PBBlack(13740) is higher than DelayIP(500)- last 
penalty reason was: AUTHErrors
Nov 12 14:02:49 localhost assp.pl[446339]: [Worker_2] Worker_2 will sleep 
now
Nov 12 14:03:29 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: damping - stolen 58 seconds
Nov 12 14:04:26 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 info: PB-IP-Score for '45.144.225.61' is 13740, added 60 in 
this session
Nov 12 14:04:26 localhost assp.pl[446339]: m1-22150-05202 [Worker_1] 
45.144.225.61 disconnected: session:7F0F3C318670 45.144.225.61 - command 
list was 'EHLO,RSET,AUTH,QUIT' - used 4 SocketCalls - processing time 117 
seconds - damped 116 seconds
Nov 12 14:04:26 localhost assp.pl[446339]: [Worker_1] Worker_1 will sleep 
now

Why the different behaviour on these two connections? And shouldn't there 
be log lines, that refer to the IP score beyond PenaltyExtreme?
I'm curious what I did wrong this time 

Best regards
Dirk



___
Assp-test mailing list
Assp-test@lists.sourceforge.net

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[Thomas Eckardt]

>Is there logic to having a separate MaxBytes setting like 
MaxBytesForBombs that's used only during message delivery?  That way, the 
entire message can be scanned for bombs, but the rebuild could use a lower 
number to better balance the differential between the average sized spam 
and average sized not-spam message.

DID YOU EVER thougth about that ??? Or do you only write 
something to fillup the community mailing list?

No - no way!

Thomas







Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  10.11.2021 20:22
Betreff:Re: [Assp-test] Concept Question: Scan entire message for 
Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?



After about 12 weeks of going from MaxBytes of 4k to MaxBytes of 50k, 've 
seen:
1) Rebuild go from just over an hour (with 30k MaxFiles) to just over 2 
hours.  I'm fine with that, there's more to scan
2) Bomb detections improve, as a lot of what's detected is beyond the 20k 
or 30k mark
3) but, bayesian false positives going way up.  Lots of mail that would 
have (correctly) been delivered, is now getting too high of a score and is 
blocked.

Surely #3 is specific to the types of messages my users are getting and I 
can tweak settings.  BUT, it makes me raise this question again:
Is there logic to having a separate MaxBytes setting like MaxBytesForBombs 
that's used only during message delivery?  That way, the entire message 
can be scanned for bombs, but the rebuild could use a lower number to 
better balance the differential between the average sized spam and average 
sized not-spam message.



On Mon, Nov 1, 2021 at 2:43 PM K Post  wrote:
When looking at the "Use this HTML Parser" section on the GUI, I found 
this line:
it is recommended to set MaxBytes to 5 (be carefull on heavy load 
systems - spam bomb regular expressions will take longer using 5!).\
I'm going to change my settings and see how bad the rebuild time is.  I've 
got enough processing power and RAM now, but the disks aren't SSD.  Just a 
4 disk Raid 1+0 traditional HDD setup.  We'll see...

Since HTMl email accounts for a big percentage of all mail,  might it be a 
good idea to update/expand the guidance in the MaxBytes section of the 
GUI?   



On Fri, Oct 29, 2021 at 8:40 PM K Post  wrote:
Summary:
Should/could any consideration be given to having ASSP scan the entire 
message at the time it is received for Bombs (only), while still using 
MaxBytes for Bayesian/HMM?

We've been having some cleverly crafted messages slipping through all 
filters that would be easy to catch with Bombs if only the catchable 
content came before MaxBytes.  These messages are 20kb+, They have a scam 
phone number at the very end of the larger than MaxBytes messages.  I 
want/need to use bombs to catch the scam phone numbers.

With MaxBytes set to 3000, which is useful for faster RebuildSpamDB, these 
BombDataRE matches just aren't being caught.  If I increase MaxBytes, my 
BombDataRE catches them, but then rebuildspamdb is (probably? see below) 
longer than it needs to be.

So, is there any value in considering a MaxBytesAdditionalForBombs 
variable which would be added to MaxBytes and only used when scanning for 
bombs as messages arrive?   Would that kill performance??  Other 
downsides?

We could still only look at MaxBytes for Bayesian/HMM since it's only 
MaxBytes used when building those databases.

What do you think?

And while we're talking MaxBytes:
I've asked this before, is the guidance for 3kb for MaxBytes once there's 
a mature corpus still a valid recommendation?  With unlimited horsepower 
and ram, sure, why not, do 30kb or 100kb.  That's not my reality, so I 
want to see where to best allocate resources. If 3kb is still the 
guidance, even though the spam files I'm seeing have a median size around 
20kb, so be it.  I feel like when that guidance was written, html wasn't 
used as prolifically in spam.  The median size of notspam in my corpus is 
about 40kb.  That's determined unscientifically by sorting by size and 
scrolling to approximately half way down.

Thanks.  Have a good weekend.
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[Thomas Eckardt]

>are you saying that BombRe will look at headers that ASSP ads

No, it looks only in to the original header.

>I'm still worried about fake/invalid DKIM still getting the bonus score,

Invalid DKIM signatures should be blocked or scored very high, so the 
bonus score does not matter

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  09.11.2021 05:53
Betreff:Re: [Assp-test] Another Concept Question: 
DKIMBousScoreList



ah, wait, are you saying that BombRe will look at headers that ASSP ads, 
like X-ASSP-DKIM-Identity (which would only be added for a valid 
signature)?   (!)  I always assumed that the bomb functionality was 
only on the mail's original headers.

On Mon, Nov 8, 2021 at 2:28 PM K Post  wrote:
The bombHeaderRe with the DEFINE or list should be sufficient.  I'm still 
worried about fake/invalid DKIM still getting the bonus score, but this 
will have to do.  Thanks.

On Mon, Nov 8, 2021 at 12:01 PM Thomas Eckardt  wrote:
I told you to score such domains elsewhere - just do it and the result is 
the same like you wanted. 

for example: 

bombHeaderRe: 

\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(
The_Wanted_IDENTITY))\;=>the_wanted_negative_score

currently the (?(DEFINE)...) is not working with assp (is destroyed if 
a-d-n-o-r is not set for the file) - but the next version will do it  -   
and you can use: 

(?(DEFINE)(?the_wanted_identity
|ident2|ident3|..))\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= 
\;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(?))\;=>
the_wanted_negative_score - e.g. -10 
(?(DEFINE)(?the_wanted_identity
|ident5|ident6|..))\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= 
\;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(?))\;=>
the_wanted_negative_score - eg -20 
... 

CLOSED for me 


Thomas 



Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:05.11.2021 20:03 
Betreff:Re: [Assp-test] Another Concept Question: 
DKIMBousScoreList 



Having the dkimBonusScoreList would be like applying 
dkimBonusValancePB but ONLY for those that DKIM validate AND are on the 
scorelist.  Here's why I think that would be helpful and what you proposed 
could be problematic.  Essentially: I'm thinking: "look, this organization 
usually sends good stuff, but not always.  They might also have people 
sending non-dkim signed messages through a myriad of channels.  Deal with 
them separately, but if we KNOW it's from them because of their DKIM 
signature, help that message get through with the idea that it'll be 
stored in okmail unless whitelisted through something other than dkim." 

> there is already dkimOkValencePB - increase it 
But a high percentage of all messages that are received, spam and not, 
have valid signatures.  I don't think we should use that to give a bonus 
regardless of who the signer is.  All gmail messages are signed, almost 
everyting from office365.  Yes, I could do a univieral bonus then reduce 
gmail and onmicroosft.com, but that doesn't get 365 users with their own 
signatures and all of the millions of other domains out there.   

It was one thing when DKIM signing was a new concept and only legit 
businesses signed messages.  Now that most senders are signing, giving  a 
bonus would let an awful lot of spam slip through under the rejection 
scoring threshold.  

>reduce the score for certain domains by blackListedDomains, SenderBase or 
anywhere else - if needed 
Senderbase won't work for those using AWS as an example - too many 
spammers use them, so adding to senderbase can't be negated using 
blacklist/bombs, etc because I obviously don't know all of the bad senders 
using AWS. 

I could reduce the score based on a BombRe match on squaremktg, but then 
I'm reducing when I haven't validated the signature.  It would probably 
work for this specific example, but it would be generally helpful to be 
able to reduce the score on a message based solely on the signature when 
I'm sure they're actually the sender   Dare I say that I'm in love with 
DKIM?  

Would it be life changing like DoDKIMWLAddresses?  No absolutely not, but 
if it's not a major task to add the functionality, I think there would be 
wide appeal.   

I >>almost<< want to suggest that the dkimBonusValancePB feature be 
removed altogether.  I can't think of a scenario where you'd want to give 
a bonus universally just because a message has a valid signature from 
anyone.  Same thing for the SPF pass bonus and it's default of -10!!!  I'm 
sure there are people using one or both, I just can't think of a 
scenario in which it's a good idea. 




On Fri, Nov 5, 2021 at 10:37 AM Thomas Eckardt  wrote: 
Another useless post about concepts without reading the manual. 

>dkimBonusValancePB 

there is already dkimOkValencePB - increase it 

and 

reduce the score for certain domains by blackListedDomains, Sen

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[Thomas Eckardt]

I told you to score such domains elsewhere - just do it and the result is 
the same like you wanted.

for example:

bombHeaderRe:

\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(
The_Wanted_IDENTITY))\;=>the_wanted_negative_score

currently the (?(DEFINE)...) is not working with assp (is destroyed if 
a-d-n-o-r is not set for the file) - but the next version will do it  - 
and you can use:

(?(DEFINE)(?the_wanted_identity
|ident2|ident3|..))\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= 
\;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(?))\;=>
the_wanted_negative_score - e.g. -10
(?(DEFINE)(?the_wanted_identity
|ident5|ident6|..))\nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= 
\;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(?))\;=>
the_wanted_negative_score - eg -20
...

CLOSED for me


Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  05.11.2021 20:03
Betreff:Re: [Assp-test] Another Concept Question: 
DKIMBousScoreList



Having the dkimBonusScoreList would be like applying 
dkimBonusValancePB but ONLY for those that DKIM validate AND are on the 
scorelist.  Here's why I think that would be helpful and what you proposed 
could be problematic.  Essentially: I'm thinking: "look, this organization 
usually sends good stuff, but not always.  They might also have people 
sending non-dkim signed messages through a myriad of channels.  Deal with 
them separately, but if we KNOW it's from them because of their DKIM 
signature, help that message get through with the idea that it'll be 
stored in okmail unless whitelisted through something other than dkim."

> there is already dkimOkValencePB - increase it
But a high percentage of all messages that are received, spam and not, 
have valid signatures.  I don't think we should use that to give a bonus 
regardless of who the signer is.  All gmail messages are signed, almost 
everyting from office365.  Yes, I could do a univieral bonus then reduce 
gmail and onmicroosft.com, but that doesn't get 365 users with their own 
signatures and all of the millions of other domains out there.  

It was one thing when DKIM signing was a new concept and only legit 
businesses signed messages.  Now that most senders are signing, giving  a 
bonus would let an awful lot of spam slip through under the rejection 
scoring threshold. 

>reduce the score for certain domains by blackListedDomains, SenderBase or 
anywhere else - if needed
Senderbase won't work for those using AWS as an example - too many 
spammers use them, so adding to senderbase can't be negated using 
blacklist/bombs, etc because I obviously don't know all of the bad senders 
using AWS.

I could reduce the score based on a BombRe match on squaremktg, but then 
I'm reducing when I haven't validated the signature.  It would probably 
work for this specific example, but it would be generally helpful to be 
able to reduce the score on a message based solely on the signature when 
I'm sure they're actually the sender   Dare I say that I'm in love with 
DKIM? 

Would it be life changing like DoDKIMWLAddresses?  No absolutely not, but 
if it's not a major task to add the functionality, I think there would be 
wide appeal.  

I >>almost<< want to suggest that the dkimBonusValancePB feature be 
removed altogether.  I can't think of a scenario where you'd want to give 
a bonus universally just because a message has a valid signature from 
anyone.  Same thing for the SPF pass bonus and it's default of -10!!!  I'm 
sure there are people using one or both, I just can't think of a 
scenario in which it's a good idea.




On Fri, Nov 5, 2021 at 10:37 AM Thomas Eckardt  wrote:
Another useless post about concepts without reading the manual. 

>dkimBonusValancePB 

there is already dkimOkValencePB - increase it 

and 

reduce the score for certain domains by blackListedDomains, SenderBase or 
anywhere else - if needed 

Thomas 





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:04.11.2021 22:38 
Betreff:[Assp-test] Another Concept Question: DKIMBousScoreList 




SUMMARY: Would there be benefit (that wouldn't be terrible to code) in 
adding the ability for use to assign a score to emails that match a list 
of DKIM signature identities?  


The DKIMWLAddress and DKIMNPAddress functionality has been an absolute 
game changer here.  Thank you so much for implementing that (it was my 
idea, but we all know that I could never code such a thing). 

I've combined that functionality with closely monitored SenderBase lists 
to dramatically improve ASSP's accuracy. 

One place where Senderbase shines is it's scoring ability for bulk 
senders.  For example, I can give anything that Senderbase says is coming 
from constant contact's network a -10 score, by adding it into 
whiteSenderBase like 
^constantcontact\.com$=>-10   
I don't want to blindly l

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[Thomas Eckardt]

Another useless post about concepts without reading the manual.

>dkimBonusValancePB

there is already dkimOkValencePB - increase it

and

reduce the score for certain domains by blackListedDomains, SenderBase or 
anywhere else - if needed

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  04.11.2021 22:38
Betreff:[Assp-test] Another Concept Question: DKIMBousScoreList




SUMMARY: Would there be benefit (that wouldn't be terrible to code) in 
adding the ability for use to assign a score to emails that match a list 
of DKIM signature identities? 


The DKIMWLAddress and DKIMNPAddress functionality has been an absolute 
game changer here.  Thank you so much for implementing that (it was my 
idea, but we all know that I could never code such a thing).

I've combined that functionality with closely monitored SenderBase lists 
to dramatically improve ASSP's accuracy.

One place where Senderbase shines is it's scoring ability for bulk 
senders.  For example, I can give anything that Senderbase says is coming 
from constant contact's network a -10 score, by adding it into 
whiteSenderBase like
^constantcontact\.com$=>-10  
I don't want to blindly let through constant contact signed messages, but 
if it's coming from their network, make it a little easier for messages to 
pass through. That's worked well for a long long time.


Recently, I'm seeing several bulk senders having legitimate messages DKIM 
signed by the bulk sender them, but being sent through Amazon AWS (
amazonses.com) and is classified by senderbase as being Amazon / 
amazonses.com.  There's a lot of volume coming in from amazonses.com, but 
unfortunately, it's a mix of perfectly legitimate messages and others that 
are pure garbage.  So that takes Senderbase off the table.  Coming from 
amazonses shouldn't impact the score either way.  And I can't 
DKIMWLAddress the signature, then bad stuff would absolutely get through.

An example is Square, the credit card processor and software company.  
They send mail, DKIM signed @squaremktg.com on behalf of clients.  Most 
mail from square is good, but sometimes it gets spammy, just like we see 
with mail from other bulk senders.  Real world, I paid for a car wash 
using their mobile payment platform, I received the receipt and later got 
an email with a promotion from the car wash.  All good.  The provider's 
signature was in DKIMWLAddresses.  Today, I received an advertisement from 
them for what is apparently a "gentleman's club" next door, offering a 
complimentary car wash (I took that literally) for visiting the 
establishment.  The language in that email would have absolutely had it 
rejected if it hadn't been on DKIMWLAddresses.  Worse, it wound up in the 
not-spam corpus.


So, I'd like for certain DKIM signatures to be able to SCORE.  DKIM 
scoring would help it get through (or make it harder depending on the 
score) without automatically passing it and adding it to the corpus like 
DKIMWLAddresses does.   That would let me give the message a negative 
score based on the DKIM but still let Bayesian/HMM and other features stay 
in play to score the message further.

Conceptually, I could see this working similarly to senderbase.  There 
would be a default valance like
dkimBonusValancePB 
set to a default of -25

Then we'd have a list, maybe called DKIMBousScoreList.  Like 
DKIMWLAddresses, it would match the end of the validated DKIM identity, 
but also accepts a score override:
(@|.)squaremktg.com<--- gets the default of -25
(@|.)someUsuallyOKsigner.com=>-12<-- gets -12 for a score
(@|.)prettygood.com=>5<--- gets 1/5 of the default 
-25   -25/5 = -5
(@|.)UsuallyBad.com=>-5  <-- this isn't a bonus, a 
negative default divided by a negative is a positive.  it will be -25/-5 
or adding 5 to the score  


>From a management standpoint, it would certainly be easier to "just" be 
able to assign an optional 2nd parameter to DKIMWLAddresses that would 
score instead of whitelisting, but I feel like that could be too big of a 
coding project.

I tried to come up with a way to accomplish the same thing based on DKIM 
signature, but came up very short.  I know I could ignore DKIM and just 
score based on the from line, but I really appreciate the certainty that 
DKIM gives that the message is really from that organization.

What do you think?  Would a  DKIMBousScoreList feature have universal 
appeal?

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!

Re: [Assp-test] RegEx Backreferences - the basics

[Thomas Eckardt]

timization might break the 
2nd parameter from being recognized.   Am I doing something very wrong or 
is this a bug?



And now for the regex I've come up with as a new starting point:


DISCLAIMER TO ANYONE READING THIS IN THE FUTURE - while this seems to work 
for me, it's surely at least imperfect if not horribly inefficient or even 
wrong or broken!!!

Here's the regex I've built.  It seems to work in ASSP and  test properly 
at https://regextr.com with PCRE selected as the engine and the case 
insensitive flag ticked

(?:^|\r?\n)(?:to:(?:.*?[\s\<])*?(?[a-z\d\-]+)\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n(?:.+\r?\n)*?from:.*?\@(?:[a-z\d\-]+\.)*?\g{TOFirstMatch}\.[a-z]{2,6}\>?\r?\n|from:.*?\@(?:[a-z\d\-]+\.)*?(?[a-z\d\-]+)\.[a-z]{2,6}\>?\r?\n(?:.+\r?\n)*?to:(?:.*?[\s\<])*?\g{FROMFirstMatch}\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n)


This appears to match:

x-whatever: bla bla
to: "my name" 
subject: testing
from: "them" 
asdf
and with the from appearing before the to.

I do not know of a way to make the order of to and from insignificant, so 
I've done an "or" in between the first part of the regex which looks for 
to then from and the second part which looks for from then to.   Would it 
be more efficient for ASSP to have 2 separate lines, one for to first the 
other for from first?

Here's my thinking and explanation of my understanding of the regex that I 
wrote. I am VERY interested in corrections and suggestions for 
improvement, especially relating to efficiency (and obviously flawed logic 
and/or cases where what I've done would or wouldn't match as I'm 
thinking).  Guidance here won't only help me perfect this specific regex 
for ASSP use, but will hopefully help others looking for other more 
complex than typical regex help with ASSP.  I'll definitely be limiting 
the to domains to those that we use here to speed this up a bit, but I 
kept it more generic here.

I also tried to see a way where lookaheads might help, but I'm not quite 
there yet  Would they be helpful here?

Starting from the beginning:

(?:^|\r?\n)
start with either the start of the string or a \r?\n   - sometimes there's 
a \r but always a \n Is \r?\n recommended?  Is there a better way? 

Then we're going to do 2 big OR's,  first looking for to then from, then 
from then to.
(?: starts this big or, with the ?: indicating that it's a non-capturing 
group

The TO then From part is this:
to:(?:.*?[\s\<])*?(?[a-z\d\-]+)\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n(.+\r?\n)*?from:.*?\@(?:[a-z\d\-]+\.)*?\g{TOFirstMatch}\.[a-z]{2,6}\>?\r?\n

broken out

to:  Find to:  immediately after the previously found newline or start of 
string)

(?:.*?[\s\<])*?
non-capturing match for any characters repeated as long as they end with a 
space or <

now we should be at the point where the username starts

(?[a-z\d\-]+)\@
get a named match called TOFirstMatch for any a-z number - combination 
that ends in the now escaped @

(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n
then just make sure that what follows the @ is a-z decimal and dahes, each 
part ending in a . with a 2-6 letter TLD ending the hostname followed by 
an optional > and then \n or \r to end the line



(?:.+\r?\n)*?
then ignore future lines which aren't blank until we a line starting with 
from:

from:.*?
line stars with from: followed by any characters

\@(?:[a-z\d\-]+\.)*?
find @valid.sub. part of from address

\g{TOFirstMatch}
use the \g{} syntax to match the named backreference

\.[a-z]{2,6}\>?\r?\n)
immediately followed by .tld 2-6 characters in length, an optional > and a 
\n or \r

|
then an OR

and we do the whole thing again but with From First
from:.*?\@(?:[a-z\d\-]+\.)*?(?[a-z\d\-]+)\.[a-z]{2,6}\>?\r?\n(?:.+\r?\n)*?to:(?:.*?[\s\<])*?\g{FROMFirstMatch}\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n)

from:.*?
from: followed by anything until we hit

\@(?:[a-z\d\-]+\.)*?
and @ sign followed by any number of hostname followed by .

(?[a-z\d\-]+)
find the second level domain name and call is FROMFirstMatch

\.[a-z]{2,6}\>?\r?\n
followed by a .tld of 2 to 6 characters, an optional closing > and a \n or 
\r

(?:.+\r?\n)*?
move past non blank lines until we hit

to:(?:.*?[\s\<])*?
to: optionally followed by whatever characters ending in space or <


\g{FROMFirstMatch}\@
now look for the second level domain match from the from: line immediately 
followed by an @ sign

(?:[a-z\d\-]+\.)+
then hostnames separated by dots, at least 1

[a-z]{2,6}\>?\r?\n)
followed by a 2-6 character tld, an optional > and a \n or \r?

)
closing out the or between the MatchToFirst and FROMFirstMatch sections.


Whew.
:


On Thu, Nov 4, 2021 at 4:53 AM Thomas Eckardt  
wrote:
forgot to say: 

if assp requires to capture the match for a regex, the code would be for 
example 

$string =~ /($testReRE)/ 
$match = $1;

so - at runtime the regex is 

((?^u:(?is:(?:^|\n\r).*(searchstring).*@.*\1.*))) 

IMHO you need to use named

Re: [Assp-test] RegEx Backreferences - the basics

[Thomas Eckardt]

>>If "searchstring" is to the right and left of an @ sign, it should 
match.

After having some time while driving a car today - my 2 cents for using a 
named  capture group and backreference ? \k -  instead 
of numbered  (...) \1

(?searchstring)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k

When ever possible don not use   .*  - instead look for a less greedy 
variant. .* with the (?is:  switch (assp uses it) will force the regex 
engine to search until the end of the string (mail) and to trace back 
(several thousand times).
There are better (less CPU, less time) solutions for the regex using 
lookahead assertations - but this will go beyond the scope.


There is no need to disable regex-optimization anyway - because there is a 
\r in the regex. The optimizer has a bug, which destroys the \r to \\r - 
so assp disables the optimizer if a regex contains a \r .
How ever, to prevent the regex from failing after possible future assp 
code changes, you may use:

<<<(?:(?searchstring)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k)>>>
or

(?searchstring)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k.?

the later one seems to be the best of both


normaly you would write

<<<(?searchstring)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k>>>

but assp will destroy the trailing four   (I'm sure)


If more than one string is searched - use:

~<<<(?:(?searchstring1|searchstring2|searchstring3)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k)>>>~
or
~<<<(?searchstring1|searchstring2|searchstring3)[^@\r\n]{0,64}\@[^@\r\n]{0,64}\k.?>>>~


Want to know more about regular expressions?  ->  
http://www.rexegg.com/regex-lookarounds.html


Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  04.11.2021 02:29
Betreff:[Assp-test] RegEx Backreferences - the basics



I've got nothing in my TestRe file except for a single line:

~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~

The idea is to log any time there's a line that includes "searchstring" on 
the right and left of an @.  This is just a very rudimentary test because 
backreferences seem to error for me.  I would expect this to match
searchstring@searchstring
something else seachstring more @ whatever searchstring bla
If "searchstring" is to the right and left of an @ sign, it should match.  
Regex101.com seems to confirm that this works.  Like I said, super basic.

However, if I enter ~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~ as the 
only line in TestRe file, I get a warning in the log:

- Reference to nonexistent group in regex; marked by <-- HERE in 
m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/ 
- try using unoptimized regex

To my understanding, the <<< >>> surround should turn of regex 
optimization for that line, which enables backreferencing (\1) to work and 
the ~ is required because there's an or in there.   Shouldn't the \1 
reference (searchstring) ?  I don't understand why assp thinks that \1 is 
a reference to a non-existent group.

I also tried removing the <<< >>> and adding assp-do-not-optimize to the 
top of the TestRe file.  No difference.No matter how simple I make the 
regex, even (.*)@\1,  it still complains about the invalid backreference.
 

I've got to be missing something incredibly obvious.  I've read through 
the regex doc in docs, but that doesn't talk about backreferencing in ASSP 
and I can't find anything in the GUI that makes mention. I've seen posts 
here indicating that backreferencing matches is possible with an 
unoptimized expression.
  
A shove in the right direction would be greatly appreciated.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] RegEx Backreferences - the basics

[Thomas Eckardt]

forgot to say:

if assp requires to capture the match for a regex, the code would be for 
example

$string =~ /($testReRE)/
$match = $1;

so - at runtime the regex is

((?^u:(?is:(?:^|\n\r).*(searchstring).*@.*\1.*)))

IMHO you need to use named capture groups or \g or (?|

Thomas



Von:"Thomas Eckardt" 
An: "ASSP development mailing list" 
Datum:  04.11.2021 09:22
Betreff:Re: [Assp-test] RegEx Backreferences - the basics



to make backreferences working, regex optimization must be switched off 
for the complete regex -> tested -> worked 

>I've seen posts here indicating that backreferencing matches is possible 
with an unoptimized expression. 

so - the problem is sitting in front of the monitor :):) 

m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/  

optimized - default is : 'no extra group capturing is allowed' 

>I've got to be missing something incredibly obvious. 

assp-do-not-optimize-regex

>  (?:^|\n\r).*(searchstring).*@.*\1.* 

assp makes it: 

(?is:(?:^|\n\r).*(searchstring).*@.*\1.*) 

think about your regex - read it from left to right as 'perl regex engine' 
- what will happen? 
beside the other mistakes the @ should be escaped  \@ , because an ARRAY 
@. may exist 

>Regex101.com seems to confirm that this works. 

does not check perl pcre 

and if I read the explanation there, I sure it will not work like you 
expect 


Thomas 



Von:"K Post"  
An:"ASSP development mailing list" 
 
Datum:04.11.2021 02:29 
Betreff:[Assp-test] RegEx Backreferences - the basics 



I've got nothing in my TestRe file except for a single line: 

~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~ 

The idea is to log any time there's a line that includes "searchstring" on 
the right and left of an @.  This is just a very rudimentary test because 
backreferences seem to error for me.  I would expect this to match 
searchstring@searchstring 
something else seachstring more @ whatever searchstring bla 
If "searchstring" is to the right and left of an @ sign, it should match. 
Regex101.com seems to confirm that this works.  Like I said, super basic. 

However, if I enter ~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~ as the 
only line in TestRe file, I get a warning in the log: 

- Reference to nonexistent group in regex; marked by <-- HERE in 
m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/ 
- try using unoptimized regex 

To my understanding, the <<< >>> surround should turn of regex 
optimization for that line, which enables backreferencing (\1) to work and 
the ~ is required because there's an or in there.   Shouldn't the \1 
reference (searchstring) ?  I don't understand why assp thinks that \1 is 
a reference to a non-existent group. 

I also tried removing the <<< >>> and adding assp-do-not-optimize to the 
top of the TestRe file.  No difference.No matter how simple I make the 
regex, even (.*)@\1,  it still complains about the invalid backreference. 
 

I've got to be missing something incredibly obvious.  I've read through 
the regex doc in docs, but that doesn't talk about backreferencing in ASSP 
and I can't find anything in the GUI that makes mention. I've seen posts 
here indicating that backreferencing matches is possible with an 
unoptimized expression. 
 
A shove in the right direction would be greatly appreciated. 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] RegEx Backreferences - the basics

[Thomas Eckardt]

to make backreferences working, regex optimization must be switched off 
for the complete regex -> tested -> worked

>I've seen posts here indicating that backreferencing matches is possible 
with an unoptimized expression.

so - the problem is sitting in front of the monitor :):)

m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/ 

optimized - default is : 'no extra group capturing is allowed'

>I've got to be missing something incredibly obvious.

assp-do-not-optimize-regex

>  (?:^|\n\r).*(searchstring).*@.*\1.*

assp makes it:

(?is:(?:^|\n\r).*(searchstring).*@.*\1.*)

think about your regex - read it from left to right as 'perl regex engine' 
- what will happen?
beside the other mistakes the @ should be escaped  \@ , because an ARRAY 
@. may exist

>Regex101.com seems to confirm that this works.

does not check perl pcre

and if I read the explanation there, I sure it will not work like you 
expect


Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  04.11.2021 02:29
Betreff:[Assp-test] RegEx Backreferences - the basics



I've got nothing in my TestRe file except for a single line:

~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~

The idea is to log any time there's a line that includes "searchstring" on 
the right and left of an @.  This is just a very rudimentary test because 
backreferences seem to error for me.  I would expect this to match
searchstring@searchstring
something else seachstring more @ whatever searchstring bla
If "searchstring" is to the right and left of an @ sign, it should match.  
Regex101.com seems to confirm that this works.  Like I said, super basic.

However, if I enter ~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~ as the 
only line in TestRe file, I get a warning in the log:

- Reference to nonexistent group in regex; marked by <-- HERE in 
m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/ 
- try using unoptimized regex

To my understanding, the <<< >>> surround should turn of regex 
optimization for that line, which enables backreferencing (\1) to work and 
the ~ is required because there's an or in there.   Shouldn't the \1 
reference (searchstring) ?  I don't understand why assp thinks that \1 is 
a reference to a non-existent group.

I also tried removing the <<< >>> and adding assp-do-not-optimize to the 
top of the TestRe file.  No difference.No matter how simple I make the 
regex, even (.*)@\1,  it still complains about the invalid backreference.
 

I've got to be missing something incredibly obvious.  I've read through 
the regex doc in docs, but that doesn't talk about backreferencing in ASSP 
and I can't find anything in the GUI that makes mention. I've seen posts 
here indicating that backreferencing matches is possible with an 
unoptimized expression.
  
A shove in the right direction would be greatly appreciated.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21306

[Thomas Eckardt]

Hi all

fixed in assp 2.6.6 *SPAM-Evaporator* build 21306:

- An email, starting with an invalid MIME header, sent to the analyzer - 
prevented the analyzer from finding the subject of the mail.

- If the same regular expression was used with different weights, tagged 
by different 'NWLI' extensions, the regex itself was defined and executed 
multiple times.


changed:

- GUI help changed for
AddDKIMHeader
EmailSpam
EmailHam
NWLI

- If ReportLog is set to diagnostic and a .msg (outlook OLE) file is 
reported, which starts with invalid header content, this invalid content 
will be corrected and the original .msg file
  and the converted .eml file are stored in the assp/debug folder. So 
possible problems with outlook reporting can be better solved.
  .msg files can be converted to the MIME content (.eml) using the 
following command.
 
  Linux/nix: perl -e 'use Email::Outlook::Message; print 
Email::Outlook::Message->new(q(FILENAME))->to_email_mime->as_string;'
  Windows: perl -e "use Email::Outlook::Message; print 
Email::Outlook::Message->new(q(FILENAME))->to_email_mime->as_string;"

  Replace FILNAME with the name of the .msg file. The MIME content is 
written to STDOUT - ' >file.eml' at the end of the command will write the 
MIME content to an output file.


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[Thomas Eckardt]

>But isn't the message that's attached as a .msg file fully saved, intact, 
in the appropriate corrected corpus folder? 

Just compare the original received and stored .eml file with the one 
stored by the report (.rpt.eml). If they are identical (except some added 
header lines in front by the report) - everything is working well.

>Zipping works 100%. 

In case of zip assp does nothing else than unzip and process, instead of 
simply process attached files. So, if zip works and not zip does'nt, the 
attached .msg or .eml files are not identical for both cases (but they 
should be).

And for the records : the analyzer code is the same for all cases.

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  01.11.2021 11:33
Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* 
build 21302



Outlook itself is a typo.  I think it's supposed to be called LookOut! 

-Using the outlook function "send email as attachment" or "forward email 
as attachment" has NEVER worked - because outlook destroys the original 
MIME header. But the original header is particularly important for 
analysis systems!

But isn't the message that's attached as a .msg file fully saved, intact, 
in the appropriate corrected corpus folder?   I'm using subject names for 
file names and with 21302 the files names no longer are the subject of the 
.msg, but with 21293 it was.  The content of the saved file is still 
correct though.  So why can't ASSP use that for the analysis?

The analysis on those saved files go off without an error.  It's only the 
analysis that's triggered at the time the correction report is emailed 
that's misfiring.  If outlook is destroying the mime header of the 
original .msg file, could whatever ASSP is doing to that be used then the 
analyze report runs?   Should/could the code be changed to instead run the 
newly saved report file (or it's contents that may already be in a 
variable) through analyze?

I've done the best I can to interpret what the ASSP code's doing. To me, 
it doesn't look like the ConfigureAnalyze that runs is done on the saved 
file, rather it's done looking at least at the headers of the report email 
itself (vs the reported email).  ASSP seems to yell when it hits a tab in 
the header of the report email, NOT the header of the .msg file that's 
attached, but the header of the report itself.  If I change the TO line of 
the report email to be short enough (demonstrated in the previous message 
in this thread), assp doesn't complain about the to line anymore, instead 
it complains when it hits the next tab.  In the example, that was when it 
hit one of the attachment boundaries.

With Outlook doing line continuations with a tab, shouldn't ASSP account 
for that even with report emails?  It seems to handle it just fine 
everywhere else.  I read through parts of RFC822 and it seems to suggest 
that a tab is a valid line continuation method.  ASSP's headerUnwrap seems 
to account for this just fine, but maybe that's not being used with the 
analyze?


We're on the same page about Outlook being an evil beast.  Unfortunately, 
it is by far the most common mail client in use, so I've got to deal with 
it.  What I'm trying to accomplish is having a reliable and easy way for 
all staff to report.  Forwarding as an attachment from Outlook does seem 
to work, just not once the additional analysis report is run.  

- the mail is to be exported (.msg or .eml) and then (possibly zip) sent 
to the system
Zipping works 100%.  Saving message as msg then attaching doesn't seem to 
be any different from forward as attachment

- or to attach the mail to a new mail via drag and drop (this may not work 
in every case and every outlook versions)
No difference here, at least with Outlook 2019

- or to use a plugin which provides any of the both options (
https://sourceforge.net/projects/assp/files/assp_mail_client_plugins/
Unfortunately I don't have access to all end user's machines, especially 
with the pandemic having so many users with email on personal devices.  I 
wish we could provide charity owned laptops for home use, but I can't even 
get budget approval for a new desk chair.

I've got to keep it easy easy easy for end users to report.  No 2 or 3 
step processes, no plugins required.  Just forward as attachment.  That 
works (even with 21302's wrong analyze warnings and file name mangling).  
It would just be nice to get propper analyze reports to make it easier for 
me to stay on top of what's being reported.


Another thing that doesn't seem to work is the handling of multiple .msg 
attachments in one report. That happens if a user selects several messages 
and does a forward as attachment.  That's never worked, but if when you 
have the time and energy to review what's going on with the above, maybe 
you could also consider fixing (adding?) this functionality?

...as always...  Thank you
Ken

 
 

On Sun, Oct 31, 2021

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[Thomas Eckardt]

It's always been like this - outlook forwards WRONG. To send a mail to an 
SMTP-based analysis system (this applies to all, not just ASSP) for 
analysis purposes:

- the mail is to be exported (.msg or .eml) and then (possibly zip) sent 
to the system
- or to attach the mail to a new mail via drag and drop (this may not work 
in every case and every outlook versions)
- or to use a plugin which provides any of the both options (
https://sourceforge.net/projects/assp/files/assp_mail_client_plugins/) 

Using the outlook function "send email as attachment" or "forward email as 
attachment" has NEVER worked - because outlook destroys the original MIME 
header. But the original header is particularly important for analysis 
systems!


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[Thomas Eckardt]

 out loud here.  I'm cool with everything on, but maybe there are 
others who would prefer to more granularly configure?


related: GUI mistake.  the AddDKIMHeader description still says that it 
adds X-ASSP-DKIM: instead of "X-ASSP-DKIMidentity


DoRBBlack removal of deny matches --  curiosity:
For the new DoRBBlack, why is it checking denySMTPConnectionsFromAlways 
and denySMTPConnectionsFrom?  Aren't additions made to that list after 
we've collected what we've wanted (good or bad) from those IP's / emails 
which would be good to have in the corpus?

NWLI
I'd like to rewrite the NWLI description at the bottom of the GUI, but I 
need clarification first.  I'm sure NWLI functionality works in the code, 
it's just not explained well in the GUI.

I see the revised language, but I'm still not sure that I follow.  When 
you say "optional to use '+'" do you mean something N and N+ are 
functionally identical?  If that's true, why bother having the + as an 
option at all?  If there's a difference between having a plus and not, 
please explain.  

For starters, you have

"So the line ~Heuristics|Email~=>50:>N-W-LI could be read as: take the 
regex with a weight of 50, never score noprocessing mails, never score 
whitelisted mails, score local mails and mails from ISP's."

But if it's ANDed together, it would really mean
score 50 to a mail that matches Heuristics|Email when that mail is NOT 
noprocessing, is NOT whitelisted, IS local AND IS ALSO is an ISP mail.  
To me, the way you have it written implies that it would score 50 if it's 
either local OR ISP as long as it's neither whitelisted nor noprocessing.



Also, parameters are separated everywhere else that I see by => but the 
third parameter here needs to be :>   My lousy eyes missed that until just 
now.  Why the the inconsistency?





Other thanks, notes, and reqests
Thanks for adding that explanation bit into the GUI and the icon!  Also, 
this:
Info: file D:/assp/IP-Lists/IPS-facebookmail.com.cfg is now stored 
encrypted, because it is used in secured config Groups Excellent

The ReportLog addition will be really helpful, even if just for periodic 
reviews of user submitted reclassifications vs ones I've done through the 
GUI.   Could we have it use the same file extension as we use for the 
corpus?  (maillogExt)

In the code, in some places X-Assp- headers are referenced, others have 
X-ASSP-   All of the compares I've seen appear to be case insensitive, but 
you might consider standardizing so that type-a personalities can be 
calmed :)




On Fri, Oct 29, 2021 at 10:31 AM Thomas Eckardt <
thomas.ecka...@thockar.com> wrote:
Hi all, 

fixed in assp 2.6.6 *SPAM-Evaporator* build 21302: 

- Improved email address detection in the emailinterface list reports 
(whitelistadd , whitelistremove, ). 

- The change time for include files used in the 'Groups' feature were not 
recorded in workers. This caused unexpected configuration reloads in the 
workers, until 
  assp was restarted. 

- Any change made for 'Groups' caused a reload for all configuration 
parameters where a group was used, even the related group was not changed. 
A configuration reload is now 
  only done for changed groups and there related configuration parameters. 


- Unexpected results were produced by the analyzer, if emails were sent as 
(not zipped) attachment to the emailinterface for analyzing - using 
outlook as mail client (+exchange). 
  Notice: always compress (e.g. zip) reported emails before they are sent 
to assp! 


changed: 

- If the hidden parameter 'DoRBWhite' is set, the rebuildspamdb process 
searches for matches in 
  whiteRe, npRe, whiteListedDomains, noProcessingDomains, whiteListedIPs, 
noProcessingIPs, DKIMWLAddresses and DKIMNPAddresses - 
  and removes those mails from the assp/spam folder. 


- 'ReportLog','Enable Report logging' 
  'If set to diagnostic, each received report mail will be stored in the 
assp/debug folder.' 
   This makes it more easy to track down report problems, based on the 
data sent by the mail client to assp. 

- The GUI description for the NWLI enhancement (for regular expressions) 
was updated. The code was changed to get the NWLI results exactly like 
descriped in the GUI. 

- A hint (and context help) about encryped configuration parameters and 
files was added to GUI. 


added: 

The set hidden parameter 
DoRBBlack = 0;  # (0/1) check blacklisted mails on 
rebuildspamdb (default 0 - 1 = skip rebuild for notspam if black) 
removes all mails in the assp/notspam folder, which matches  : 
 noBlockingIPs, denySMTPConnectionsFromAlways, denySMTPConnectionsFrom and 
blackListedDomains 

Notice: if all of DoRBWhite, DoRBBlack and DoRBRed are enabled, the 
rebuild process will take ~12 times (or very much) longer than without 
setting these switches. 
Don't be confused. If .eml files were corrected by spam/ham 
reports, assp will process them correctly. But it m

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21302:

- Improved email address detection in the emailinterface list reports 
(whitelistadd , whitelistremove, ).

- The change time for include files used in the 'Groups' feature were not 
recorded in workers. This caused unexpected configuration reloads in the 
workers, until
  assp was restarted.

- Any change made for 'Groups' caused a reload for all configuration 
parameters where a group was used, even the related group was not changed. 
A configuration reload is now
  only done for changed groups and there related configuration parameters.

- Unexpected results were produced by the analyzer, if emails were sent as 
(not zipped) attachment to the emailinterface for analyzing - using 
outlook as mail client (+exchange).
  Notice: always compress (e.g. zip) reported emails before they are sent 
to assp!


changed:

- If the hidden parameter 'DoRBWhite' is set, the rebuildspamdb process 
searches for matches in
  whiteRe, npRe, whiteListedDomains, noProcessingDomains, whiteListedIPs, 
noProcessingIPs, DKIMWLAddresses and DKIMNPAddresses -
  and removes those mails from the assp/spam folder. 


- 'ReportLog','Enable Report logging'
  'If set to diagnostic, each received report mail will be stored in the 
assp/debug folder.'
   This makes it more easy to track down report problems, based on the 
data sent by the mail client to assp.

- The GUI description for the NWLI enhancement (for regular expressions) 
was updated. The code was changed to get the NWLI results exactly like 
descriped in the GUI.

- A hint (and context help) about encryped configuration parameters and 
files was added to GUI.


added:

The set hidden parameter 
DoRBBlack = 0;  # (0/1) check blacklisted mails on 
rebuildspamdb (default 0 - 1 = skip rebuild for notspam if black)
removes all mails in the assp/notspam folder, which matches  : 
noBlockingIPs, denySMTPConnectionsFromAlways, denySMTPConnectionsFrom and 
blackListedDomains

Notice: if all of DoRBWhite, DoRBBlack and DoRBRed are enabled, the 
rebuild process will take ~12 times (or very much) longer than without 
setting these switches.
Don't be confused. If .eml files were corrected by spam/ham 
reports, assp will process them correctly. But it may help to maintain the 
corpus from time to time.



Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[Thomas Eckardt]

 
0.199 seconds and 1 cycles - got (ok)
Oct-26-21 10:37:33 Info: Main_Thread got connection request
Oct-26-21 10:37:33 Info: Main_Thread freed by idle Worker_4 in 0.014 
seconds and zero cycles - got (ok)
Oct-26-21 10:37:33 Info: Main_Thread got connection request
Oct-26-21 10:37:33 Info: Main_Thread freed by idle Worker_6 in 0.014 
seconds and zero cycles - got (ok)
Oct-26-21 10:37:33 Info: Worker_3 is interrupted to get new connection
Oct-26-21 10:37:33 Connected: session:258FF548 216.160.207.10:42420 > 
a.b.c.60:25 > a.b.c.35:25
Oct-26-21 10:37:33 msg58965-29399 193.169.253.240 info: PB-IP-Score for 
'193.169.253.0' is 2268, added 126 in this session
Oct-26-21 10:37:33 msg58965-29399 193.169.253.240 disconnected: 
session:7327C910 193.169.253.240 - processing time 89 seconds
Oct-26-21 10:37:33 Worker_4 finished reloading configuration
Oct-26-21 10:37:33 Worker_4 will sleep now
Oct-26-21 10:37:33 Worker_6 will sleep now
Oct-26-21 10:37:33 Info: Worker_3 is interrupted to get new connection
Oct-26-21 10:37:33 Connected: session:8FC33A40 104.237.139.48:57300 > 
a.b.c.60:25 > a.b.c.35:25
Oct-26-21 10:37:33 Worker_4 wakes up
Oct-26-21 10:37:33 Info: Worker_4 got connection from MainThread

This type of behavior shouldn't be expected should it?






On Tue, Oct 26, 2021 at 6:01 AM Thomas Eckardt  wrote:
>I tried to see where SaveConfig() 

SaveConfig() is called in the Main_Thread (Worker_0) every time, anything 
in the assp configuration was changed. 
And under normal cirumstances, after that, all workers are told to check 
there configuration (and reload) in relation to the new one (lists, files, 
recompile regexes  ). 

> I >>think<< 

if I would have a problem with >"Saving config"< (which is a maintenance 
task) - I would increase the 'MaintenanceLog' level 
if I would have a problem with workers >"unable to transfer connection to 
any worker"< - I would enable WorkerLog and WorkerLogging

example log: 

Oct-25-21 23:12:58 [Worker_1] Downloading level-2-TLDlist via direct 
HTTP connection 
Oct-25-21 23:12:58 [Worker_1] Level-2-TLDlist download completed 
Oct-25-21 23:12:58 [Worker_1] Downloading level-3-TLDlist via direct 
HTTP connection 
Oct-25-21 23:12:59 [Worker_1] Level-3-TLDlist download completed 
Oct-25-21 23:12:59 [Worker_1] Info: next TLDlist download in 1 day 5 
hours 58 minutes 
Oct-25-21 23:12:59 [Worker_1] Info: file c:/assp/files/URIBLCCTLDS.txt 
updated for URIBLCCTLDS 
Oct-25-21 23:13:00 [Main_Thread] Saving config 
Oct-25-21 23:13:00 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file c:/assp/assp.cfg is unchanged 
Oct-25-21 23:13:00 [Main_Thread] Adminupdate: file 
'c:/assp/files/URIBLCCTLDS.txt' for config 'URIBLCCTLDS' was changed 
Oct-25-21 23:13:00 [Main_Thread] Option list file: 
'c:/assp/files/URIBLCCTLDS.txt' reloaded (URIBLCCTLDS) with 12,280 records 

Oct-25-21 23:13:02 [Worker_1] Worker_1 wakes up 
Oct-25-21 23:13:02 [Worker_5] Worker_5 wakes up 
Oct-25-21 23:13:02 [Worker_4] Worker_4 wakes up 
Oct-25-21 23:13:02 [Worker_3] Worker_3 wakes up 
Oct-25-21 23:13:02 [Worker_2] Worker_2 wakes up 
Oct-25-21 23:13:03 [Worker_1] Worker_1 finished reloading configuration 
Oct-25-21 23:13:03 [Worker_1] Worker_1 will sleep now 
Oct-25-21 23:13:04 [Worker_2] Worker_2 finished reloading configuration 
Oct-25-21 23:13:04 [Worker_2] Worker_2 will sleep now 
Oct-25-21 23:13:05 [Worker_3] Worker_3 finished reloading configuration 
Oct-25-21 23:13:05 [Worker_3] Worker_3 will sleep now 
Oct-25-21 23:13:06 [Worker_4] Worker_4 finished reloading configuration 
Oct-25-21 23:13:06 [Worker_4] Worker_4 will sleep now 
Oct-25-21 23:13:07 [Worker_5] Worker_5 finished reloading configuration 
Oct-25-21 23:13:07 [Worker_5] Worker_5 will sleep now 
Oct-25-21 23:13:08 [Worker_1] Worker_1 finished reloading 
configuration 
Oct-25-21 23:13:10 [Worker_10001] Worker_10001 finished reloading 
configuration 
Oct-25-21 23:13:29 [Worker_1] Downloading Extended Droplist via direct 
HTTP connection 
Oct-25-21 23:13:29 [Worker_1] Extended Droplist already up to date 
Oct-25-21 23:13:29 [Worker_1] Info: next droplist download in 7 hours 
34 minutes 


debug may help 
for time related debugging, I (or some one who read the manual) would 
consider to use ConfigChangeSchedule 

notice: analyzing all the produced debug files (in general debug mode) is 
a very time consuming task 

Check your option files for bad (too greedy) regular expressions. Check, 
if there are other processes modifying assp files. Check that required 
services (DNS, SQL,ClamAV,. ) are responsive at this time. 

If (for any reason) it is expected, that the config reload takes 30 
seconds or longer - 'ConnectionTransferTimeOut' should be changed - or the 
"Warning: Main_Thread is unable to transfer connection to any worker - try 
again!" should be ignored. 

If the reload takes X seconds for the MainThre

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[Thomas Eckardt]

>I tried to see where SaveConfig()

SaveConfig() is called in the Main_Thread (Worker_0) every time, anything 
in the assp configuration was changed.
And under normal cirumstances, after that, all workers are told to check 
there configuration (and reload) in relation to the new one (lists, files, 
recompile regexes  ).

> I >>think<< 

if I would have a problem with >"Saving config"< (which is a maintenance 
task) - I would increase the 'MaintenanceLog' level
if I would have a problem with workers >"unable to transfer connection to 
any worker"< - I would enable WorkerLog and WorkerLogging

example log:

Oct-25-21 23:12:58 [Worker_1] Downloading level-2-TLDlist via direct 
HTTP connection
Oct-25-21 23:12:58 [Worker_1] Level-2-TLDlist download completed
Oct-25-21 23:12:58 [Worker_1] Downloading level-3-TLDlist via direct 
HTTP connection
Oct-25-21 23:12:59 [Worker_1] Level-3-TLDlist download completed
Oct-25-21 23:12:59 [Worker_1] Info: next TLDlist download in 1 day 5 
hours 58 minutes 
Oct-25-21 23:12:59 [Worker_1] Info: file c:/assp/files/URIBLCCTLDS.txt 
updated for URIBLCCTLDS
Oct-25-21 23:13:00 [Main_Thread] Saving config
Oct-25-21 23:13:00 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file c:/assp/assp.cfg is unchanged
Oct-25-21 23:13:00 [Main_Thread] Adminupdate: file 
'c:/assp/files/URIBLCCTLDS.txt' for config 'URIBLCCTLDS' was changed
Oct-25-21 23:13:00 [Main_Thread] Option list file: 
'c:/assp/files/URIBLCCTLDS.txt' reloaded (URIBLCCTLDS) with 12,280 records
Oct-25-21 23:13:02 [Worker_1] Worker_1 wakes up
Oct-25-21 23:13:02 [Worker_5] Worker_5 wakes up
Oct-25-21 23:13:02 [Worker_4] Worker_4 wakes up
Oct-25-21 23:13:02 [Worker_3] Worker_3 wakes up
Oct-25-21 23:13:02 [Worker_2] Worker_2 wakes up
Oct-25-21 23:13:03 [Worker_1] Worker_1 finished reloading configuration
Oct-25-21 23:13:03 [Worker_1] Worker_1 will sleep now
Oct-25-21 23:13:04 [Worker_2] Worker_2 finished reloading configuration
Oct-25-21 23:13:04 [Worker_2] Worker_2 will sleep now
Oct-25-21 23:13:05 [Worker_3] Worker_3 finished reloading configuration
Oct-25-21 23:13:05 [Worker_3] Worker_3 will sleep now
Oct-25-21 23:13:06 [Worker_4] Worker_4 finished reloading configuration
Oct-25-21 23:13:06 [Worker_4] Worker_4 will sleep now
Oct-25-21 23:13:07 [Worker_5] Worker_5 finished reloading configuration
Oct-25-21 23:13:07 [Worker_5] Worker_5 will sleep now
Oct-25-21 23:13:08 [Worker_1] Worker_1 finished reloading 
configuration
Oct-25-21 23:13:10 [Worker_10001] Worker_10001 finished reloading 
configuration
Oct-25-21 23:13:29 [Worker_1] Downloading Extended Droplist via direct 
HTTP connection
Oct-25-21 23:13:29 [Worker_1] Extended Droplist already up to date
Oct-25-21 23:13:29 [Worker_1] Info: next droplist download in 7 hours 
34 minutes 


debug may help
for time related debugging, I (or some one who read the manual) would 
consider to use ConfigChangeSchedule

notice: analyzing all the produced debug files (in general debug mode) is 
a very time consuming task

Check your option files for bad (too greedy) regular expressions. Check, 
if there are other processes modifying assp files. Check that required 
services (DNS, SQL,ClamAV,. ) are responsive at this time.

If (for any reason) it is expected, that the config reload takes 30 
seconds or longer - 'ConnectionTransferTimeOut' should be changed - or the 
"Warning: Main_Thread is unable to transfer connection to any worker - try 
again!" should be ignored.

If the reload takes X seconds for the MainThread. Within these X seconds 
all new connections are queued by the OS. After this time (the reload) the 
MainThread tries to transfer all these new connections within some 
(milli)seconds to the workers - this may overload the SMTP-workers for 
some time.

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  25.10.2021 17:20
Betreff:Re: [Assp-test] Main_Thread is unable to transfer 
connection to any worker - try again



We are lucky to have a new (to us) faster server donated since this 
original May posting on the "unable to transfer connection to any worker" 
error.  However, with the new box with Windows 2019 installation, fully 
patched, MySQL latest community, and Strawberry perl 5.32 it's still 
happening in spurts.Not a heavy load, 16gb ram.  12 cores total.  ASSP 
uses about 1.7gb after running for a bit.  MySQL seems fast, ClamAV in 
use.

Sometimes I get the warning just once in a day, often it's every 5 minutes 
for a while.  This morning, there were 3 occurrences with 10 minutes in 
between each.  But always, when I look at the log, it's right after the 
"Saving config" process, always when there's no config changes. 

I tried to see where SaveConfig() is being called from every 5 minutes.  I 
>>think<< it's when the ReloadOptionFiles timer hits, but I'm not sure.
ReloadOptionFiles is set to 300 (5 minutes)
AutoReloadCfg is enabled, though I'm never 

Re: [Assp-test] Concept question: At rebuild, look at DKIMWLAddresses?

[Thomas Eckardt]

>without rebuild taking too much of a performance hit?

fastest case: all in the file model

slowest case: 
- no file model
- checking whiteRe, whitelist, npRe, DKIMWLAddresses, DKIMNPAddresses, 
redRe - for assp/spam
- checking denySMTPConnectionsFromAlways, denySMTPConnectionsFrom, 
blackListedDomains for assp/notspam

The slowest  case is 12 times slower than the fastest.

Thomas








Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  24.10.2021 02:40
Betreff:[Assp-test] Concept question: At rebuild, look at 
DKIMWLAddresses?




Would it be possible to have ASSP consider DKIMWLAddress matches during 
rebuild, removing matching messages from spam, without rebuild taking too 
much of a performance hit?  

During rebuild, ASSP runs rb_whitelisted against each message in the spam 
corpus, and if a match is found against the whitelist, that message is 
removed from spam.  (right?) It's a terrific way to help keep the corpus 
clean after a whitelist addition.

I rely heavily on DKIMWLAddresses - it's super helpful to consider a 
message whitelisted only when the DKIM signature matches.

If it's realistically possible and not ill conceived, removal of messages 
from spam where there's a DKIMWLAddress match would further clean up spam, 
and lead to more accurate HMM/Bayesian detections. 

What do you think?

Along the same lines, what about considering messages that match no 
processing rules: the regexes and DKIMNPAddresses for messages in both 
spam and notspam?
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept question: At rebuild, look at DKIMWLAddresses?

[Thomas Eckardt]

>During rebuild, ASSP runs rb_whitelisted against each message in the spam 
corpus

only for spam (not for corrected spam) AND only if enabled (DoRBWhite , 
DoRBRed) !

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  24.10.2021 02:40
Betreff:[Assp-test] Concept question: At rebuild, look at 
DKIMWLAddresses?




Would it be possible to have ASSP consider DKIMWLAddress matches during 
rebuild, removing matching messages from spam, without rebuild taking too 
much of a performance hit?  

During rebuild, ASSP runs rb_whitelisted against each message in the spam 
corpus, and if a match is found against the whitelist, that message is 
removed from spam.  (right?) It's a terrific way to help keep the corpus 
clean after a whitelist addition.

I rely heavily on DKIMWLAddresses - it's super helpful to consider a 
message whitelisted only when the DKIM signature matches.

If it's realistically possible and not ill conceived, removal of messages 
from spam where there's a DKIMWLAddress match would further clean up spam, 
and lead to more accurate HMM/Bayesian detections. 

What do you think?

Along the same lines, what about considering messages that match no 
processing rules: the regexes and DKIMNPAddresses for messages in both 
spam and notspam?
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept question with sample code: DKIMWLAddresses single line for .domain.com and @domain.com?

[Thomas Eckardt]

(@|.)domain.com

Thomas




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  24.10.2021 04:36
Betreff:[Assp-test] Concept question with sample code: 
DKIMWLAddresses single line for .domain.com and @domain.com?



(I believe I have this working, see code please)

I estimate that at least 90% of the time that I want to add an entry to 
DKIMWLAddresses that I put two lines, one for @domain.com and the other to 
match the subdomains, so .domain.com.

That's fine,and works well, but I'd like to see ASSP allow admins to use a 
single line shorthand for signatures ending in both .domain.com and @
domain.com.  So instead of
@domain.com
.domain.com
just do the shorthand of
>>domain.com

The last time I asked for a new feature, you suggested that I try coding 
it myself, so I did.  There's no pride in authorship here, I'm a LOUSY 
coder.  I'd love to see how you tackle this, provided that my concept's 
sound.

I picked the >> characters, since > is illegal in email addresses / domain 
names.  Originally, I selected the + character, but that can be in the 
user part of the email address.  >> is highly visible when scanning 
through config files, a single > wasn't as easy for me to spot.

So,I tested out modifying the setDKIMNPAddressesRE and 
setDKIMWLAddressesRE functions, from:
my $new=shift;
$new||=$neverMatch; # regexp that never matches
SetRE('DKIMWLAddressesRE',"(?:$new)\$",
  $regexMod,
  'DKIM whitelisted',$_[0]);

to:
my $new=shift;
$new||=$neverMatch; # regexp that never matches
$new=~s/>>(.*)(\||$)/(\\\.|\\\@\)$1$2/go;
SetRE('DKIMWLAddressesRE',"(?:$new)\$",
  $regexMod,
  'DKIM whitelisted',$_[0]);

If my hack coding is correct, this will take the existing string, which is 
already a regex compiled by ASSP based on the non-regex DKIMWL/NPAddresses 
entered by the user, look for

>>whatever| or >>whatever at the end of the string and change that to
(\@|\.)whever| or no | if it's the end of the string

In my rudimentary testing, that seems to work.   

What do you think?  

That would cut down my DKIMWLAddresses down by close to 50%, and make 
management much easier.



___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21293

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21293:

- if a file for regular expressions contained an incomplete default 
definition for the !!!NWLI!!! directive, this directive was not applied to 
the regexes in the file



changed:

- some corrections and additions to the main help text in the GUI

- the behavior of the 'NWLI' extension in regular expression definitions 
is enhanced

The NWLI conditions defined in a line are combined using a logical AND -- 
so N-W+ is combined to: NOT noprocessing AND whitelisted.
In fact, the weight is skipped, if any of the defined NWLI options does 
not match for a mail. If multiple lines would match, the weight of the 
first matching line is used.
This way you can define different weights for the same regular expression, 
but different mail states like in this example:
(1) foo=>0:>NW - weight is zero if noprocessing AND whitelisted
(2) foo=>0.5:>NW- - weight factor is 0.5 if noprocessing AND NOT 
whitelisted
(3) foo=>1.5:>N-W - weight factor is 1.5 if NOT noprocessing AND 
whitelisted
(4) foo=>55:>N-W- - weight is 55 if NOT noprocessing AND NOT whitelisted
(5) foo=>2:>W - this line will not be processed, because line 1 or 3 would 
have matched before, depending on the noprocessing flag
(6) foo=>2:>N- - this line will not be processed, because line 3 or 4 
would have matched before, depending on the whitelisted flag


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[Thomas Eckardt]

>Note: Messages flagged as "no processing" will not contribute to the 
corpus
there is no option to store noprocessing in the corpus folders

>will not be scored based on Bayesian/HMM detection.  
not true - config option


>However, by default, "Penalty Box" / Bomb scroing will still take place.
NOT by default - EVERYTIME - nothing else is written in the manual

See !!!NWLI!!! option to override this default behavior.

nonsense


THERE IS NO OTHER WAY to get knowledge about ASSP than reading the manual 
completely! I recommend to do this more than twice!
It is useless to stumble around the configuration options and try to 
understand something.

I'm angry  and I stop here - otherwise I would 
lose my way and my composure



Thomas

NWLI will get an improvement and a small fix in the next version.




Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  18.10.2021 17:26
Betreff:Re: [Assp-test] Does Message Score build from Bomb matches 
override a DKIMNP match??



Well that'll do it!  I incorrectly thought all these years that "no 
processing" actually meant NO processing.Looking back, the NWLI 
options have only been a choice for 11 years   I'm surprised I never 
caught this happening (correctly) before.

I agree that it the gui doesn't say anywhere that noProcessing isn't 
processed by the penaltybox, but it's literally called NoProcessing, not 
less processing, etc, hence my confusion.

GUI says:
Mail solely to or from any of these addresses are proxied without 
processing. The envelope sender and recipients are checked. Like a more 
efficient version of Spam-Lovers & redlist combined. Accepts specific 
addresses (u...@domain.com), user parts (user) or entire domains (@
domain.com). Wildcards are supported (fribo*@domain.com). If you register 
TO addresses here, all recipients for a single mail must be marked as 
noprocessing to flag the mail as "noprocessing".

My money says that 90%+ of the admins using ASSP have the same 
misperception as I do on this.  I suggest we add:
Note: Messages flagged as "no processing" will not contribute to the 
corpus and will not be scored based on Bayesian/HMM detection.  However, 
by default, "Penalty Box" / Bomb scroing will still take place.  See 
!!!NWLI!!! option to override this default behavior.







On Sun, Oct 17, 2021 at 4:25 AM Thomas Eckardt  wrote:
>(not sure why this line is in the log twice) 

because the string was found twice 


>msg rejected, even though no processing

I can't find anything in the manual, which states that 'noprocessing' 
mails are not processed by the penaltybox 

Scores are added by the bomb feature, because assp is configured to score 
noprocessing mails. 
bombReNP 
or 
=>NWLI is used (N at least for this regex) 

Thomas 





Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:17.10.2021 02:02 
Betreff:[Assp-test] Does Message Score build from Bomb matches 
override a DKIMNP match?? 



I had an inbound message rejected by ASSP, where the DKIM signature 
matched DKIMNP. I would have thought that if there's a DKIMNP match, that 
the message will just be passed and saved in discarded.   

Also, Senderbase is white for the network that it came from. so that 
should have reduced the score by a lot.   

There was a bombDataRE match. seemingly twice for the same line. and also 
in BombData.  I've got Dear Friend, in both files by mistake, that'll be 
fixed, but that pushed the score above 50, so it was rejected.  Shouldn't 
DKIMNP override the rejection though? 

Here's the log, with my notes: 

msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org DKIM-Signature found
Info: enhanced Originated IP detection ignored IP's: 102.xxx.yyy.85 
(connected IP) , 10.11.74.34
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org info: found DKIM signature identity '@
bounce.TheirDomain.com'
@bounce.TheirDomain.com @bounce.TheirDomain.com,u...@ourchairty.org 
matches .TheirDomain.com in DKIMNPAddresses
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org [scoring] DKIM signature verified-OK - header-passed - 
identity is: @bounce.TheirDomain.com - sender policy is: neutral - author 
policy s: neutral - state changed to: noprocessing
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with 
'dear friend,' - weight is 0.5   <-- we get a lot of Dear Friend, garbage, 
so I have it in BombData with a 50% score
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with 
'dear friend,' - weight is 0.5(not sure why this line is in the log 
twice)
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org spambomb Regex: bombDataRe 'PB 18: for Dear Friend,'
msg11890-19574 [BombData] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourc

[Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21290

[Thomas Eckardt]

Hi all,

fixed in assp 2.6.6 *SPAM-Evaporator* build 21290:


- build 21287 caused an error 'BerkeleyDB-ERROR: in start 
rebuildAddCorrections - syntax error at (eval 679) line 5, near 
"$main::$BDBerrLog "'

- fixes a STATS counting mistake since 21280

- if an IP was blocked by an early blocking feature (like maxAUTHError) at 
least 'maxSMTPipSessions' times, this IP was blocked by 
'maxSMTPipSessions' until assp was restarted


Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21287

[Thomas Eckardt]

this will be fixed

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  16.10.2021 22:03
Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* 
build 21287



Very shortly after startup, I received:

 BerkeleyDB-ERROR: in start rebuildAddCorrections - syntax error at (eval 
679) line 5, near "$main::$BDBerrLog "
Global symbol "$BDBerrLog" requires explicit package name (did you forget 
to declare "my $BDBerrLog"?) at (eval 679) line 5.
 - BDB:

On Fri, Oct 15, 2021 at 12:51 PM K Post  wrote:
Again, thanks.
So are you recommending that $BDBerrLog stay at the default of 0 under 
normal circumstances, and only be changed to 1 if there appears to be 
something awry with one or more BDB actions?  

I've never seen anything written to any of the BDBError.txt files, but I 
certainly could have missed errors there - it's just not something I've 
monitored.  I made the mistake of assuming, yes, assuming that errors 
would also go to maillog.txt - but as you pointed out, it's the BDB driver 
that would erroring, not ASSP.   Is there a way / does it make sense / 
have you considered having ASSP use some magic to trap any BDB error or 
warning that may occur?  

On Fri, Oct 15, 2021 at 5:35 AM Thomas Eckardt  wrote:
>If msvcrt is being used, would 512 open files ever not be enough?

No, not in every case. This depends on the configuration and the workload. 


> $winSetMaxIO_DLL and $winSetMaxIO I can understand how to set them 
for my set up. 

don't change them 

>Do you know if the Strawberry Perl installations-DUSE_PERLIO? 

yes it is  ...   

> I tried looking it up, but I'm coming up empty 
.. # Notice: PERLIO (perl compiled with -DUSE_PERLIO - check with 
:>perl -V) 

 :>perl -V 
or 
read perl/lib/Config_heavy.pl 


>With BDBErrLog set to 0, I assume that any error with BDB files would 
still be spit out to the maillog.txt file so we can be alerted that 
something's wrong?  

assume ?? . Read the perl POD for BekeleyDB.pm and the oracle 
documentation for BerkeleyDB. 
Who would need BDB-ENV -errfile if such errors could be catched elsewhere 
easely? 

After (e.g.) a HASH %bar is tied to : memory, file, orderedtie, 
BerkeleyDB, ODBC, ADO or any native RDBM 
(oracle,db2,mysql,mariadb,mssql,postgre .) - a simple call like 

$bar{$foo} 

accesses totaly different code (the driver). It is impossible to catch all 
possible errors for all cases for all tied mechanism, after such a call, 
to write them to maillog.txt. ASSP tries to do its best to catch as much 
of the errors as possible and to recover from error conditions 
automatically. 
But errors may occure at software layers, which can't be accessed by assp. 

ASSP catches all errors at init-time of BerkeleyDB (and recovers if 
possible). If there occure errors at runtime for BerkeleyDB, someone can 
enable 'BDBErrLog' to get the runtime errors recorded. 

Thomas 



Von:"K Post"  
An:"ASSP development mailing list" <
assp-test@lists.sourceforge.net> 
Datum:14.10.2021 20:25 
Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* 
build 21287 



Whew you've been busy! Thank you. 
If msvcrt is being used, would 512 open files ever not be enough?  I feel 
like I was getting the file issues when many links to BDB-error.txt files 
were getting stuck open, so exceeding 512, but that was ultimately because 
of my stupid griplist directory misconfiguration combined with bad TLS 
early talkers.  I guess I'm trying to understand why $winSetMaxIO_DLL and 
$winSetMaxIO hidden params were necessary, so I can understand how to set 
them for my set up.
Do you know if the Strawberry Perl installations at 
https://strawberryperl.com/releases.html compiled with DUSE_PERLIO?I 
tried looking it up, but I'm coming up empty
With BDBErrLog set to 0, I assume that any error with BDB files would 
still be spit out to the maillog.txt file so we can be alerted that 
something's wrong?  



On Thu, Oct 14, 2021 at 9:52 AM Thomas Eckardt  wrote: 
Hi all, 

fixed in assp 2.6.6 *SPAM-Evaporator* build 21287: 

- If a folder was defined for the parameter 'griplist' (e.g. 
grip/griplist) and this folder was not extisting, all griplist functions 
were not working. 
  If a folder is now defined, it is created by assp. 

- If 'ConfigChangeSchedule' was used to change a hidden configuration 
parameter, only the main thread (not any worker) was aware of the change. 

- If a mail subject contained a questionmark '?' in its text and the 
subject header line was encoded 'Quoted Printable' and the questionmark 
was not right MIME encoded 
  (instead it was written as '?') all internal functions related to the 
mail subject were not working correctly 


changed: 

- BerkeleyDB error logs (BDB-error.txt) are no longer permanently created 
and locked 
  Instead there is a new hidden parameter 'BDBerrLog', wh

Re: [Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[Thomas Eckardt]

>(not sure why this line is in the log twice)

because the string was found twice


>msg rejected, even though no processing

I can't find anything in the manual, which states that 'noprocessing' 
mails are not processed by the penaltybox

Scores are added by the bomb feature, because assp is configured to score 
noprocessing mails.
bombReNP
or
=>NWLI is used (N at least for this regex)

Thomas





Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  17.10.2021 02:02
Betreff:[Assp-test] Does Message Score build from Bomb matches 
override a DKIMNP match??



I had an inbound message rejected by ASSP, where the DKIM signature 
matched DKIMNP. I would have thought that if there's a DKIMNP match, that 
the message will just be passed and saved in discarded.  

Also, Senderbase is white for the network that it came from. so that 
should have reduced the score by a lot.  

There was a bombDataRE match. seemingly twice for the same line. and also 
in BombData.  I've got Dear Friend, in both files by mistake, that'll be 
fixed, but that pushed the score above 50, so it was rejected.  Shouldn't 
DKIMNP override the rejection though?

Here's the log, with my notes:

msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org DKIM-Signature found
Info: enhanced Originated IP detection ignored IP's: 102.xxx.yyy.85 
(connected IP) , 10.11.74.34
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org info: found DKIM signature identity '@
bounce.TheirDomain.com'
@bounce.TheirDomain.com @bounce.TheirDomain.com,u...@ourchairty.org 
matches .TheirDomain.com in DKIMNPAddresses
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org [scoring] DKIM signature verified-OK - header-passed - 
identity is: @bounce.TheirDomain.com - sender policy is: neutral - author 
policy s: neutral - state changed to: noprocessing
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with 
'dear friend,' - weight is 0.5   <-- we get a lot of Dear Friend, garbage, 
so I have it in BombData with a 50% score
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with 
'dear friend,' - weight is 0.5(not sure why this line is in the log 
twice)
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org spambomb Regex: bombDataRe 'PB 18: for Dear Friend,'
msg11890-19574 [BombData] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [scoring] 
(BombData 'Dear Friend,')
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org Message-Score: added 18 for Regex: bombDataRe 'PB 18: 
for Dear Friend,' BombData: 'Dear Friend,', total score for this message 
is now 18
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org spambomb Regex: bombRe 'PB 35: for Dear Friend'
msg11890-19574 [BombData][bombRe] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [scoring] 
(bombRe 'Dear Friend')
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org Message-Score: added 35 for Regex: bombRe 'PB 35: for 
Dear Friend' bombRe: 'Dear Friend', total score for this message is now 53
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org deleting spamming safelisted tuplet: (102.xxx.yyy.0,
bounce.TheirDomain.com) age: 1s
msg11890-19574 [MessageLimit] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [spam found] 
(MessageScore 53, limit 50) [Our  Newsletter October 15th 2021] -> 
messages/discarded/Our__Newsletter_October_15th_2021--254778.txt;
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org [SMTP Error] 554 5.7.1 [PE] rejected msg [PR] 
[msg11890-19574 212EA668]  <-- msg rejected, even though no processing
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org info: PB-IP-Score for '102.xxx.yyy.0' is 53, added 53 
in this session
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org finished message - received DATA size: 138.82 kByte - 
sent DATA size: 0 Byte
msg11890-19574 102.xxx.yyy.85  to: 
u...@ourchairty.org disconnected: session:212EA668 102.xxx.yyy.85 - 
processing time 2 seconds

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21287

[Thomas Eckardt]

>If msvcrt is being used, would 512 open files ever not be enough?

No, not in every case. This depends on the configuration and the workload.

> $winSetMaxIO_DLL and $winSetMaxIO I can understand how to set them 
for my set up.

don't change them

>Do you know if the Strawberry Perl installations-DUSE_PERLIO?

yes it is  ... 

> I tried looking it up, but I'm coming up empty
.. # Notice: PERLIO (perl compiled with -DUSE_PERLIO - check with 
:>perl -V)

 :>perl -V
or
read perl/lib/Config_heavy.pl


>With BDBErrLog set to 0, I assume that any error with BDB files would 
still be spit out to the maillog.txt file so we can be alerted that 
something's wrong? 

assume ?? . Read the perl POD for BekeleyDB.pm and the oracle 
documentation for BerkeleyDB.
Who would need BDB-ENV -errfile if such errors could be catched elsewhere 
easely?

After (e.g.) a HASH %bar is tied to : memory, file, orderedtie, 
BerkeleyDB, ODBC, ADO or any native RDBM 
(oracle,db2,mysql,mariadb,mssql,postgre .) - a simple call like

$bar{$foo} 

accesses totaly different code (the driver). It is impossible to catch all 
possible errors for all cases for all tied mechanism, after such a call, 
to write them to maillog.txt. ASSP tries to do its best to catch as much 
of the errors as possible and to recover from error conditions 
automatically.
But errors may occure at software layers, which can't be accessed by assp.
ASSP catches all errors at init-time of BerkeleyDB (and recovers if 
possible). If there occure errors at runtime for BerkeleyDB, someone can 
enable 'BDBErrLog' to get the runtime errors recorded.

Thomas



Von:"K Post" 
An: "ASSP development mailing list" 
Datum:  14.10.2021 20:25
Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* 
build 21287



Whew you've been busy! Thank you.

If msvcrt is being used, would 512 open files ever not be enough?  I feel 
like I was getting the file issues when many links to BDB-error.txt files 
were getting stuck open, so exceeding 512, but that was ultimately because 
of my stupid griplist directory misconfiguration combined with bad TLS 
early talkers.  I guess I'm trying to understand why $winSetMaxIO_DLL and 
$winSetMaxIO hidden params were necessary, so I can understand how to set 
them for my set up.

Do you know if the Strawberry Perl installations at 
https://strawberryperl.com/releases.html compiled with DUSE_PERLIO?I 
tried looking it up, but I'm coming up empty

With BDBErrLog set to 0, I assume that any error with BDB files would 
still be spit out to the maillog.txt file so we can be alerted that 
something's wrong? 



On Thu, Oct 14, 2021 at 9:52 AM Thomas Eckardt  wrote:
Hi all, 

fixed in assp 2.6.6 *SPAM-Evaporator* build 21287: 

- If a folder was defined for the parameter 'griplist' (e.g. 
grip/griplist) and this folder was not extisting, all griplist functions 
were not working. 
  If a folder is now defined, it is created by assp. 

- If 'ConfigChangeSchedule' was used to change a hidden configuration 
parameter, only the main thread (not any worker) was aware of the change. 

- If a mail subject contained a questionmark '?' in its text and the 
subject header line was encoded 'Quoted Printable' and the questionmark 
was not right MIME encoded 
  (instead it was written as '?') all internal functions related to the 
mail subject were not working correctly 


changed: 

- BerkeleyDB error logs (BDB-error.txt) are no longer permanently created 
and locked 
  Instead there is a new hidden parameter 'BDBerrLog', which can be set to 
1 to monitor BDB-problems. 

our $BDBerrLog = 0; # (0/1) log BerkeleyDB errors in the related BDB-ENV 
-errfile .../BDB-error.txt (default = 0) 

- The GUI-help text for 'noGriplistUpload', 'noGriplistDownload' and 
'gripValencePB' are updated - griplist functions are not changed 



added: 
- If windows systems are running out of available open file descriptors 
and the used perl installation is not compiled using the -DUSE_PERLIO 
switch, 
  the following parameters can be used to increase the available file 
descriptors for the assp process 

our $winSetMaxIO_DLL = 'msvcrt'; # the name of the microsoft 
C-runtime-library used by perl and/or perl-modules (Win32 only !!!) - 
default is msvcrt 
 # If your perl uses (is compiled 
against) any other msvcrtXXX (for example: msvcrt160 or msvcrt100) - 
change this value, if 
 # you want to set the maximum 
open files limit in the msvcrtXXX. 
 # This value is ONLY used for the 
below purpose ($winSetMaxIO), it has no other effect ! 

our $winSetMaxIO = 0;# (0/1/ 512 * 2**N) set the 
maximum open files limit (Win32 only !!!) in ($winSetMaxIO_DLL) msvcrt.dll 
(_getmaxstdio , _setmaxstdio) 
 # 0 - use the default s