[Assp-test] MalDoc in PDF

[K Post]

Info released today on a new technique called MalDoc in PDF
https://blogs.jpcert.or.jp/en/2023/08/maldocinpdf.html

Will ASSP_AFC be able to block these if we already reject VBS / OLE / VBA
in Office docs?  Will it know it's an office doc, or will it think PDF?
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unable to create UDP Socket errors

[K Post]

I never saw this prior to 22326, but have seen it twice on 22326 running on
Windows.  ASSP will run for days/weeks without issue, but once the UDP
socket errors start, I need to restart or else they'll continue every 5
minutes or so.  I get 2 of the same error messages in a row, every 5 min or
so, until ASSP is restarted.

Dec-28-22 10:49:35 Error: DNS - unable to create any UDP socket to
nameservers (2 of my servers here)
Dec-28-22 10:49:35 Error: DNS - unable to create any UDP socket to
nameservers ( 2 of my servers here  )

On Wed, Nov 23, 2022 at 3:18 AM Graziano via Assp-test <
assp-test@lists.sourceforge.net> wrote:

> Hi
>
> I have this issue too , using
>
> Centos 7.9 , Perl 5.36.0 , ASSP b22293
>
> however apparently all is working ok in ASSP (all DNS checks are working)
>
> Hi all,
>
> I'm running ASSP on CENTOS 7, with Perl 5.30.1. I recently upgraded from
> b22137 to b22318.
>
> Shortly afterwards, I'm seeing a new error from ASSP every few minutes:
>
> [Worker_1] Error: DNS - unable to create any UDP socket to nameservers
> (1.1.1.10 111.111.111.22)
>
> * Not my actual DNS servers (I tested with Google public DNS and get the
> same result)
>
> As far as I can tell, mail is still working ok, and all ASSP features seem
> fine, even my DNSBL and URIBL are apparently still working.
>
> This has been rather difficult to troubleshoot, but `ss -tulpn` doesn't
> show anything excessive or unexpected as far as UDP usage.
>
> I did note that the CENTOS bind packages did see an update installed
> around this time as well. There was also a new kernel package installed.
> Perhaps there was a change in the behavior of that software that ASSP is
> having trouble dealing with?
>
> I'm looking for ideas for continuing to troubleshoot this, and also
> wondering if anyone else is seeing anything similar.
>
> -C
>
>
>
> ___
> Assp-test mailing 
> listAssp-test@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Comments in ASSP_AFCKnownGoodEXE file

[K Post]

Minor issue/bug with ASSP_AFCKnownGoodEXE config file comments.

I've got 2 sha256 entries in my  ASSP_AFCKnownGoodEXE.  The file looks like

# PDF from accounting Dec 2022
7B366390CD4E3DB19E06009FCE54DF272BEE0D3B34003F3A786C4835239BFDFF


# PDF from IRS Dec 2022
1EBE9B7DC3AEF7F492FCD22D4430123FEEA663A3227AB9F70D59ACDCBFB06C36


When I save, I get this in the log:

Dec-17-22 22:57:30 ASSP_AFCKnownGoodEXE: invalid attachment SHA256_HEX
definition - hash: is ignored
Dec-17-22 22:57:30 ASSP_AFCKnownGoodEXE: invalid attachment SHA256_HEX
definition - hash: is ignored
Dec-17-22 22:57:30 ASSP_AFCKnownGoodEXE: 2 well known good SHA256_HEX
attachment hashes registered


It doesn't seem like it's happy with the comment lines.  If I put the
comment on the same line, after the sha256, it ASSP doesn't complain

7B366390CD4E3DB19E06009FCE54DF272BEE0D3B34003F3A786C4835239BFDFF  # PDF
from accounting

Also, I found a cosmetic mistake in the ASSP file editor GUI. The note at
the bottom says:

File should have one entry per line; anything on a line following a number
sign ( #) is ignored (a comment). Whitespace at the beginning or end of the
line is ignored.

There's a space before the #.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Allowing certain javascript in HTML to pass

[K Post]

quick question - before I dig deeper, did the previous AFC plugin not block
javascript in HTML at all?

On Mon, Oct 31, 2022 at 10:21 AM K Post  wrote:

> The new AFC is blocking a nightly report that comes in HTML format with
> javascript in it -- as I would expect, but before his new AFC, they were
> erroneously slipping through.
>
> I don't know why these reports weren't being blocked before, it's basic
> HTML with a short block of javascript at the end.  Of note, the javascript
> starts like this and has a base64 image in its code - something that the
> new AFC addresses:
>
> 

[Assp-test] Allowing certain javascript in HTML to pass

[K Post]

The new AFC is blocking a nightly report that comes in HTML format with
javascript in it -- as I would expect, but before his new AFC, they were
erroneously slipping through.

I don't know why these reports weren't being blocked before, it's basic
HTML with a short block of javascript at the end.  Of note, the javascript
starts like this and has a base64 image in its code - something that the
new AFC addresses:

Re: [Assp-test] Line Continuation in config files

[K Post]

appreciated.

On Fri, Sep 9, 2022 at 3:46 AM Thomas Eckardt 
wrote:

> The line continuation is supported in every file.
>
> How ever, I've done a positioning mistake in the used regex
>
> current: s/\\(? right: s/(?
> And I've found another mistake. The  line continuation was not working for
> the analyzer - the matching line in a file was not found (and not shown)
>
> This will be fixed.
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:08.09.2022 21:14
> Betreff:[Assp-test] Line Continuation in config files
> --
>
>
>
> Back in November 2011, I saw:
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21317:
> - files used in configuration parameters are now
> supporting line continuation by adding a backslash '\' at the end of a line
>
> I'm just trying this now, but it doesn't seem to be working.
>
> When you say "files used in configuration parameters" do you only mean
> files that use the # include syntax, or should this work for things like
> bomgHeaderRe where we have file:/files/bombHeaderRe.txt in the GUI?
>
> If line continuation is only supported in # include files, could you
> extend the functionality to work in files that are directly referenced by
> the GUI (file:files/ext.txt)?
>
> Here's what I'm experiencing:
>
> In my bombHeaderRe file, which is directly referenced in the gui as
> file:files/bomgHeaderRe.txt, I have
>
>
> ~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))(?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10
>
> all on a single line.  That scores -10 to any message where to:
> *senderdom...@ourdomain.com*  and from:
> anything@*.SenderDomain.com appear in the header, in any order.  (thanks
> for all the help building this, it's been incredibly beneficial)
>
> I just tried splitting that regex into multiple lines by adding a \ at the
> end of lines.   I'm not putting a space
>
>
> ~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))\
>
>
> (?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n\
>
> (?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10
>
> Saving the file does not trigger an error in the GUI, but the analyze GUI *no
> longer shows matches* for the same email that does when the regex is on a
> single line.
>
>
> Thanks![Anhang "att948xp.txt" gelöscht von Thomas Eckardt/eck] [Anhang
> "attmyh5k.txt" gelöscht von Thomas Eckardt/eck]
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Line Continuation in config files

[K Post]

Back in November 2011, I saw:

fixed in assp 2.6.6 *SPAM-Evaporator* build 21317:
- files used in configuration parameters are now supporting line
continuation by adding a backslash '\' at the end of a line

I'm just trying this now, but it doesn't seem to be working.

*When you say "files used in configuration parameters" do you only mean
files that use the # include syntax, or should this work for things like
bomgHeaderRe where we have file:/files/bombHeaderRe.txt in the GUI?*

If line continuation is only supported in # include files, could you extend
the functionality to work in files that are directly referenced by the GUI
(file:files/ext.txt)?

Here's what I'm experiencing:

In my bombHeaderRe file, which is directly referenced in the gui as
file:files/bomgHeaderRe.txt, I have

~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))(?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10


all on a single line.  That scores -10 to any message where to:
senderdom...@ourdomain.com and from: anything@*.SenderDomain.com appear in
the header, in any order.  (thanks for all the help building this, it's
been incredibly beneficial)

I just tried splitting that regex into multiple lines by adding a \ at the
end of lines.   I'm not putting a space

~(?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))\
(?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?)).+?\n\
(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))~=>-10

Saving the file does not trigger an error in the GUI, but the analyze GUI *no
longer shows matches* for the same email that does when the regex is on a
single line.


Thanks!
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] soft hyphen fooling Bayesian analysis

[K Post]

Thanks again for the explanation.  Looking forward to a future release when
soft-hyphens (and additional control characters?) are essentially ignored.

On Wed, Sep 7, 2022 at 9:14 AM Thomas Eckardt 
wrote:

> If unicode normalization NFKC does'nt fulfill your requirement, you may
> enable 'DoTransliterate' - by accepting some performance penalties.
>
> The "Unicode Technical Standard #39" http://www.unicode.org/reports/tr39/
> will give you some more information and
> https://www.unicode.org/Public/security/revision-05/intentional.txt shows
> a nice table for cyrillic and greek.
> If someone expects an ASCII mail, those translations may somehow help. But
> in all other cases (100% cyrillic/greek/), such a character replacement
> is contra-productive (for example: not all cyrillic letters have a valid
> latin replacement).
>
> > potentially treat look-alike characters as the latin character for
> bayesian purposes
>
> The HMM and Bayesian engines are using heuristic mechanism. Trying to
> treat single characters as latin (or anything else) will not worth the
> effort. Over a short periode of time, both engines will have learned also
> obscured words (word combinations).
>
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:06.09.2022 21:31
> Betreff:Re: [Assp-test] soft hyphen fooling Bayesian analysis
> --
>
>
>
> Eager to see what you come up with in terms of ignoring the soft hyphen.
>
>  Your <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>> regex is clear, and I
> understand using that for scoring purposes, but I'm looking for a way to
> potentially treat look-alike characters as the latin character for bayesian
> purposes and/or to catch commonly obscured words (like GeekSquad).  Is it
> okay if I reply further in my  August 1 post here to keep that in the same
> thread?
>
> On Tue, Sep 6, 2022 at 2:06 PM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> >HTML::strip
>
> html parsing to get text parts has nothing to do with html de(en)coding
>
>
> >iso-8559-1
> ASSP processes all content as UTF-8
>
>
> >
> ASSP is aware about this - and replaces soft-hyphens with hard-hyphens -
> and multiple concurrent hard-hyphens with a single one
> How ever - the option to remove the soft-hyphens instead, sounds somehow
> better. Tests are still running.
>
> >My thinking is that if it doesn't display.
> ASSP does'nt know if something displayed or not (and will never know it)
>
>
> >I suspect that other characters will be abused in the same way
>  as well as several BIG5, numerical and other unicode characters are
> already special handled by assp. Other CTL-chars are ignored by assp.
> Everything is converted to UTF8, unicode normalized (including grapheme
> clusters), stemmed and simplyfied.
>
>
> >This kind of obfuscation goes hand in hand with my previous questions
> about considering some non-Latin characters that look like Latin characters
> as those Latin alphabet characters.
>
> With some unicode knowledge, some help from the analyzer and some regex
> knowledge - such things are easy to find
> for example : <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>>
> finds a sequence where cyrillic (a p b ) are used in words - commonly
> used by spammers
>
> Thomas
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:06.09.2022 16:16
> Betreff:[Assp-test] soft hyphen fooling Bayesian analysis
> --
>
>
>
>
> Is there a way to improve the way that ASSP parses certain special,
> non-printing, characters?  I'm having trouble with spam emails that have
> their body heavily obfuscated with "soft hyphens" slipping through.  They
> all seem to have multipart bodies, first with an iso-8559-1 text part with
> *=AD* interterspersed in words and then an html part with ** all
> over the place.  These are the "soft hyphen," a hyphen that only prints if
> it is needed to break the word to the next line.  It's clever.  The user
> doesn't see the character, but ASSP thinks it's a word boundary.
>
> The part first part
> Content-Type: text/plain; charset="*iso-8859-1*"
> Content-Transfer-Encoding: quoted-printable
> will be plain text, and have have spammy words with *=AD* inserted in the
> middle of them, for example, "This is a sentence with spammy phrase." could
> be 

Re: [Assp-test] soft hyphen fooling Bayesian analysis

[K Post]

Eager to see what you come up with in terms of ignoring the soft hyphen.

 Your <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>> regex is clear, and I
understand using that for scoring purposes, but I'm looking for a way to
potentially treat look-alike characters as the latin character for bayesian
purposes and/or to catch commonly obscured words (like GeekSquad).  Is it
okay if I reply further in my  August 1 post here to keep that in the same
thread?

On Tue, Sep 6, 2022 at 2:06 PM Thomas Eckardt 
wrote:

> >HTML::strip
>
> html parsing to get text parts has nothing to do with html de(en)coding
>
>
> >iso-8559-1
> ASSP processes all content as UTF-8
>
>
> >
> ASSP is aware about this - and replaces soft-hyphens with hard-hyphens -
> and multiple concurrent hard-hyphens with a single one
> How ever - the option to remove the soft-hyphens instead, sounds somehow
> better. Tests are still running.
>
> >My thinking is that if it doesn't display.
> ASSP does'nt know if something displayed or not (and will never know it)
>
>
> >I suspect that other characters will be abused in the same way
>  as well as several BIG5, numerical and other unicode characters are
> already special handled by assp. Other CTL-chars are ignored by assp.
> Everything is converted to UTF8, unicode normalized (including grapheme
> clusters), stemmed and simplyfied.
>
>
> >This kind of obfuscation goes hand in hand with my previous questions
> about considering some non-Latin characters that look like Latin characters
> as those Latin alphabet characters.
>
> With some unicode knowledge, some help from the analyzer and some regex
> knowledge - such things are easy to find
> for example : <<<\P{Cyrillic}\p{Cyrillic}+\P{Cyrillic}>>>
> finds a sequence where cyrillic (a p b ) are used in words - commonly
> used by spammers
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:06.09.2022 16:16
> Betreff:[Assp-test] soft hyphen fooling Bayesian analysis
> --
>
>
>
>
> Is there a way to improve the way that ASSP parses certain special,
> non-printing, characters?  I'm having trouble with spam emails that have
> their body heavily obfuscated with "soft hyphens" slipping through.  They
> all seem to have multipart bodies, first with an iso-8559-1 text part with
> *=AD* interterspersed in words and then an html part with ** all
> over the place.  These are the "soft hyphen," a hyphen that only prints if
> it is needed to break the word to the next line.  It's clever.  The user
> doesn't see the character, but ASSP thinks it's a word boundary.
>
> The part first part
> Content-Type: text/plain; charset="*iso-8859-1*"
> Content-Transfer-Encoding: quoted-printable
> will be plain text, and have have spammy words with *=AD* inserted in the
> middle of them, for example, "This is a sentence with spammy phrase." could
> be written something like
> This is a sentence with sp=ADammy p=ADhr=ADase.
>
> The next mime part is the html, which does the same thing, but uses 
> (html for soft hyphen) mid-word.  So, something like:
> This is a sentence with spammy phrase in it
>
> The whole body of the message is filled with these soft hyphens anywhere
> that there's spammy words/phrases, and in many cases, there are soft
> hyphens every couple of letters across the entire body.  When I do an
> analysis, it appears that the soft hyphen tricks ASSP into thinking that
> each part of the word is a separate word, so for spammy
> phrase, it thinks the words are
> sp ammy p hr ase
>
> I am using HTML::strip.  Would TreeBuilder work better?  I'm concerned
> about performance there.
>
> Is there a way (and is it a good idea) to somehow instruct ASSP to treat
> certain html special characters as ones to ignore, and others to be treated
> as a word separator?  My thinking is that if it doesn't display, then it
> should be ignored when doing bayesian / HMM evaluation.
>
> *https://cs.stanford.edu/people/miles/iso8859.html*
> <https://cs.stanford.edu/people/miles/iso8859.html> has a bunch of
> Control Characters and Special Characters that don't print - or in the case
> of the soft hyphen, only print when the contained word is at the end of a
> line.  I suspect that other characters will be abused in the same way.
>
> This kind of obfuscation goes hand in hand with my previous questions
> about considering some non-Latin characters that look like Latin characters
> as those Latin alphabet characters.
>
> Thanks
>
>
>
>
>
>

[Assp-test] soft hyphen fooling Bayesian analysis

[K Post]

Is there a way to improve the way that ASSP parses certain special,
non-printing, characters?  I'm having trouble with spam emails that have
their body heavily obfuscated with "soft hyphens" slipping through.  They
all seem to have multipart bodies, first with an iso-8559-1 text part with
*=AD* interterspersed in words and then an html part with ** all over
the place.  These are the "soft hyphen," a hyphen that only prints if it is
needed to break the word to the next line.  It's clever.  The user doesn't
see the character, but ASSP thinks it's a word boundary.

The part first part

Content-Type: text/plain; charset="*iso-8859-1*"
Content-Transfer-Encoding: quoted-printable

will be plain text, and have have spammy words with *=AD* inserted in the
middle of them, for example, "This is a sentence with spammy phrase." could
be written something like

This is a sentence with sp=ADammy p=ADhr=ADase.


The next mime part is the html, which does the same thing, but uses 
(html for soft hyphen) mid-word.  So, something like:

This is a sentence with spammy phrase in it


The whole body of the message is filled with these soft hyphens anywhere
that there's spammy words/phrases, and in many cases, there are soft
hyphens every couple of letters across the entire body.  When I do an
analysis, it appears that the soft hyphen tricks ASSP into thinking that
each part of the word is a separate word, so for spammy
phrase, it thinks the words are

sp ammy p hr ase


I am using HTML::strip.  Would TreeBuilder work better?  I'm concerned
about performance there.

Is there a way (and is it a good idea) to somehow instruct ASSP to treat
certain html special characters as ones to ignore, and others to be treated
as a word separator?  My thinking is that if it doesn't display, then it
should be ignored when doing bayesian / HMM evaluation.

https://cs.stanford.edu/people/miles/iso8859.html has a bunch of Control
Characters and Special Characters that don't print - or in the case of the
soft hyphen, only print when the contained word is at the end of a line.  I
suspect that other characters will be abused in the same way.

This kind of obfuscation goes hand in hand with my previous questions about
considering some non-Latin characters that look like Latin characters as
those Latin alphabet characters.

Thanks
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Warnings for unable to resolve IP

[K Post]

Has anything changed with recent dev versions of ASSP in terms of
warnings in the logs for being unable to lookup an IP?

For example:

warning: can't resolve the IP-address for the destination
stonewallkitchenvip.com using the configured DNS-servers


Nothing's wrong with assp, stonewallkitchenvip.com doesn't exist, but I'm
getting a lot of warnings in the logs for misconfigured (or non existent)
domains since updating to the latest ASSP and I don't think I've had that
before.  I can stop the alert emails based on the warning, I just wanted to
see if these are new / different warnings that ASSP is doing now or if
there's been an uptick in bulk sending services doing things wrong.

Thanks
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[K Post]

This is TERRIFIC. Terrific, terrific, terrific.
I've done a lot of regex work in my days mostly in php, vb, and linux batch
scripts, but wasn't aware of the *FAIL syntax.  I was thinking a return of
1 or 0, not no return.  That makes much more sense.

And thanks for the continued examples.  I'd be shocked if more than 10
people out of the couple thousand remaining ASSP users (??) has an
CorrectASSPcfg functions in a regex.

Three small charities that used to use my charity's ASSP installation for
email filtering have gone direct to Office365 and removed ASSP from
the equation in the last 2 years.  I'm still in touch with their staff.
While the like 365's features, they can't stand the spam filtering. You've
spoiled them with never having to sort through a junk folder!!




On Fri, Jul 15, 2022 at 6:37 AM Thomas Eckardt 
wrote:

> Ken - learn perl regular expressions!
>
> >Why is the *FAIL bit in your example of
>
> - the (*FAIL) or (*F) statement makes a regex fail, even a match was found.
>
> Because the return value (e.g. setting $_) from a code execution in a perl
> regex does not modify the 'match found/no match found' flag of the regex.
> But the $_ can be used in a conditional regex to tell the regex engine what
> to do (in which case).
> There are multiple ways to do it. And possibly there are better ways - but
> this one I found nice.
>
> (\@.+\.docusign\.net|next domain|next
> domain|...|...)(?(?{::myWantedDKIMCheck($fh,$+)})|(*FAIL))
>
> explanation:
> ( the matching strings/domains, match captured in $+) # if failed, the
> next parts of the regex are ignored and the regex fails - if matched 'match
> found' is set by the regex engine
>  (?( # start of a
> conditional (yes|no) regex (?(cond)yes|no)
> ?{  # start of the
> code to be executed ?{code}
>
> ::myWantedDKIMCheck # call this sub
>
>   ($fh,$+) # provide the filehandle and the last match
> result to the sub
>
>   }) # end of the code and the condition - the
> return value of the sub is the conditional result
>
> | # the 'yes' part (before the pipe [empty]) -
> if the code returned 1, nothing is to do, regex keeps 'match found'
>
>  (*FAIL) # the 'no' part (after the pipe) - if
> the sub returned 0 or undef make the regex fail (no match found)
>
> ) # end of the conditional regex
>
>
> > seems to return if there's no DKIM  (return unless $this->{isDKIM};)
> wouldn't that not match the regex, so the 60 score wouldn't be applied?
>
> right!
> returns undef in case there is no DKIM-signature found - which makes the
> regex fail (*FAIL) -> no score
>
> If you need it the opposit way change the return value of the sub  - or
> the regex })(*FAIL)|) <- here the 'yes' part sets 'no match found'
> and the 'no' part leaves the 'match found' flag
>
> How ever, these all are *examples* on how you can get such or similar
> problems solved.  It's on you to define rules, to find the required
> $Con{$fh}->{.} flags in assp, to change the regex as needed and to
> write the code.
>
> The sub used in  CorrectASSPcfg should be solid rock against crashes (in
> doubt: eval {} is a good friend)!
>
> Keep in mind: the header checks and body checks are done in a fixed order
> (assp_check_order.txt) - for  example, it is useless to require a Con-flag
> in a headerRe, if this flag was not already processed (set) before by assp!
>be careful if you modify such flags - know what
> you do!
>
>
> >Not just that, but it helps me to understand ways that CorrectASSPcfg
> can be used.
>
> Reading the SSL GUI section, makes this really clear!
>
> CallBack to Verify 
> Call to Configure  .
> 
> 
>
> To make it complete - the following subs in CorrectASSPcfg.pm will be
> called if they are available:
>
> from assp.pl:
>
> set - set hidden variables (prevents the requirement to modify assp.pl
> for  hidden variable changes/settings)
> custom_mlog - do something with or because of log lines
> custom_reply - customize ASSP SMTP replies (literals, error explanations,
> .)
> modMyHeader - check/modify X-ASSP- headers before they are added to
> the mail header
> translateReply - translate MTA SMTP reply codes (some MTA's provides bad
> reply codes)
> customAnalyze - called before a mail is parsed and processed by the
> analyzer
>
> from ASSP_AFC.pm:
>
> AFC_Executable_Detection - custome code to detect executable code in
> attachments, called if no code was found
>

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[K Post]

low the penalty limit.
>
> If a DKIM signature is invalid - assp scores.
> If the domain has ever sent a mail with a valid DKIM signature before (a
> DKIMCache entry is found), assp scores for DKIM if a mail  without a DKIM
> signature from this domain is received.
>
> Now, if there was not added any other score (the mail is 100% ok, except
> DKIM) the mail will pass because the penalty limit is not reached. But you
> want to block the mail if the sender matches @*.*docusign.net*
> <http://docusign.net/>
>
> sender??? ... matches???... - assp has weighted regular expressions -
> like: bombSenderRe - where you can add or remove scoring points
> if you set there
> \@.+\.docusign\.net=>20
>
> all mails from those domains will get a penalty of 20 points, which is
> harmless if there is everything else ok with the mail
> if dkim fails, the penalty limit will be reached and the mail will be
> blocked
> this can be finetuned using :>NWLI
>
> You are also able to implement code in to the regex (for example to check
> for the DKIM result). This is much less complicated than writing a plugin.
> \@.+\.docusign\.net(?{::myWantedDKIMCheck($fh)})=>60
> "score with 60 if the sender matches and the sub
> CorrectASSPcfg::myWantedDKIMCheck returned 1"
>
> Both examples should only show, that there are more ways to get wanted
> results in assp. If someone solved a similar problem using another way, it
> would be nice to hear, how this was done.
>
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:07.07.2022 15:56
> Betreff:Re: [Assp-test] blockStrictDKIMRe -- also thoughts on
> DMARC rejects
> --
>
>
>
> All of your points are clear, and the explanation is greatly appreciated.
>   I now understand why it may be unwise to generally honor reject DMARC
> policy if we've overridden spf/dkim policy once we start manipulating
> results with ASSP.  That makes sense.
>
> I still feel like a *blockStrictDKIMRe* type of new feature, where a
> failed OR missing dkim signature where the message matches the regex would
> be strictly blocked (just like we can do with blockstrictSPFRe for spf
> failures) would be helpful.
>
> For example (hopefully this is more illustrative of the desire), I want to
> outright block any message from @*.*docusign.net* <http://docusign.net/>
> that isn't signed or that has an invalid signature.  I don't care if it's
> from a whitelisted email address, from an IP that's in the SPF record, and
> with a message body that is 100% great.  If there's no DKIM signature or an
> invalid one for a message that matches the regex, reject the message (just
> like their DMARC policy says to do).
>
> Is there another way with current ASSP features to accomplish this only if
> a message matches this proposed regex?
>
> Ken
>
>
> On Fri, Jun 17, 2022 at 4:35 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> >*Would you please consider adding a feature to do the same for a failed
> DKIM signature?*
>
> NO!
>
> Contrary to SPF, a DKIM signature has only two options : OK and FAIL -
> Based on the signature it self or based on a trusted forwarders
> authentication result (ARC).
> A DKIM signature has to be valid every time for any of the above reasons.
>
> > I score failed spf and score failed dkim, so DoDMARC is only scoring
> even though p=reject.
>
> What else makes sense?
> If SPF is scored and DKIM is scored and DMARC is score - AND the resulting
> score does'nt block the mail at the pealtybox, your settings are wrong!
>
>
> >*If DMARC says p=reject, why shouldn't assp outright honor that*,
> regardless of if we have spf / dkim failures set to only score?
>
> SPF has too many options to change/override the original result in assp
> (more or less strict, overwrite, skip ), some these options also exists
> for DKIM.
> If we ignore/change/override   sender policies for SPF and DKIM, it is
> not wise to honor the reject DMARC policy strictly.
>
> Thomas
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:16.06.2022 19:28
> Betreff:[Assp-test] blockStrictDKIMRe -- also thoughts on DMARC
> rejects
> --
>
>
>
> The ability to block failed SPF, instead of just scoring them, for delect
> regex matches has been a terrific feature of ASSP for a long time.
>  (Block SPF Processing Regex* (blockstrictSPFRe

Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[K Post]

All of your points are clear, and the explanation is greatly appreciated.
 I now understand why it may be unwise to generally honor reject DMARC
policy if we've overridden spf/dkim policy once we start manipulating
results with ASSP.  That makes sense.

I still feel like a *blockStrictDKIMRe* type of new feature, where a failed
OR missing dkim signature where the message matches the regex would be
strictly blocked (just like we can do with blockstrictSPFRe for spf
failures) would be helpful.

For example (hopefully this is more illustrative of the desire), I want to
outright block any message from @*.docusign.net that isn't signed or that
has an invalid signature.  I don't care if it's from a whitelisted email
address, from an IP that's in the SPF record, and with a message body that
is 100% great.  If there's no DKIM signature or an invalid one for a
message that matches the regex, reject the message (just like their DMARC
policy says to do).

Is there another way with current ASSP features to accomplish this only if
a message matches this proposed regex?

Ken


On Fri, Jun 17, 2022 at 4:35 AM Thomas Eckardt 
wrote:

> >*Would you please consider adding a feature to do the same for a failed
> DKIM signature?*
>
> NO!
>
> Contrary to SPF, a DKIM signature has only two options : OK and FAIL -
> Based on the signature it self or based on a trusted forwarders
> authentication result (ARC).
> A DKIM signature has to be valid every time for any of the above reasons.
>
> > I score failed spf and score failed dkim, so DoDMARC is only scoring
> even though p=reject.
>
> What else makes sense?
> If SPF is scored and DKIM is scored and DMARC is score - AND the resulting
> score does'nt block the mail at the pealtybox, your settings are wrong!
>
>
> >*If DMARC says p=reject, why shouldn't assp outright honor that*,
> regardless of if we have spf / dkim failures set to only score?
>
> SPF has too many options to change/override the original result in assp
> (more or less strict, overwrite, skip ), some these options also exists
> for DKIM.
> If we ignore/change/override   sender policies for SPF and DKIM, it is
> not wise to honor the reject DMARC policy strictly.
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:16.06.2022 19:28
> Betreff:[Assp-test] blockStrictDKIMRe -- also thoughts on DMARC
> rejects
> --
>
>
>
> The ability to block failed SPF, instead of just scoring them, for
> delect regex matches has been a terrific feature of ASSP for a long time.
>   (Block SPF Processing Regex* (blockstrictSPFRe) )   *Would you please
> consider adding a feature to do the same for a failed DKIM signature?*
> Outright blocking of a matching message that fails DKIM, regardless of the
> domain's DMARC settings.   -- maybe that's not necessary if DoDMARC will
> honor =reject, see more below.
>
> Reasoning:
> I already score failed DKIM signatures, but I can't set that score too
> high because so many organizations still send messages through 3rd parties
> with invalid DKIM signatures.  It really is incredible how many I see.  But
> for frequently abused sender addresses (docusign for example), who are
> often spoofed but send otherwise unspammy content, I want to outright block
> if the DKIM signature fails.  blockStrictSPFRe usually works because these
> bad DKIM sigs are on mails that also violate SPF rules, still though it
> would be helpful if I could also just say "if a specific regex is matched
> on an email with an invalid DKIM, reject the message"
>
> RELATED: DMARC p=reject should always reject if failed
> Docusign.net has a dmarc rule of p=reject.  I want to honor that.  The
> last scam that came in from them failed SPF and failed DKIM validation, but
> the message was from a whitelisted address..  DoDMARC says that the
> blocking will be the "most less aggressive" (least aggressive) and the
> published DMARC record.  I score failed spf and score failed dkim, so
> DoDMARC is only scoring even though p=reject.
>
> Enable DMARC Check (DoDMARC)
> If enabled and ValidateSPF and DoDKIM are enabled and the sending domain
> has published a DMARC-record/policy, assp will act on the mail according to
> the senders DMARC-policy using the results of the SPF and DKIM check and
> validating the SPF/DKIM address/domain Identifier Alignment rules (RFC7489
> section 3). It is safe to leave this feature ON, it will not produce false
> positives! The blocking mode (block, monitor, score, testmode) is adapted
> from the most less aggressive setting of ValidateSPF and DoDKIM - and the
> published DMARC record ([p][sp

[Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects

[K Post]

The ability to block failed SPF, instead of just scoring them, for
delect regex matches has been a terrific feature of ASSP for a long time.
  (Block SPF Processing Regex* (blockstrictSPFRe) )   *Would you please
consider adding a feature to do the same for a failed DKIM signature?*
Outright blocking of a matching message that fails DKIM, regardless of the
domain's DMARC settings.   -- maybe that's not necessary if DoDMARC will
honor =reject, see more below.

Reasoning:
I already score failed DKIM signatures, but I can't set that score too high
because so many organizations still send messages through 3rd parties with
invalid DKIM signatures.  It really is incredible how many I see.  But for
frequently abused sender addresses (docusign for example), who are often
spoofed but send otherwise unspammy content, I want to outright block if
the DKIM signature fails.  blockStrictSPFRe usually works because these bad
DKIM sigs are on mails that also violate SPF rules, still though it would
be helpful if I could also just say "if a specific regex is matched on an
email with an invalid DKIM, reject the message"

RELATED: DMARC p=reject should always reject if failed
Docusign.net has a dmarc rule of p=reject.  I want to honor that.  The last
scam that came in from them failed SPF and failed DKIM validation, but the
message was from a whitelisted address..  DoDMARC says that the blocking
will be the "most less aggressive" (least aggressive) and the published
DMARC record.  I score failed spf and score failed dkim, so DoDMARC is only
scoring even though p=reject.

Enable DMARC Check (DoDMARC)
If enabled and ValidateSPF and DoDKIM are enabled and the sending domain
has published a DMARC-record/policy, assp will act on the mail according to
the senders DMARC-policy using the results of the SPF and DKIM check and
validating the SPF/DKIM address/domain Identifier Alignment rules (RFC7489
section 3). It is safe to leave this feature ON, it will not produce false
positives! The blocking mode (block, monitor, score, testmode) is adapted
from the most less aggressive setting of ValidateSPF and DoDKIM - and the
published DMARC record ([p][sp]=[reject][quarantine]). Scoring is done
using dmarcValencePB.


*If DMARC says p=reject, why shouldn't assp outright honor that*,
regardless of if we have spf / dkim failures set to only score?

Thanks
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] blocking new MS doc vunerability (URI attack vector)

[K Post]

Thanks for getting back with me Thomas.  I know that we can't block ms-msdt
because that's downloaded by Word after opening the file, but I was talking
about blocking files that have the URI reference, like:
  http://schemas.openxmlformats.org/officeDocument/2006/relationships/oleObject;
Target="hxxps://www[.]xmlformats[.]com/office/word/2022/wordprocessingDrawing/RDF842l.html!"
TargetMode="External"/>

Basically, if a document has an external reference, strip the file out of
the email, essentially inspecting it like we do .docx files looking for bad
content - similar to removing a PDF that contains javascript.


On Mon, Jun 13, 2022 at 3:55 AM Thomas Eckardt 
wrote:

> This is not possible because:
>
> 
> Note that the suspicious scheme ("ms-msdt:/")* is** not **present in the
> document*. It's present in the first stage payload that will be
> downloaded by Office.
> 
> and
> 
> The document contains an external reference *pointing to a malicious URL*:
> 
>
> If the malicious URL is known, it can be detected by assp using URIBL.
> Keep in mind that those malicious URL's can be generated and changed very
> quickly!
>
> >Hopefully clamav will eventually catch it,
>
> I don't think this is possible for every case. Also traditional AV
> scanners need to know all used malicious URL's. Only a behavior analysis of
> the document will be able to detect the malicious download and playload.
>
>
> Solutions for CVE-2022-30190 are provided by Microsoft:
>
>
> https://msrc-blog.microsoft.com/2022/05/30/guidance-for-cve-2022-30190-microsoft-support-diagnostic-tool-vulnerability/
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:31.05.2022 20:14
> Betreff:[Assp-test] blocking new MS doc vunerability (URI attack
> vector)
> --
>
>
>
> Hello Thomas,
>
> Any way for ASSP to block this kind of thing?
>
>
> *https://isc.sans.edu/forums/diary/New+Microsoft+Office+Attack+Vector+via+msmsdt+Protocol+Scheme+CVE202230190/28694*
> <https://isc.sans.edu/forums/diary/New+Microsoft+Office+Attack+Vector+via+msmsdt+Protocol+Scheme+CVE202230190/28694>
>
> Hopefully clamav will eventually catch it, but be nice great to be able
> strip documents off using AFC if they contain the URI protocol, just like
> we do for VBA code, etc.
>
> Thanks___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] bombHeaderRe matching every email

[K Post]

Hi Scott,
Did you ever figure this out?
I'm no regex wiz like Thomas is, but what you have appears pretty simple to
me -- and I don't see anything wrong with it...
I tried

from\:.*\_

in testRE and see it matching everything too.  I don't understand why.  I
know this doesn't help you with why this is happening, but figured that it
would at least help to hear that you're not the only one whose system
generates that result.




On Wed, Jun 1, 2022 at 5:32 PM Scott MacLean  wrote:

> I've been seeing a bunch of spam getting through my filter recently, and
> they all have the same thing in common: an underscore at the beginning
> of the "From" and/or "Subject" lines. This should be really easy to pick
> up with bombHeaderRe, but something's not working.
>
> Here's an example of the spam I'm seeing:
>
> From:_Male Health  >
> Subject:_Size matters and we can help
>
> Sometimes there is a space in between the colon and the underscore,
> usually there is not.
>
> Here is the regex I added to my bombHeaderRe:
>
> From\:.*\_=>60
> Subject\:.*\_=>60
>
> However, I quickly realized that this was tagging EVERY email coming
> through the server! For instance, here's an email:
>
> From: Readly 
>
> And looking at mail analysis, it's being caught by this regex, even
> though there is no underscore:
>
> BombHeader RE: 'highest match: "(matchlength:84) From: Readly
>  matching bombHeaderRe(file:files/bombheaderre.txt[line 188]): 'From\:.*_'
>
> Any idea what's going wrong and causing this?
>
>
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] blocking new MS doc vunerability (URI attack vector)

[K Post]

Hello Thomas,

Any way for ASSP to block this kind of thing?

https://isc.sans.edu/forums/diary/New+Microsoft+Office+Attack+Vector+via+msmsdt+Protocol+Scheme+CVE202230190/28694

Hopefully clamav will eventually catch it, but be nice great to be able
strip documents off using AFC if they contain the URI protocol, just like
we do for VBA code, etc.

Thanks
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] assp development switched to perl 5.34.0

[K Post]

Nicely done!  How'd you get this done it's released at strawberryperl.com?

On Tue, Mar 8, 2022 at 10:18 AM Thomas Eckardt 
wrote:

> Hi all,
>
> the assp development is switched to perl 5.34.0
>
> strawberry perl 5.34.0 (for win_x64) is available at the sourceforge assp
> download repository
>
> ASSP V2 multithreading/ASSP V2 module
> installation/strawberry-perl-5.34.0.1-64bit-relocateable_4-assp.7z
>
>
> Thomas
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 22019

[K Post]

Thank you Thomas.

You know that I love the SPF addition option for IP address lists!!  Happy
to no longer need to use the script I wrote to accomplish this.  I'm glad
(and proud) that you've recognized this as a good idea.

I have some questions about the new SPF functionality as well as the
auto-restart if possible after SEGV error on Windows when ASSP is run as a
service.  Where on the forums would you like me to post these questions, or
is here okay now?





On Wed, Jan 19, 2022 at 8:26 AM Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 22019:
>
> - If the used perl version was compiled without linking to libcypto, the
> perl 'crypt' command was without function. This caused the assp internal
> encryption engine to fail
>   and all encrypted configuration values and files were unuseable. The
> password for the root user was not stored.
>   Now, if such a perl version is found by assp, it will try to load the
> module Crypt::UnixCrypt, which has the same function like the perl internal
> crypt command. If this module
>   can't be loaded, assp will die and shows a related hint at the command
> line.
>
> - ASSP contains code to handle unexpected SEGV signal errors. The past has
> shown, that recovering assp to a normal state after a SEGV occured is
> impossible.
>   Most times the maillog.txt was filled with thousands or even million of
> error lines.
>   For this reason, assp will now try a restart, if a SEGV happens - if a
> restart is not possible, the assp process will be ended.
>
> - If a query string in RBL-, RWL- and URIBL-queries was longer than 62
> byte, the query was not processed by assp. The length of such a query
> string is now limited to 253 byte.
>   The length of the labels in a domain string are limited to 63 byte.
>
>
> changed:
>
> - It was possible for years now (but undocumented) to provide api keys for
> RBLServiceProvider and URIBLServiceProvider.
>   The documentation for both parameters is extended.
>   ...  It can be possible, that you need to provide a privat key or ID in
> the query string for a URIBL Service Provider - like:
> your-key.query-data.uribl-provider.org
>In this case, define the URIBL Service Provider like:
> your-key.$DATA$.uribl-provider.org
>The string $DATA$ will be replaced by the queried data in each
> request.
>
>
> - A new function is implemented in to all IP-address lists. It is now
> possible to include all IP's of a SPF-record of a domain in to IP-address
> lists.
>   The help text is extended:
>    For several IP-address lists in assp, it can be advantageous to
> include all IP's (and ranges) listed in the SPF-record of a specific domain
> (for example in noPB, noHelo, whiteListedIPs, ...).
>To provide this, simply write SPF: in front of the domain name in a
> list entry - like 182.82.10.0/24|SPF:amazon.com|2201:1::1
>  .
>In this example assp will replace the term SPF:amazon.com with the
> list of all IP's and resolved IP's defined in the SPF-record of amazon.com
> .
>This will also work for IP lists in a group definition. Assignments
> made to such an entry - like SPF:amazon.com=>[usergroup] will be added to
> each resolved SPF-IP-address.
>
>
> Thomas
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

I can not decypher what this means:

most - where? -> forum , bug tracker , self testing, forced by attackers

and it's my lack of clarity on your short replies which leads me to
question further.

I need to find a way to still be able to report my findings and ask my
questions without being a bother.  The last thing I want to be is a burden,
but I have no other way to communicate with you, as the sole developer on a
project that has minimal user communication other than what you and I
discuss.

While I wish it were easier for me to be more concise, my persistence and
full description of issues and challenges has resulted in far more than the
one change you referenced.  I've outlined some of them from the last 7
versions below.


   - 1 of the changes in 21277 is because of my report.  Very slow startup
   of the rebuild process.
   - 2+ of the changes in 21280 stemmed from my messages.  Too many open
   files in Windows, early bad SSL changes, catching invalid regex instead of
   ASSP crashing
   - 21287 & 21290: your changes to griplist folder creation, changes/fixes
   to BereleyDB error logging, gui changes, and windows file descriptor
   changes are because of things I've brought up
   - 21293: The NWLI changes are because of what I asked
   - 7 of the 8 changes in 21302 are because of my reports, questions,
   requests, and suggestions.  Related to external file change times not being
   recorded in ASSP (long time bug), improvement in a single file changing
   causing all to be reloaded, changes to the analyzer for reports from
   Outlook, corpus cleanup for DKIM WL/NP matches.
   - 21396 more changes because of discussions about Outlook reporting
   (FYI  forward as attachment from Outlook still doesn't result in correct
   analyze reports nor does multiple report attachments in a single email from
   Outlook work at all.)
   - 21317 After my questions about the unusual request for help for a way
   to match username of the recipient to the sender we discovered the bug
   about unoptimized weighted bombs with a scoring parameter and the bug with
   definite statements

And over the years you've added useful features and fixed bugs because of
my questions or requests which you originally dismissed as being misguided

There's a trend here. When I'm active on this forum, I discuss things that
lead you to improve ASSP which benefits everyone.

If I had asked my question and then not responded to your short "no" or
"have you thought about this" type of replies, would these changes have
been made?  If I hadn't fully described the issue/question/challenge, how
would you have known what I was talking about?

I will now step away from this form as requested for as long as I am able.
I do hope that you are willing to entertain future questions/concerns once
I return, if not for me, then for the rest of the quiet spam fighters on
this list.

On Sun, Nov 14, 2021 at 5:59 AM Thomas Eckardt 
wrote:

> >How many of the changes in the last 10 or so versions of ASSP have been
> from the requests of anyone else on this list?
>
> how many? 1 at 5.11.2021 - weight bug
>
> most - where? -> forum , bug tracker , self testing, forced by attackers
>
> You may use the forum, where everyone is free to skip reading your endless
> posts and blogs. It takes simply too much time to pick up the 1 to 5% of
> helpful content and to be forced by you to answer also the rest.
>
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:14.11.2021 00:14
> Betreff:Re: [Assp-test] Concept Question: Scan entire message for
> Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?
> --
>
>
>
> I don't know what I've done to deserve that reply, but regardless, I'm
> sorry to have upset you.  I will take a long break from posting
> further here, but please do know that I'm appreciative of your continued
> support of this important program.
>
> Before I go, please entertain these thoughts:
>
> I hope that you're able to re-evaluate your request for me to go away.
> I've recommended more very good change requests to ASSP than ones that you
> consider to be bad.  I'm not able to implement them myself.  I'm not
> perfect, but your request for me to sign off of this list, which is a
> critical resource, is unfair.
>
> How many of the changes in the last 10 or so versions of ASSP have been
> from the requests of anyone else on this list?  How many bugs have been
> quashed because of things I've discovered?  How many improvements did you,
> and only you, make because of questions I've asked and because of feature
> requests I've made (recently and over the many years)?
>
> Are you angry because I'm (a

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

I don't know what I've done to deserve that reply, but regardless, I'm
sorry to have upset you.  I will take a long break from posting
further here, but please do know that I'm appreciative of your continued
support of this important program.

Before I go, please entertain these thoughts:

I hope that you're able to re-evaluate your request for me to go away.
I've recommended more very good change requests to ASSP than ones that you
consider to be bad.  I'm not able to implement them myself.  I'm not
perfect, but your request for me to sign off of this list, which is a
critical resource, is unfair.

How many of the changes in the last 10 or so versions of ASSP have been
from the requests of anyone else on this list?  How many bugs have been
quashed because of things I've discovered?  How many improvements did you,
and only you, make because of questions I've asked and because of feature
requests I've made (recently and over the many years)?

Are you angry because I'm (adminitedly) long winded?  Please understand
that this is not out of disrespect, it's because I want to make sure that
I'm being clear.  When I get a short answer, I try to continue the
conversation.  This is a discussion list after all.

Are you angry because I'm persistent?  My persistence is also not out of
disrespect, it's because I'm inquisitive,  am by no means an expert in
coding or the inner workings of spam detection, and have a burning desire
to continue to see ASSP improve.  Often I ask a detailed question, and only
get an answer back from you like "have you considered this?" or "no"
without explanation.  Is it so bad that I ask why not?  I wait patiently
for your replies, but do inquire more if my questions haven't been fully
answered.  If you don't have the time or desire to entertain my questions,
so be it, but please remember that most of what I ask has ultimately led to
you eventually improving ASSP.

Anyway, I don't expect and certainly don't require a reply here.  But
please know that my intentions are pure, I'm charitable, patient, and a
good person. It hurts deeply that you seem to think otherwise.  I don't
have the experience nor the ability that you do, not even close, but I like
to think that even if I can be frustrating that I'm ultimately bring some
good to the ASSP world by offering suggestions and asking questions.



On Sat, Nov 13, 2021 at 3:56 AM Thomas Eckardt 
wrote:

> Ken , it would be nice if you consider to signoff this list or at least to
> no longer post here.
>
> Thank you.
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:12.11.2021 22:46
> Betreff:Re: [Assp-test] Concept Question: Scan entire message for
> Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?
> --
>
>
>
> First off, WOW.  Our rebuild times are in no way similar.   At first I
> thought it was you with fancy SSD's and lots of horsepower, but I'm seeing
> now that you have both useDB4Rebuild off and RebuildUseFileModel on.  The
> opposite of my settings.  I have useDB4Rebuild on and never enabled the
> RebuildUsedFileModel after initial attempts were failing (Early on with
> that feature).  useDB4Rebuild is the default and I was always worried about
> RAM when I started using ASSP 10+ years ago and never looked back.
>
> A long rebuild time doesn't bother me, but seeing how fast you can do one
> has got me back to needing to test the settings on my end again.  Thanks
> for that encouragement.
>
>
> I'm worried that going up to 50k maxbytes on my system seemed to cause a
> lot of false positives.  I don't understand how that's possible, but it's
> what happened.  I would have thought it was the other way around, too much
> spam getting through vs. too much legit being blocked.  Plus, I don't think
> that generally using that much for bayesian is necessary (or maybe it's
> even detrimental?)  Accuracy was very high for me at  6k and 10k, but I was
> missing the bombs.
>
>
> The question remains for me about the >CONCEPT< of optionally scanning
> more of a message at the time of attempted delivery for bombs.  ClamAV uses
> its own maximum size setting.  Why not also give us that option for Bombs?
> For the case I explained where bombs are late in the email body and likely
> other scenarios, don't you think it would be helpful to have a
> BombAddlBytes variable in the GUI?
>
> You know there's no way that I could ever code a plugin and that there's
> even less of a chance of this charity paying for one to be built!  I still
> have duct tape holding my desk chair together.
>
> Modifying getbody seems pretty straight forward.  Add a new variable
> called $bombdataref that would be us

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

mprove detection rates)
>
> If you need to process complete mails for bombs - you'll need to write
> your own level 2 assp-plugin.
>
> Thomas
>
>
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:12.11.2021 16:56
> Betreff:Re: [Assp-test] Concept Question: Scan entire message for
> Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?
> --
>
>
>
>
> Absolutely I've thought about this.  I consider everything I post prior to
> posting.
>
> Can you briefly explain why the ability to scan (MaxBytes + some
> additional amount)kb on incoming mails for bombs but only use MaxBytes for
> bayesian and the rebuild would be such a bad idea?
>
> Since you questioned if I ever thought about this, here's what the thought
> process is and the reason for the request.  Maybe I didn't explain myself
> well enough in the previous messages:
>
> The MaxBytes "documentation" says to lower it to 3000 for a mature
> installation, but 10x larger than that if you can handle it.
>
> How many bytes of the message body will ASSP look at - the message header
> is always included in all checks. Mails stored in the collecting folders
> will be truncated to this size, if StoreCompleteMail is disabled. *The
> average of Ham messages (message body) is 6K, the average of Spam messages
> is 3K.* Usually the spam folder will be filled quicker than the notspam
> folder, therefore set this value to 4000 to get more wordpairs per Ham
> Message. When both folders are close to the maxfiles limit, reduce it to
> 3000.
>
> If your system is fast enough and has enough RAM multiply all the above
> recommendations and the default value by ten.
>
>
> The gui doesn't say "IF the average is 6k ham, 3k spam," is says that it
> IS 6k ham / 3k spam.  That's not true of my installation.  My average spam
> size, as I've mentioned before, has a median size of about 20kb because of
> all of the html in them.  And not-spam has a median size of 40kb.  Using
> the logic in your gui, *I believe I should set my MaxBytes to 20kb*, the
> median size of my spam corpus.
>
> But, if I set my MaxBytes to 20kb (which it appears to be able to handle
> okay, rebuilding in an hour and change), then bombs after 20kb aren't
> detected when a message is attempting delivery.
>
> Why does this matter to me?
> We're seeing messages with @*gmail.com* <http://gmail.com/> and @
> *whatever.onmicrosoft.com* <http://whatever.onmicrosoft.com/> addresses
> that are copying legitimate looking order receipts from vendors like
> Amazon.com, BestBuy (US based big box electronics store), and Norton.  Many
> look identical to a legitimate message.  Ultimately, they want to call them
> on the phone and give your credit card number, using the guise that they're
> going to refund it.  Classic scam.
>
> These messages will always pass bayesian, they read identically to real
> messages.  BUT, I can detect some with the phone numbers that they direct
> people to.   The email addresses change frequently, but the scam phone
> numbers remain pretty constant.  I could maintain a list of known bad phone
> numbers (also available online) to capture these messages before they're
> delivered.  Simple.  If the message has one of these phone numbers, score
> it such that it'll get blocked.
>
> *The problem with many of these emails is that the phone number is way
> past the 3k mark, and past the 20k mark too.  The scammers have a bunch of
> HTML in the "confirmation" email, just like real stores tend to do.  I
> tried increasing MaxBytes up to 50kb, which easily caught messages with
> bombs later in the body, but that then seemed to cause a lot of false
> positives and obviously much longer rebuild process.  *
>
> If there could be a "continue canning for bombs for ___kb after maxbytes"
> setting, that would let bombs later in the body be detected.  I don't know
> what the downside to having such a feature would be.
>
>
> Based on your reaction to my question, I'm obviously missing something
> important.
>
>
>
>
>
> On Thu, Nov 11, 2021 at 1:38 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> >Is there logic to having a separate MaxBytes setting like
> MaxBytesForBombs that's used only during message delivery?  That way, the
> entire message can be scanned for bombs, but the rebuild could use a lower
> number to better balance the differential between the average sized spam
> and average sized not-spam message.
>
> DID YOU EVER thougth about that ??? Or d

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

Absolutely I've thought about this.  I consider everything I post prior to
posting.

Can you briefly explain why the ability to scan (MaxBytes + some additional
amount)kb on incoming mails for bombs but only use MaxBytes for bayesian
and the rebuild would be such a bad idea?

Since you questioned if I ever thought about this, here's what the thought
process is and the reason for the request.  Maybe I didn't explain myself
well enough in the previous messages:

The MaxBytes "documentation" says to lower it to 3000 for a mature
installation, but 10x larger than that if you can handle it.

How many bytes of the message body will ASSP look at - the message header
is always included in all checks. Mails stored in the collecting folders
will be truncated to this size, if StoreCompleteMail is disabled. *The
average of Ham messages (message body) is 6K, the average of Spam messages
is 3K.* Usually the spam folder will be filled quicker than the notspam
folder, therefore set this value to 4000 to get more wordpairs per Ham
Message. When both folders are close to the maxfiles limit, reduce it to
3000.
If your system is fast enough and has enough RAM multiply all the above
recommendations and the default value by ten.


The gui doesn't say "IF the average is 6k ham, 3k spam," is says that it IS
6k ham / 3k spam.  That's not true of my installation.  My average spam
size, as I've mentioned before, has a median size of about 20kb because of
all of the html in them.  And not-spam has a median size of 40kb.  Using
the logic in your gui, *I believe I should set my MaxBytes to 20kb*, the
median size of my spam corpus.

But, if I set my MaxBytes to 20kb (which it appears to be able to handle
okay, rebuilding in an hour and change), then bombs after 20kb aren't
detected when a message is attempting delivery.

Why does this matter to me?
We're seeing messages with @gmail.com and @whatever.onmicrosoft.com
addresses that are copying legitimate looking order receipts from vendors
like Amazon.com, BestBuy (US based big box electronics store), and Norton.
Many look identical to a legitimate message.  Ultimately, they want to call
them on the phone and give your credit card number, using the guise that
they're going to refund it.  Classic scam.

These messages will always pass bayesian, they read identically to real
messages.  BUT, I can detect some with the phone numbers that they direct
people to.   The email addresses change frequently, but the scam phone
numbers remain pretty constant.  I could maintain a list of known bad phone
numbers (also available online) to capture these messages before they're
delivered.  Simple.  If the message has one of these phone numbers, score
it such that it'll get blocked.

*The problem with many of these emails is that the phone number is way past
the 3k mark, and past the 20k mark too.  The scammers have a bunch of HTML
in the "confirmation" email, just like real stores tend to do.  I tried
increasing MaxBytes up to 50kb, which easily caught messages with bombs
later in the body, but that then seemed to cause a lot of false positives
and obviously much longer rebuild process.  *

If there could be a "continue canning for bombs for ___kb after maxbytes"
setting, that would let bombs later in the body be detected.  I don't know
what the downside to having such a feature would be.


Based on your reaction to my question, I'm obviously missing something
important.





On Thu, Nov 11, 2021 at 1:38 AM Thomas Eckardt 
wrote:

> >Is there logic to having a separate MaxBytes setting like
> MaxBytesForBombs that's used only during message delivery?  That way, the
> entire message can be scanned for bombs, but the rebuild could use a lower
> number to better balance the differential between the average sized spam
> and average sized not-spam message.
>
> DID YOU EVER thougth about that ??? Or do you only write
> something to fillup the community mailing list?
>
> No - no way!
>
> Thomas
>
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:10.11.2021 20:22
> Betreff:Re: [Assp-test] Concept Question: Scan entire message for
> Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?
> --
>
>
>
> After about 12 weeks of going from MaxBytes of 4k to MaxBytes of 50k, 've
> seen:
> 1) Rebuild go from just over an hour (with 30k MaxFiles) to just over 2
> hours.  I'm fine with that, there's more to scan
> 2) Bomb detections improve, as a lot of what's detected is beyond the 20k
> or 30k mark
> 3) but, bayesian false positives going way up.  Lots of mail that would
> have (correctly) been delivered, is now getting too high of a score and is
> blocked.
>
> Surely #3 is specific to the

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

After about 12 weeks of going from MaxBytes of 4k to MaxBytes of 50k, 've
seen:
1) Rebuild go from just over an hour (with 30k MaxFiles) to just over 2
hours.  I'm fine with that, there's more to scan
2) Bomb detections improve, as a lot of what's detected is beyond the 20k
or 30k mark
3) but, bayesian false positives going way up.  Lots of mail that would
have (correctly) been delivered, is now getting too high of a score and is
blocked.

Surely #3 is specific to the types of messages my users are getting and I
can tweak settings.  BUT, it makes me raise this question again:
Is there logic to having a separate MaxBytes setting like MaxBytesForBombs
that's used only during message delivery?  That way, the entire message can
be scanned for bombs, but the rebuild could use a lower number to better
balance the differential between the average sized spam and average sized
not-spam message.



On Mon, Nov 1, 2021 at 2:43 PM K Post  wrote:

> When looking at the "Use this HTML Parser" section on the GUI, I found
> this line:
>
> it is recommended to set MaxBytes to 5 (be carefull on heavy load
> systems - spam bomb regular expressions will take longer using 5!).\
>
> I'm going to change my settings and see how bad the rebuild time is.  I've
> got enough processing power and RAM now, but the disks aren't SSD.  Just a
> 4 disk Raid 1+0 traditional HDD setup.  We'll see...
>
> Since HTMl email accounts for a big percentage of all mail,  might it be a
> good idea to update/expand the guidance in the MaxBytes section of the
> GUI?
>
>
>
> On Fri, Oct 29, 2021 at 8:40 PM K Post  wrote:
>
>> Summary:
>> *Should/could any consideration be given to having ASSP scan the entire
>> message at the time it is received for Bombs (only), while still using
>> MaxBytes for Bayesian/HMM?*
>>
>> We've been having some cleverly crafted messages slipping through all
>> filters that would be easy to catch with Bombs if only the catchable
>> content came before MaxBytes.  These messages are 20kb+, They have a scam
>> phone number at the very end of the larger than MaxBytes messages.  I
>> want/need to use bombs to catch the scam phone numbers.
>>
>> With MaxBytes set to 3000, which is useful for faster RebuildSpamDB,
>> these BombDataRE matches just aren't being caught.  If I increase MaxBytes,
>> my BombDataRE catches them, but then rebuildspamdb is (probably? see below)
>> longer than it needs to be.
>>
>> So, is there any value in considering a* MaxBytesAdditionalForBombs *variable
>> which would be *added to MaxBytes *and only used when scanning for bombs
>> as messages arrive?   Would that kill performance??  Other downsides?
>>
>> We could still only look at MaxBytes for Bayesian/HMM since it's only
>> MaxBytes used when building those databases.
>>
>> What do you think?
>>
>> And while we're talking MaxBytes:
>> I've asked this before, is the guidance for 3kb for MaxBytes once there's
>> a mature corpus still a valid recommendation?  With unlimited horsepower
>> and ram, sure, why not, do 30kb or 100kb.  That's not my reality, so I want
>> to see where to best allocate resources. If 3kb is still the guidance, even
>> though the spam files I'm seeing have a median size around 20kb, so be it.
>> I feel like when that guidance was written, html wasn't used as
>> prolifically in spam.  The median size of notspam in my corpus is about
>> 40kb.  That's determined unscientifically by sorting by size and scrolling
>> to approximately half way down.
>>
>> Thanks.  Have a good weekend.
>> Ken
>>
>>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[K Post]

Ok, then I still think it's an idea worth considering at some point in the
future.  Alternatives exist, but don't *exactly* do what I want.  Bonus
score based solely on signature, regardless of what Senderbase returns (to
help bulk mailers through who might use AWS or somewhere else that's abused
often).  It takes all of the perfection that DWIMWLAddresses is, and just
helps a message along, instead of outright whitelisting it.  Again, not
high priority or even that super useful, just raised it as an idea.  I'll
let it be, but if at some point in the future, you're so inclined
Thanks for hearing me out!

On Tue, Nov 9, 2021 at 12:32 AM Thomas Eckardt 
wrote:

> >are you saying that BombRe will look at headers that ASSP ads
>
> No, it looks only in to the original header.
>
> >I'm still worried about fake/invalid DKIM still getting the bonus score,
>
> Invalid DKIM signatures should be blocked or scored very high, so the
> bonus score does not matter
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:09.11.2021 05:53
> Betreff:Re: [Assp-test] Another Concept Question:
> DKIMBousScoreList
> --
>
>
>
> ah, wait, are you saying that BombRe will look at headers that ASSP ads,
> like X-ASSP-DKIM-Identity (which would only be added for a valid
> signature)?   (!)  I always assumed that the bomb functionality was
> only on the mail's original headers.
>
> On Mon, Nov 8, 2021 at 2:28 PM K Post <*nntp.p...@gmail.com*
> > wrote:
> The bombHeaderRe with the DEFINE or list should be sufficient.  I'm still
> worried about fake/invalid DKIM still getting the bonus score, but this
> will have to do.  Thanks.
>
> On Mon, Nov 8, 2021 at 12:01 PM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> I told you to score such domains elsewhere - just do it and the result is
> the same like you wanted.
>
> for example:
>
> bombHeaderRe:
>
> \nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(
> The_Wanted_IDENTITY))\;=>the_wanted_negative_score
>
> currently the (?(DEFINE)...) is not working with assp (is destroyed if
> a-d-n-o-r is not set for the file) - but the next version will do it  -
> and you can use:
>
> (?(DEFINE)(?the_wanted_identity|ident2|ident3|..))\nDKIM-Signature:(?:[
> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
> - e.g. -10
> (?(DEFINE)(?the_wanted_identity|ident5|ident6|..))\nDKIM-Signature:(?:[
> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
> - eg -20
> ...
>
> CLOSED for me
>
>
> Thomas
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:05.11.2021 20:03
> Betreff:Re: [Assp-test] Another Concept Question:
> DKIMBousScoreList
> --
>
>
>
> Having the dkimBonusScoreList would be like applying
> dkimBonusValancePB but ONLY for those that DKIM validate AND are on the
> scorelist.  Here's why I think that would be helpful and what you proposed
> could be problematic.  Essentially: I'm thinking: "look, this organization
> usually sends good stuff, but not always.  They might also have people
> sending non-dkim signed messages through a myriad of channels.  Deal with
> them separately, but if we KNOW it's from them because of their DKIM
> signature, help that message get through with the idea that it'll be
> stored in okmail unless whitelisted through something other than dkim."
>
> > there is already dkimOkValencePB - increase it
> But a high percentage of all messages that are received, spam and not,
> have valid signatures.  I don't think we should use that to give a bonus
> regardless of who the signer is.  All gmail messages are signed, almost
> everyting from office365.  Yes, I could do a univieral bonus then reduce
> gmail and *onmicroosft.com* <http://onmicroosft.com/>, but that doesn't
> get 365 users with their own signatures and all of the millions of
> other domains out there.
>
> It was one thing when DKIM signing was a new concept and only legit
> businesses signed messages.  Now that most senders are signing, giving  a
> bonus would let an awful lot of spam slip through under the rejection
> scoring threshold.
>
> >reduce the score for certain domains by blackListedDomains, SenderBase or
> anywhere else - if needed
> Senderbase won't work for those using AWS as an example - too many
> spammers us

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[K Post]

ah, wait, are you saying that BombRe will look at headers that ASSP ads,
like X-ASSP-DKIM-Identity (which would only be added for a valid
signature)?   (!)  I always assumed that the bomb functionality was
only on the mail's original headers.

On Mon, Nov 8, 2021 at 2:28 PM K Post  wrote:

> The bombHeaderRe with the DEFINE or list should be sufficient.  I'm still
> worried about fake/invalid DKIM still getting the bonus score, but this
> will have to do.  Thanks.
>
> On Mon, Nov 8, 2021 at 12:01 PM Thomas Eckardt 
> wrote:
>
>> I told you to score such domains elsewhere - just do it and the result is
>> the same like you wanted.
>>
>> for example:
>>
>> bombHeaderRe:
>>
>> \nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(
>> The_Wanted_IDENTITY))\;=>the_wanted_negative_score
>>
>> currently the (?(DEFINE)...) is not working with assp (is destroyed
>> if a-d-n-o-r is not set for the file) - but the next version will do it  -
>>   and you can use:
>>
>> (?(DEFINE)(?the_wanted_identity|ident2|ident3|..))\nDKIM-Signature:(?:[
>> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
>> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
>> - e.g. -10
>> (?(DEFINE)(?the_wanted_identity|ident5|ident6|..))\nDKIM-Signature:(?:[
>> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
>> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
>> - eg -20
>> ...
>>
>> CLOSED for me
>>
>>
>> Thomas
>>
>>
>>
>> Von:"K Post" 
>> An:"ASSP development mailing list" <
>> assp-test@lists.sourceforge.net>
>> Datum:05.11.2021 20:03
>> Betreff:Re: [Assp-test] Another Concept Question:
>> DKIMBousScoreList
>> --
>>
>>
>>
>> Having the dkimBonusScoreList would be like applying
>> dkimBonusValancePB but ONLY for those that DKIM validate AND are on the
>> scorelist.  Here's why I think that would be helpful and what you proposed
>> could be problematic.  Essentially: I'm thinking: "look, this organization
>> usually sends good stuff, but not always.  They might also have people
>> sending non-dkim signed messages through a myriad of channels.  Deal with
>> them separately, but if we KNOW it's from them because of their DKIM
>> signature, help that message get through with the idea that it'll be
>> stored in okmail unless whitelisted through something other than dkim."
>>
>> > there is already dkimOkValencePB - increase it
>> But a high percentage of all messages that are received, spam and not,
>> have valid signatures.  I don't think we should use that to give a bonus
>> regardless of who the signer is.  All gmail messages are signed, almost
>> everyting from office365.  Yes, I could do a univieral bonus then reduce
>> gmail and onmicroosft.com, but that doesn't get 365 users with their own
>> signatures and all of the millions of other domains out there.
>>
>> It was one thing when DKIM signing was a new concept and only legit
>> businesses signed messages.  Now that most senders are signing, giving  a
>> bonus would let an awful lot of spam slip through under the rejection
>> scoring threshold.
>>
>> >reduce the score for certain domains by blackListedDomains, SenderBase
>> or anywhere else - if needed
>> Senderbase won't work for those using AWS as an example - too many
>> spammers use them, so adding to senderbase can't be negated using
>> blacklist/bombs, etc because I obviously don't know all of the bad senders
>> using AWS.
>>
>> I could reduce the score based on a BombRe match on squaremktg, but then
>> I'm reducing when I haven't validated the signature.  It would probably
>> work for this specific example, but it would be generally helpful to be
>> able to reduce the score on a message based solely on the signature when
>> I'm sure they're actually the sender   Dare I say that I'm in love with
>> DKIM?
>>
>> Would it be life changing like DoDKIMWLAddresses?  No absolutely not, but
>> if it's not a major task to add the functionality, I think there would be
>> wide appeal.
>>
>> I >>almost<< want to suggest that the dkimBonusValancePB feature be
>> removed altogether.  I can't think of a scenario where you'd want to give a
>> bonus universally just because a message has a valid signature from
>> anyone.  Same thing for the SPF pass bonus and it's default of -10!!!  I'm
>> sure there are people using one or both, I just can't think of a
>> scenario in which it'

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[K Post]

The bombHeaderRe with the DEFINE or list should be sufficient.  I'm still
worried about fake/invalid DKIM still getting the bonus score, but this
will have to do.  Thanks.

On Mon, Nov 8, 2021 at 12:01 PM Thomas Eckardt 
wrote:

> I told you to score such domains elsewhere - just do it and the result is
> the same like you wanted.
>
> for example:
>
> bombHeaderRe:
>
> \nDKIM-Signature:(?:[ \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ \t]*([di]=\@?(
> The_Wanted_IDENTITY))\;=>the_wanted_negative_score
>
> currently the (?(DEFINE)...) is not working with assp (is destroyed if
> a-d-n-o-r is not set for the file) - but the next version will do it  -
> and you can use:
>
> (?(DEFINE)(?the_wanted_identity|ident2|ident3|..))\nDKIM-Signature:(?:[
> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
> - e.g. -10
> (?(DEFINE)(?the_wanted_identity|ident5|ident6|..))\nDKIM-Signature:(?:[
> \t]*[^= \;]+=[^= \;]+\;(?:\r\n)?)+?[ 
> \t]*([di]=\@?(?))\;=>the_wanted_negative_score
> - eg -20
> ...
>
> CLOSED for me
>
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:05.11.2021 20:03
> Betreff:Re: [Assp-test] Another Concept Question:
> DKIMBousScoreList
> --
>
>
>
> Having the dkimBonusScoreList would be like applying
> dkimBonusValancePB but ONLY for those that DKIM validate AND are on the
> scorelist.  Here's why I think that would be helpful and what you proposed
> could be problematic.  Essentially: I'm thinking: "look, this organization
> usually sends good stuff, but not always.  They might also have people
> sending non-dkim signed messages through a myriad of channels.  Deal with
> them separately, but if we KNOW it's from them because of their DKIM
> signature, help that message get through with the idea that it'll be
> stored in okmail unless whitelisted through something other than dkim."
>
> > there is already dkimOkValencePB - increase it
> But a high percentage of all messages that are received, spam and not,
> have valid signatures.  I don't think we should use that to give a bonus
> regardless of who the signer is.  All gmail messages are signed, almost
> everyting from office365.  Yes, I could do a univieral bonus then reduce
> gmail and onmicroosft.com, but that doesn't get 365 users with their own
> signatures and all of the millions of other domains out there.
>
> It was one thing when DKIM signing was a new concept and only legit
> businesses signed messages.  Now that most senders are signing, giving  a
> bonus would let an awful lot of spam slip through under the rejection
> scoring threshold.
>
> >reduce the score for certain domains by blackListedDomains, SenderBase or
> anywhere else - if needed
> Senderbase won't work for those using AWS as an example - too many
> spammers use them, so adding to senderbase can't be negated using
> blacklist/bombs, etc because I obviously don't know all of the bad senders
> using AWS.
>
> I could reduce the score based on a BombRe match on squaremktg, but then
> I'm reducing when I haven't validated the signature.  It would probably
> work for this specific example, but it would be generally helpful to be
> able to reduce the score on a message based solely on the signature when
> I'm sure they're actually the sender   Dare I say that I'm in love with
> DKIM?
>
> Would it be life changing like DoDKIMWLAddresses?  No absolutely not, but
> if it's not a major task to add the functionality, I think there would be
> wide appeal.
>
> I >>almost<< want to suggest that the dkimBonusValancePB feature be
> removed altogether.  I can't think of a scenario where you'd want to give a
> bonus universally just because a message has a valid signature from
> anyone.  Same thing for the SPF pass bonus and it's default of -10!!!  I'm
> sure there are people using one or both, I just can't think of a
> scenario in which it's a good idea.
>
>
>
>
> On Fri, Nov 5, 2021 at 10:37 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> Another useless post about concepts without reading the manual.
>
> >dkimBonusValancePB
>
> there is already dkimOkValencePB - increase it
>
> and
>
> reduce the score for certain domains by blackListedDomains, SenderBase or
> anywhere else - if needed
>
> Thomas
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:04.11.2021 22:38
> Betre

Re: [Assp-test] RegEx Backreferences - the basics

[K Post]

Now you've taken up the entirety of any free time I would have had this
coming weekend so I fully dissect your (I'm sure vastly) improved regex!
Can't wait to learn from it, especially how you're using the negative
lookahead.

Already I'm learning from it, I didn't know you could do defines at all,
let alone in an ASSP config file.  That makes it S much more readable
and easier to write.

My questions at this point are more specific to email header matching and
ASSP.

1) You never have \r  as an option as far as I can tell.  In my testing of
my original regex, I found a sample email that seemingly wouldn't match my
expression unless the regex looked for \r?\n as  line ending.  If I just
had \n, no match.  It was only one email, but still.  Does that sound
possible? Might your sample need to have the optional \r added?

2) wouldn't what you have after the tld .+?\n make your expression match
"fake", instead of "real" in
FROM: "fakesen...@some.fake.com" 
My regex required the TLD immediately be followed by an optional >,
optional \r, and a \n  so that we can be sure that it's at the end of the
line.  I feel like editing

(?(DEFINE)(?[a-z]{2,6}))

to

(?(DEFINE)(?[a-z]{2,6}\>?\r?\n))

would give more accuracy.  Necessary? Is there a scenario with the to or
from wouldn't end with that?

3) Are you saying that the next version supporting "line continuation" is
the equivalent of the \s or DOTALL functionality or do you just mean that
we can continue a regex on the next line for readibility?

3a) if by line continuation, you mean dotall,  that means that your .+?
would also match newlines right??  And also match *blank* newlines, which
would be invalid, but still... We wouldn't want to match
to: whate...@domain.com
another: line
 <--- blank line, just a \n
from: somet...@whatever.com

3b) If line continuation means allowing the config file to continue onto
the next line, I don't think your regex accounts for other header lines
between the to and from (or from and to)?   Add in a (.+?\n)*? part just
before the negative lookahead to match:

to: whate...@domain.com
someother: header
more: headers
from: e...@else.com

or is there something else going on that accounts for those lines in
between?



I clearly have a lot to learn about the negative lookahead, and I'm eager
to do so on my own.   I'm going to do this on my own, but with what I
understand so far, am I correct to say that

\n(?!\k)(?:to:(?)\k\@(?:(?&
HOSTorNAME)\.)+(?)|from:(?)\@(?:(?)\.)
*?\k\.(?))

is \n last character is newline
then it's acting like if then else logic.  Simplified with comments

\n [last character is newline]
if [TAG] is not next  (?!\k)
 then

look for a line starting with To that has the 2nd level domain name in the
user part

(?:to:(?)\k\@(?:(?)\.)+(?)

| else

look for a line starting with from that was the 2nd level domain name in
the hostname part
from:(?)\@(?:(?)\.)*?\k\.(?)

)  // end if


Tag will either be To or From depending on which it was set to earlier.
Yes?  If so, this is such powerful logic, so simple, yet clearly only comes
with quite a lot of regex experience!!



And thanks for the upcoming bug fix too.

Have a wonderful weekend.  Thanks for again burning brainpower this week
for me and for sharing all of your thoughts.
Ken


On Fri, Nov 5, 2021 at 9:09 AM Thomas Eckardt 
wrote:

> >*It seems like using <<< >>> to turn of regex optimization might break
> the 2nd parameter from being recognized.*
>
> That's true. It is fixed in the next release.
>
>
> regex:
>
> something like this is better to read after some time, it is much less
> greedy, faster and selfexplaining:
>
>
> (?(DEFINE)(?[a-z]{2,6}))(?(DEFINE)(?[a-z\d\-]+))(?(DEFINE)(?[^\n]*?))
>
> (?:^|\n)(?:(?to):(?)(?(?))\@(?:(?)\.)+(?)|(?from):(?)\@(?:(?)\.)*?(?(?))\.(?))
>
> .+?\n(?!\k)(?:to:(?)\k\@(?:(?)\.)+(?)|from:(?)\@(?:(?)\.)*?\k\.(?))
>
>
> (all in one line)
> the next release supports line continuation in files
>
> The regex is as simple as it can be, except one small trick - the negative
> lookahead (?!\k). So, yes - looking around the string without moving
> the position around makes some things more easy.
>
>
> This thread should be stopped here. This is a test list for development
> versions - it is not a blog and it is not a place to learn perl regular
> expressions.
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:05.11.2021 00:20
> Betreff:Re: [Assp-test] RegEx Backreferences - the basics
> --
>
>
>
> First of all, to say that the problem is sitting in front of the monitor
> is *insulting my keyboard*.  He sits between me and the monitor and has
> done nothi

Re: [Assp-test] Another Concept Question: DKIMBousScoreList

[K Post]

Having the dkimBonusScoreList would be like applying dkimBonusValancePB but
ONLY for those that DKIM validate AND are on the scorelist.  Here's why I
think that would be helpful and what you proposed could be
problematic.  Essentially:
I'm thinking: "look, this organization usually sends good stuff, but not
always.  They might also have people sending non-dkim signed messages
through a myriad of channels.  Deal with them separately, but if we KNOW
it's from them because of their DKIM signature, help that message get
through with the idea that it'll be stored in okmail unless whitelisted
through something other than dkim."

> there is already dkimOkValencePB - increase it
But a high percentage of all messages that are received, *spam and not*,
have valid signatures.  I don't think we should use that to give a bonus
regardless of who the signer is.  All gmail messages are signed, almost
everyting from office365.  Yes, I could do a univieral bonus then reduce
gmail and onmicroosft.com, but that doesn't get 365 users with their own
signatures and all of the millions of other domains out there.

It was one thing when DKIM signing was a new concept and only legit
businesses signed messages.  Now that most senders are signing, giving  a
bonus would let an awful lot of spam slip through under the rejection
scoring threshold.

>reduce the score for certain domains by blackListedDomains, SenderBase or
anywhere else - if needed
Senderbase won't work for those using AWS as an example - too many
spammers use them, so adding to senderbase can't be negated using
blacklist/bombs, etc because I obviously don't know all of the bad senders
using AWS.

I could reduce the score based on a BombRe match on squaremktg, but then
I'm reducing when I haven't validated the signature.  It would probably
work for this specific example, but it would be generally helpful to be
able to reduce the score on a message* based solely on the signature when
I'm sure they're actually the sender*   Dare I say that I'm in love with
DKIM?

Would it be life changing like DoDKIMWLAddresses?  No absolutely not, but
if it's not a major task to add the functionality, I think there would be
wide appeal.

I >>almost<< want to suggest that the dkimBonusValancePB feature be removed
altogether.  I can't think of a scenario where you'd want to give a bonus
universally just because a message has a valid signature from anyone.  Same
thing for the SPF pass bonus and it's default of -10!!!  I'm sure there are
people using one or both, I just can't think of a scenario in which it's a
good idea.




On Fri, Nov 5, 2021 at 10:37 AM Thomas Eckardt 
wrote:

> Another useless post about concepts without reading the manual.
>
> >dkimBonusValancePB
>
> there is already dkimOkValencePB - increase it
>
> and
>
> reduce the score for certain domains by blackListedDomains, SenderBase or
> anywhere else - if needed
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:04.11.2021 22:38
> Betreff:[Assp-test] Another Concept Question: DKIMBousScoreList
> --
>
>
>
>
> SUMMARY: Would there be benefit (that wouldn't be terrible to code) in
> adding the ability for use to assign a score to emails that match a list of
> DKIM signature identities?
>
>
> The DKIMWLAddress and DKIMNPAddress functionality has been an absolute
> game changer here.  Thank you so much for implementing that (it was my
> idea, but we all know that I could never code such a thing).
>
> I've combined that functionality with closely monitored SenderBase lists
> to dramatically improve ASSP's accuracy.
>
> One place where Senderbase shines is it's scoring ability for bulk
> senders.  For example, I can give anything that Senderbase says is coming
> from constant contact's network a -10 score, by adding it into
> whiteSenderBase like
> ^constantcontact\.com$=>-10
> I don't want to blindly let through constant contact signed messages, but
> if it's coming from their network, make it a little easier for messages to
> pass through. That's worked well for a long long time.
>
>
> Recently, I'm seeing several bulk senders having legitimate messages DKIM
> signed by the bulk sender them, but being sent through Amazon AWS (
> *amazonses.com* <http://amazonses.com/>) and is classified by senderbase
> as being Amazon / *amazonses.com* <http://amazonses.com/>.  There's a lot
> of volume coming in from *amazonses.com* <http://amazonses.com/>, but
> unfortunately, it's a mix of perfectly legitimate messages and others that
> are pure garbage.  So that takes Senderbase off the table.  Coming from
> amazonses shouldn't impact the score either way.  And I ca

Re: [Assp-test] RegEx Backreferences - the basics

[K Post]

ot; in between the first part of the regex which looks for to
then from and the second part which looks for from then to.   *Would it be
more efficient for ASSP to have 2 separate lines, one for to first the
other for from first?*

Here's my thinking and explanation of my understanding of the regex that I
wrote. I am VERY interested in corrections and suggestions for improvement,
especially relating to efficiency (and obviously flawed logic and/or cases
where what I've done would or wouldn't match as I'm thinking).  Guidance
here won't only help me perfect this specific regex for ASSP use, but will
hopefully help others looking for other more complex than typical regex
help with ASSP.  I'll definitely be limiting the to domains to those that
we use here to speed this up a bit, but I kept it more generic here.

I also tried to see a way where lookaheads might help, but I'm not quite
there yet  Would they be helpful here?

Starting from the beginning:

(?:^|\r?\n)
start with either the start of the string or a \r?\n   - sometimes there's
a \r but always a \n Is \r?\n recommended?  Is there a better way?

Then we're going to do 2 big OR's,  first looking for to then from, then
from then to.
(?: starts this big or, with the ?: indicating that it's a non-capturing
group

The TO then From part is this:
to:(?:.*?[\s\<])*?(?[a-z\d\-]+)\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n(.+\r?\n)*?from:.*?\@(?:[a-z\d\-]+\.)*?\g{TOFirstMatch}\.[a-z]{2,6}\>?\r?\n

broken out

to:  Find to:  immediately after the previously found newline or start of
string)

(?:.*?[\s\<])*?
non-capturing match for any characters repeated as long as they end with a
space or <

now we should be at the point where the username starts

(?[a-z\d\-]+)\@
get a named match called TOFirstMatch for any a-z number - combination that
ends in the now escaped @

(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n
then just make sure that what follows the @ is a-z decimal and dahes, each
part ending in a . with a 2-6 letter TLD ending the hostname followed by an
optional > and then \n or \r to end the line



(?:.+\r?\n)*?
then ignore future lines which aren't blank until we a line starting with
from:

from:.*?
line stars with from: followed by any characters

\@(?:[a-z\d\-]+\.)*?
find @valid.sub. part of from address

\g{TOFirstMatch}
use the \g{} syntax to match the named backreference

\.[a-z]{2,6}\>?\r?\n)
immediately followed by .tld 2-6 characters in length, an optional > and a
\n or \r

|
then an OR

and we do the whole thing again but with From First
from:.*?\@(?:[a-z\d\-]+\.)*?(?[a-z\d\-]+)\.[a-z]{2,6}\>?\r?\n(?:.+\r?\n)*?to:(?:.*?[\s\<])*?\g{FROMFirstMatch}\@(?:[a-z\d\-]+\.)+[a-z]{2,6}\>?\r?\n)

from:.*?
from: followed by anything until we hit

\@(?:[a-z\d\-]+\.)*?
and @ sign followed by any number of hostname followed by .

(?[a-z\d\-]+)
find the second level domain name and call is FROMFirstMatch

\.[a-z]{2,6}\>?\r?\n
followed by a .tld of 2 to 6 characters, an optional closing > and a \n or
\r

(?:.+\r?\n)*?
move past non blank lines until we hit

to:(?:.*?[\s\<])*?
to: optionally followed by whatever characters ending in space or <


\g{FROMFirstMatch}\@
now look for the second level domain match from the from: line immediately
followed by an @ sign

(?:[a-z\d\-]+\.)+
then hostnames separated by dots, at least 1

[a-z]{2,6}\>?\r?\n)
followed by a 2-6 character tld, an optional > and a \n or \r?

)
closing out the or between the MatchToFirst and FROMFirstMatch sections.


Whew.
:


On Thu, Nov 4, 2021 at 4:53 AM Thomas Eckardt 
wrote:

> forgot to say:
>
> if assp requires to capture the match for a regex, the code would be for
> example
>
> $string =~ /($testReRE)/
> $match = $1;
>
> so - at runtime the regex is
>
> ((?^u:(?is:(?:^|\n\r).*(searchstring).*@.*\1.*)))
>
> IMHO you need to use named capture groups or \g or (?|
>
> Thomas
>
>
>
> Von:"Thomas Eckardt" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:04.11.2021 09:22
> Betreff:Re: [Assp-test] RegEx Backreferences - the basics
> --
>
>
>
> to make backreferences working, regex optimization must be switched off
> for the complete regex -> tested -> worked
>
> >I've seen posts here indicating that backreferencing matches is possible
> with an unoptimized expression.
>
> so - the problem is sitting in front of the monitor :):)
>
> m/(?is:(?:^|\n\r).*(?:searchstring)*.*@.*\1* <.*@.*%5C1> <-- HERE .*)/
>
> optimized - default is : 'no extra group capturing is allowed'
>
> >I've got to be missing something incredibly obvious.
>
> assp-do-not-optimize-regex
>
> >  (?:^|\n\r).*(searchstring).*@.*\1.*
>
> assp makes it:
>
> (?is:(?:^|\n\r).*(searchstring).*@.*\1.*)
>
>

[Assp-test] Another Concept Question: DKIMBousScoreList

[K Post]

SUMMARY: Would there be benefit (that wouldn't be terrible to code) in
adding the ability for use to assign a score to emails that match a list of
DKIM signature identities?


The DKIMWLAddress and DKIMNPAddress functionality has been an absolute game
changer here.  Thank you so much for implementing that (it was my idea, but
we all know that I could never code such a thing).

I've combined that functionality with closely monitored SenderBase lists to
dramatically improve ASSP's accuracy.

One place where Senderbase shines is it's scoring ability for bulk
senders.  For example, I can give anything that Senderbase says is coming
from constant contact's network a -10 score, by adding it into
whiteSenderBase like
^constantcontact\.com$=>-10
I don't want to blindly let through constant contact signed messages, but
if it's coming from their network, make it a little easier for messages to
pass through. That's worked well for a long long time.


Recently, I'm seeing several bulk senders having legitimate messages DKIM
signed by the bulk sender them, but being sent through Amazon AWS (
amazonses.com) and is classified by senderbase as being Amazon /
amazonses.com.  There's a lot of volume coming in from amazonses.com, but
unfortunately, it's a mix of perfectly legitimate messages and others that
are pure garbage.  So that takes Senderbase off the table.  Coming from
amazonses shouldn't impact the score either way.  And I can't DKIMWLAddress
the signature, then bad stuff would absolutely get through.

An example is Square, the credit card processor and software company.  They
send mail, DKIM signed @squaremktg.com on behalf of clients.  Most mail
from square is good, but sometimes it gets spammy, just like we see with
mail from other bulk senders.  Real world, I paid for a car wash using
their mobile payment platform, I received the receipt and later got
an email with a promotion from the car wash.  All good.  The provider's
signature was in DKIMWLAddresses.  Today, I received an advertisement from
them for what is apparently a "gentleman's club" next door, offering a
complimentary car wash (I took that literally) for visiting the
establishment.  The language in that email would have absolutely had it
rejected if it hadn't been on DKIMWLAddresses.  Worse, it wound up in the
not-spam corpus.


So, I'd like for certain DKIM signatures to be able to SCORE.  DKIM scoring
would help it get through (or make it harder depending on the score)
without automatically passing it and adding it to the corpus like
DKIMWLAddresses does.   That would let me give the message a negative score
based on the DKIM but still let Bayesian/HMM and other features stay in
play to score the message further.

Conceptually, I could see this working similarly to senderbase.  There
would be a default valance like

dkimBonusValancePB

set to a default of -25

Then we'd have a list, maybe called DKIMBousScoreList.  Like
DKIMWLAddresses, it would match the end of the validated DKIM identity, but
also accepts a score override:

(@|.)squaremktg.com<--- gets the default of -25

(@|.)someUsuallyOKsigner.com=>-12<-- gets -12 for a score

(@|.)prettygood.com=>5<--- gets 1/5 of the default
-25   -25/5 = -5

(@|.)UsuallyBad.com=>-5  <-- this isn't a bonus, a
negative default divided by a negative is a positive.  it will be -25/-5 or
adding 5 to the score



>From a management standpoint, it would certainly be easier to "just" be
able to assign an optional 2nd parameter to DKIMWLAddresses that would
score instead of whitelisting, but I feel like that could be too big of a
coding project.

I tried to come up with a way to accomplish the same thing based on DKIM
signature, but came up very short.  I know I could ignore DKIM and just
score based on the from line, but I really appreciate the certainty that
DKIM gives that the message is really from that organization.

What do you think?  Would a  DKIMBousScoreList feature have universal
appeal?
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] RegEx Backreferences - the basics

[K Post]

I've got nothing in my TestRe file except for a single line:

~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~


The idea is to log any time there's a line that includes "searchstring" on
the right and left of an @.  This is just a very rudimentary test because
backreferences seem to error for me.  I would expect this to match

searchstring@searchstring
something else seachstring more @ whatever searchstring bla

If "searchstring" is to the right and left of an @ sign, it should match.
Regex101.com seems to confirm that this works.  Like I said, super basic.

However, if I enter ~<<<(?:^|\n\r).*(searchstring).*@.*\1.*>>>~ as the only
line in TestRe file, I get a warning in the log:

- Reference to nonexistent group in regex; marked by <-- HERE in
m/(?is:(?:^|\n\r).*(?:searchstring).*@.*\1 <-- HERE .*)/
- try using unoptimized regex

To my understanding, the <<< >>> surround should turn of regex optimization
for that line, which enables backreferencing (\1) to work and the ~ is
required because there's an or in there.   Shouldn't the \1 reference
(searchstring) ?  I don't understand why assp thinks that \1 is a reference
to a non-existent group.

I also tried removing the <<< >>> and adding assp-do-not-optimize to the
top of the TestRe file.  No difference.No matter how simple I make the
regex, even (.*)@\1,  it still complains about the invalid backreference.


I've got to be missing something incredibly obvious.  I've read through the
regex doc in docs, but that doesn't talk about backreferencing in ASSP and
I can't find anything in the GUI that makes mention. I've seen posts here
indicating that backreferencing matches is possible with an unoptimized
expression.

A shove in the right direction would be greatly appreciated.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Manual clarification suggestions for RedList

[K Post]

Under the spam control section, I suggest:

1) Moving the "Redlist serves 2 purposes..." bit from the GUI up to its own
explanation section instead of having it in the RedRE section.  That useful
description applies to more than just RedRE

2) Adding some information to the redlist description.  My changes in GREEN

By default, Redlisted messages will not be stored in the
SPAM/NOTSPAM-collection (see DoNotCollectRedList and DoNotCollectRedRe).

3) Change RedRe description to:
If an email matches this Perl regular expression it will be considered
redlisted.
redRe detects tags to process a mail like the recipient were redlisted -
nothing else (no redlist addition/removal).  By default, RedRe matching
messages will not contribute to the corpus (see DoNotCollectRedRe).

As all fields marked by * this field accepts a list separated by | or a
specified file 'file:files/redre.txt'.

4) Consider adding reference to DoNotCollectRedList to the list GUI screen?

Yes, if someone changes the default of DoNotCollectRedList, they need to
know the ramifications of that choice.  However, if 2 years after making
that change, they don't remember and then read what the redlist does, where
it says redlisted mail won't be saved without mention of
DoNotCollectRedList, they could be really confused.  These additions to the
GUI will make it more clear.


I hope this is helpful.
ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

When looking at the "Use this HTML Parser" section on the GUI, I found this
line:

it is recommended to set MaxBytes to 5 (be carefull on heavy load
systems - spam bomb regular expressions will take longer using 5!).\

I'm going to change my settings and see how bad the rebuild time is.  I've
got enough processing power and RAM now, but the disks aren't SSD.  Just a
4 disk Raid 1+0 traditional HDD setup.  We'll see...

Since HTMl email accounts for a big percentage of all mail,  might it be a
good idea to update/expand the guidance in the MaxBytes section of the
GUI?



On Fri, Oct 29, 2021 at 8:40 PM K Post  wrote:

> Summary:
> *Should/could any consideration be given to having ASSP scan the entire
> message at the time it is received for Bombs (only), while still using
> MaxBytes for Bayesian/HMM?*
>
> We've been having some cleverly crafted messages slipping through all
> filters that would be easy to catch with Bombs if only the catchable
> content came before MaxBytes.  These messages are 20kb+, They have a scam
> phone number at the very end of the larger than MaxBytes messages.  I
> want/need to use bombs to catch the scam phone numbers.
>
> With MaxBytes set to 3000, which is useful for faster RebuildSpamDB, these
> BombDataRE matches just aren't being caught.  If I increase MaxBytes, my
> BombDataRE catches them, but then rebuildspamdb is (probably? see below)
> longer than it needs to be.
>
> So, is there any value in considering a* MaxBytesAdditionalForBombs *variable
> which would be *added to MaxBytes *and only used when scanning for bombs
> as messages arrive?   Would that kill performance??  Other downsides?
>
> We could still only look at MaxBytes for Bayesian/HMM since it's only
> MaxBytes used when building those databases.
>
> What do you think?
>
> And while we're talking MaxBytes:
> I've asked this before, is the guidance for 3kb for MaxBytes once there's
> a mature corpus still a valid recommendation?  With unlimited horsepower
> and ram, sure, why not, do 30kb or 100kb.  That's not my reality, so I want
> to see where to best allocate resources. If 3kb is still the guidance, even
> though the spam files I'm seeing have a median size around 20kb, so be it.
> I feel like when that guidance was written, html wasn't used as
> prolifically in spam.  The median size of notspam in my corpus is about
> 40kb.  That's determined unscientifically by sorting by size and scrolling
> to approximately half way down.
>
> Thanks.  Have a good weekend.
> Ken
>
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[K Post]

Outlook itself is a typo.  I think it's supposed to be called LookOut!

-Using the outlook function "send email as attachment" or "forward email as
attachment" has NEVER worked - because outlook destroys the original MIME
header. But the original header is particularly important for analysis
systems!

But isn't the message that's attached as a .msg file fully saved, intact,
in the appropriate corrected corpus folder?   I'm using subject names for
file names and with 21302 the files names no longer are the subject of the
.msg, but with 21293 it was.  The content of the saved file is still
correct though.  So why can't ASSP use that for the analysis?

The analysis on those saved files go off without an error.  It's only the
analysis that's triggered at the time the correction report is emailed
that's misfiring. * If outlook is destroying the mime header of the
original .msg file, could whatever ASSP is doing to that be used then the
analyze report runs?   *Should/could the code be changed to instead run the
newly saved report file (or it's contents that may already be in a
variable) through analyze?

I've done the best I can to interpret what the ASSP code's doing. To me, it
doesn't look like the ConfigureAnalyze that runs is done on the saved file,
rather it's done looking at least at the headers of the report email itself
(vs the reported email).  ASSP seems to yell when it hits a tab in the
header of the report email, NOT the header of the .msg file that's
attached, but the header of the report itself.  If I change the TO line of
the report email to be short enough (demonstrated in the previous message
in this thread), assp doesn't complain about the to line anymore, instead
it complains when it hits the next tab.  In the example, that was when it
hit one of the attachment boundaries.

With Outlook doing line continuations with a tab, shouldn't ASSP account
for that even with report emails?  It seems to handle it just fine
everywhere else.  I read through parts of RFC822 and it seems to suggest
that a tab is a valid line continuation method.  ASSP's headerUnwrap seems
to account for this just fine, but maybe that's not being used with the
analyze?


We're on the same page about Outlook being an evil beast.  Unfortunately,
it is by far the most common mail client in use, so I've got to deal with
it.  What I'm trying to accomplish is having a reliable and easy way for
all staff to report.  Forwarding as an attachment from Outlook does seem to
work, just not once the additional analysis report is run.

- the mail is to be exported (.msg or .eml) and then (possibly zip) sent to
the system

Zipping works 100%.  Saving message as msg then attaching doesn't seem to
be any different from forward as attachment

- or to attach the mail to a new mail via drag and drop (this may not work
in every case and every outlook versions)

No difference here, at least with Outlook 2019

- or to use a plugin which provides any of the both options (
https://sourceforge.net/projects/assp/files/assp_mail_client_plugins/

Unfortunately I don't have access to all end user's machines, especially
with the pandemic having so many users with email on personal devices.  I
wish we could provide charity owned laptops for home use, but I can't even
get budget approval for a new desk chair.

I've got to keep it easy easy easy for end users to report.  No 2 or 3 step
processes, no plugins required.  Just forward as attachment.  That works
(even with 21302's wrong analyze warnings and file name mangling).  It
would just be nice to get propper analyze reports to make it easier for me
to stay on top of what's being reported.


Another thing that doesn't seem to work is the handling of multiple .msg
attachments in one report. That happens if a user selects several messages
and does a forward as attachment.  That's never worked, but if when you
have the time and energy to review what's going on with the above, maybe
you could also consider fixing (adding?) this functionality?

...as always...  Thank you
Ken




On Sun, Oct 31, 2021 at 5:50 AM Thomas Eckardt 
wrote:

> It's always been like this - outlook forwards WRONG. To send a mail to an
> SMTP-based analysis system (this applies to all, not just ASSP) for
> analysis purposes:
>
> - the mail is to be exported (.msg or .eml) and then (possibly zip) sent
> to the system
> - or to attach the mail to a new mail via drag and drop (this may not work
> in every case and every outlook versions)
> - or to use a plugin which provides any of the both options (
> https://sourceforge.net/projects/assp/files/assp_mail_client_plugins/)
>
> Using the outlook function "send email as attachment" or "forward email as
> attachment" has NEVER worked - because outlook destroys the original MIME
> header. But the original header is particularly important for analysis
> systems!
>
>
> Thomas
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may 

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[K Post]

I've spent the better part of 2 hours trying to figure out what's going on.

First, please note that for me, both 21293 and 21302 appear to record the
.msg file in errors-spam/errors-notspam correctly.  The .msg is getting
extracted from the report properly.  If I do an analyze from the GUI on
these files, it works perfectly.

ConfigAnalyze seems to be analyzing the wrong content, differently in 21302
than 21293.  I believe that with 21302, when ConfigAnalyze is called when
processing a report that's sent with a .msg attachment, it might be looking
at the entirety of the email report instead of just what's in the .msg
file.  I don't know if that's as intended.

Using the (very helpful) ability of 21302 to write reports to debug, here's
what I've discovered:

I see in ConfigAnalyze where it's spitting out the mime headers starting in
the middle error.  I don't really know why it's adding the CR LF ,
but suspect it has to do with what I actually did find, *TABS saved in the
file in the debug folder.   *I don't know if Outlook is doing the tab
wrapping or if it's ASSP, but either way, ASSP isn't happy about them.

My reporting address is pretty long, something like
report-not-s...@assp.detroit.ourcharity.org
When I forward a report as a .msg attachment, I've just been using that
address from remembered addresses in Outlook.  There's no address book
entry in Outlook, so Outlook behaves like Outlook behaves, adding the email
address in quotes to the to line.  The message that outlook sends is saved
to the debug folder starting like this:

From: Ken Post 
To: "'report-not-s...@assp.detroit.ourcharity.org'"   <-- that's a double
quote, then single quote, then the address, then a single quote, then a
double.
 <-- *this line starts
with TAB, not spaces*
Subject: FW: test of report message


When it's received like this, ASSP gives me

Found a possible MIME header start in the middle of the mail - the analyze
may be wrong
[CR][LF] •
Subject: FW: test of report message[CR][LF]



To test this further, I sent the same message from outlook, but this used a
very short name in the To field. The message shows in the debug folder like:

From: Ken Post 
To: RNS<-- no quotes
around the name, to line on single line now
Subject: FW: test of report message


The headers continue with stuff about message id, content language, etc,
until it gets to a line like:

Content-Type: multipart/mixed;

 
boundary="_004_06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a_"

MIME-Version: 1.0


That second line starts with a tab again.

ASSP this time *doesn't complain about the subject*, instead, I get

found a possible MIME header start in the middle of the mail - the analyze
may be wrong
boundary="_
004_06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a_"[CR][LF] •
MIME-Version: 1.0[CR][LF]



With 21302, I tried saving the message in Outlook, zipping it, and
attaching that.  The report in the debug still has the tab indented To
line, but ASSP does not complain, analyze accurately reports on the
extracted message.   I still don't know if your note in the changelog "
Notice: always compress (e.g. zip) reported emails before they are sent to
assp!" applies to .msg attachments too.  Doing do, makes this work, but
requiring users to zip would spell the end of my already reluctant users
from ever reporting again :(


I hope information is helpful and this gives you enough information to
either change ASSP further or point me in the direction of where I'm going
wrong.





On Sat, Oct 30, 2021 at 11:28 AM Thomas Eckardt 
wrote:

> >found a possible MIME header start in the middle of the mail - the
> analyze may be wrong
> [CR][LF] •
> Subject: FW: test of report message[CR][LF]
>
>
>
> Is there any one else on this mailing list, who expects an email received
> by assp to start with an empty line followed by  (html code) or that
> the first header line of such a mail is the subject header line ?
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:30.10.2021 09:15
> Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
> build 21302
> --
>
>
>
> sorry, I sent the last message before proofing or finishing.  Grr, gmail.
> I'll wait to hear from you.  I have more thoughts on NWLI and other
> sections.
>
> On Fri, Oct 29, 2021 at 6:00 PM K Post <*nntp.p...@gmail.com*
> > wrote:
> This is simply terrific.  You keep making ASAP better! The rebuild config
> efficiency improvements are especially appreciated.  Thanks so much as
> usual for spending what must have been a long time thinking about and
> making all of these changes.
>
> SURPRISE, I have questions and comments:
>

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[K Post]

>> Is there any one else on this mailing list, who expects an email
received by assp to start with an empty line followed by  (html code)
or that the first header line of such a mail is the subject header line ?
That doesn't happen when I send the exact same report with 21293.  It does
with 21302 though.  I will attempt to figure this out by looking through
the code, but I haven't changed anything on my end, it's ASSP that has
changed.

On Sat, Oct 30, 2021 at 11:28 AM Thomas Eckardt 
wrote:

> >found a possible MIME header start in the middle of the mail - the
> analyze may be wrong
> [CR][LF] •
> Subject: FW: test of report message[CR][LF]
>
>
>
> Is there any one else on this mailing list, who expects an email received
> by assp to start with an empty line followed by  (html code) or that
> the first header line of such a mail is the subject header line ?
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:30.10.2021 09:15
> Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
> build 21302
> --
>
>
>
> sorry, I sent the last message before proofing or finishing.  Grr, gmail.
> I'll wait to hear from you.  I have more thoughts on NWLI and other
> sections.
>
> On Fri, Oct 29, 2021 at 6:00 PM K Post <*nntp.p...@gmail.com*
> > wrote:
> This is simply terrific.  You keep making ASAP better! The rebuild config
> efficiency improvements are especially appreciated.  Thanks so much as
> usual for spending what must have been a long time thinking about and
> making all of these changes.
>
> SURPRISE, I have questions and comments:
>
> *Fix to emailing report with attached .msg report not working?*
> *(if email reports need to be zipped, ignore this)*
>
> I just tested, sending a zip of a .msg has the analyze report works
> correctly.  Sending just the .msg attachment with 21293 give the previous
> errors.   *Sending just the .msg with this new 21302 unfortunately is
> worse and isn't working*.  I'm now getting:
> found a possible MIME header start in the middle of the mail - the analyze
> may be wrong
> [CR][LF] •
> Subject: FW: test of report message[CR][LF]
> and even less info is shown in the emailed analyze report than before
>
> I had ReportLog set to debug.  The .eml file that goes to debug is
> attached to this reply.
>
>
>
> *Clarification on the need now(?) to compress Emailed reports from Outlook*
> You wrote: "Notice: always compress (e.g. zip) reported emails before they
> are sent to assp!"   The GUI says " It is also possible to send MS-outlook
> '.msg' files (possibly zipped)."  Is it now required?
>
>  I've always done "Forward as Attachment" in Outlook to report which
> attaches the current message as a .msg formatted file (I previously
> incorrectly wrote .eml) and seems to work (except for the analyze big that
> you're attempting to squash). The vast majority of our staff do the same,
> but requiring them to first save the .msg, then zip, then attach to
> report might be asking too much.
>
> While attached Outlook .msg files are binary, I don't >>think<< they're
> compressed.  The .msg files always have made it to the corpus saved as
> plain text files which seems right.  It was only the analyze report that
> was failing.
>
>
>
>
>
>
> *MaxBytesReports*
>
> The gui talks about MaxBytesReports.
>
> Any mail sent or forwarded by local/authenticated users to this username
> will be interpreted as a spam report. Multiple attachments get truncated to
> MaxBytesReports. Do not put the full address here, just the user part.
>
> For example: asspspam . Use a fake domain like @assp.local or @
> *assp-nospam.org* <http://assp-nospam.org/> when you send the email- so
> the full address would be then asspspam@assp.local.
>
> You can sent multiple mails as attachments and/or zipped file(s). Each
> attached email-file must have the extension defined in "maillogExt". In
> this case only the attachments will be processed. To use this
> multi-attachment-feature an installed Email::MIME module in PERL is needed.
> It is also possible to send MS-outlook '.msg' files (possibly zipped). To
> use this MS-outlook-feature in addition an installed
> Email::Outlook::Message module in PERL is needed.
>
>
> I don't see MaxBytesReports any where else in the code.  Is this supposed
> to be MaxBytes?
>
> Also, GUI correction  " You can sent multiple mails as attachments and/or
> zipped file(s)" should be "send"
>
>
>
> *Finding DKIM matches 

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[K Post]

sorry, I sent the last message before proofing or finishing.  Grr, gmail.
I'll wait to hear from you.  I have more thoughts on NWLI and other
sections.

On Fri, Oct 29, 2021 at 6:00 PM K Post  wrote:

> This is simply terrific.  You keep making ASAP better! The rebuild config
> efficiency improvements are especially appreciated.  Thanks so much as
> usual for spending what must have been a long time thinking about and
> making all of these changes.
>
> SURPRISE, I have questions and comments:
>
>
> *Fix to emailing report with attached .msg report not working?*
>
> *(if email reports need to be zipped, ignore this)*
>
>
> I just tested, sending a zip of a .msg has the analyze report works
> correctly.  Sending just the .msg attachment with 21293 give the previous
> errors.   *Sending just the .msg with this new 21302 unfortunately is
> worse and isn't working*.  I'm now getting:
>
> found a possible MIME header start in the middle of the mail - the analyze
> may be wrong
>
> [CR][LF] •
>
> Subject: FW: test of report message[CR][LF]
>
> and even less info is shown in the emailed analyze report than before
>
> I had ReportLog set to debug.  The .eml file that goes to debug is
> attached to this reply.
>
>
>
>
> *Clarification on the need now(?) to compress Emailed reports from Outlook*
>
> You wrote: "Notice: always compress (e.g. zip) reported emails before they
> are sent to assp!"   The GUI says " It is also possible to send MS-outlook
> '.msg' files (possibly zipped)."  Is it now required?
>
>  I've always done "Forward as Attachment" in Outlook to report which
> attaches the current message as a .msg formatted file (I previously
> incorrectly wrote .eml) and seems to work (except for the analyze big that
> you're attempting to squash). The vast majority of our staff do the same,
> but requiring them to first save the .msg, then zip, then attach to
> report might be asking too much.
>
> While attached Outlook .msg files are binary, I don't >>think<< they're
> compressed.  The .msg files always have made it to the corpus saved as
> plain text files which seems right.  It was only the analyze report that
> was failing.
>
>
>
>
>
>
> *MaxBytesReports*
>
> The gui talks about MaxBytesReports.
>
> Any mail sent or forwarded by local/authenticated users to this username
> will be interpreted as a spam report. Multiple attachments get truncated to
> MaxBytesReports. Do not put the full address here, just the user part.
>
> For example: asspspam . Use a fake domain like @assp.local or @
> assp-nospam.org when you send the email- so the full address would be
> then asspspam@assp.local.
>
> You can sent multiple mails as attachments and/or zipped file(s). Each
> attached email-file must have the extension defined in "maillogExt". In
> this case only the attachments will be processed. To use this
> multi-attachment-feature an installed Email::MIME module in PERL is needed.
> It is also possible to send MS-outlook '.msg' files (possibly zipped). To
> use this MS-outlook-feature in addition an installed
> Email::Outlook::Message module in PERL is needed.
>
>
> I don't see MaxBytesReports any where else in the code.  Is this supposed
> to be MaxBytes?
>
>
> Also, GUI correction  " You can sent multiple mails as attachments and/or
> zipped file(s)" should be "send"
>
>
>
>
> *Finding DKIM matches during rebuild:*
>
>
> AddDKIMHeader needs to be on right?
>
> Since it looks like the code is looking for the X-ASSP-DKIMIdentity line,
> I think you should add a comment for the hidden DoRBWhite parameter and
> others
> * that AddDKIMHeader in the GUI needs to be on for this to work. *
>
>
> Speed -> more configuration choices necessary?
>
> Great point about slowness in rebuild if this is all on and the
> recommendation of potentially only turning it on periodically.
>
>
> Since we're now doing more checks, does it make sense to have additional
> hidden parameters to give more granular control?  I feel like I'll always
> want to check for whitelisted in spam no matter how it matches, but other
> might only want to consider DKIMWLAddress, while others don't want
> DKIMWLAddresses and only matches to the actual whitelist.  How about
> letting DoRBWhite be configured with different values like we have for
> DoNoFrom?
>
> Match:
>
> 0: nothing
>
> 1: whiteRe
>
> 2: npRe
>
> 4: whiteListedDomains
>
> 8: noProcessingDomains
>
> 16: whiteListedIPs
>
> 32: noProcessingIPs
>
> 64: DKIMWLAddresses
>
> 128: DKIMNPAddresses
>
> and have those summed up?  Too compl

[Assp-test] Concept Question: Scan entire message for Bombs, regardless of MaxBytes setting? New MaxBytes recommendation?

[K Post]

Summary:
*Should/could any consideration be given to having ASSP scan the entire
message at the time it is received for Bombs (only), while still using
MaxBytes for Bayesian/HMM?*

We've been having some cleverly crafted messages slipping through all
filters that would be easy to catch with Bombs if only the catchable
content came before MaxBytes.  These messages are 20kb+, They have a scam
phone number at the very end of the larger than MaxBytes messages.  I
want/need to use bombs to catch the scam phone numbers.

With MaxBytes set to 3000, which is useful for faster RebuildSpamDB, these
BombDataRE matches just aren't being caught.  If I increase MaxBytes, my
BombDataRE catches them, but then rebuildspamdb is (probably? see below)
longer than it needs to be.

So, is there any value in considering a* MaxBytesAdditionalForBombs *variable
which would be *added to MaxBytes *and only used when scanning for bombs as
messages arrive?   Would that kill performance??  Other downsides?

We could still only look at MaxBytes for Bayesian/HMM since it's only
MaxBytes used when building those databases.

What do you think?

And while we're talking MaxBytes:
I've asked this before, is the guidance for 3kb for MaxBytes once there's a
mature corpus still a valid recommendation?  With unlimited horsepower and
ram, sure, why not, do 30kb or 100kb.  That's not my reality, so I want to
see where to best allocate resources. If 3kb is still the guidance, even
though the spam files I'm seeing have a median size around 20kb, so be it.
I feel like when that guidance was written, html wasn't used as
prolifically in spam.  The median size of notspam in my corpus is about
40kb.  That's determined unscientifically by sorting by size and scrolling
to approximately half way down.

Thanks.  Have a good weekend.
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21302

[K Post]

This is simply terrific.  You keep making ASAP better! The rebuild config
efficiency improvements are especially appreciated.  Thanks so much as
usual for spending what must have been a long time thinking about and
making all of these changes.

SURPRISE, I have questions and comments:


*Fix to emailing report with attached .msg report not working?*

*(if email reports need to be zipped, ignore this)*


I just tested, sending a zip of a .msg has the analyze report works
correctly.  Sending just the .msg attachment with 21293 give the previous
errors.   *Sending just the .msg with this new 21302 unfortunately is worse
and isn't working*.  I'm now getting:

found a possible MIME header start in the middle of the mail - the analyze
may be wrong

[CR][LF] •

Subject: FW: test of report message[CR][LF]

and even less info is shown in the emailed analyze report than before

I had ReportLog set to debug.  The .eml file that goes to debug is attached
to this reply.




*Clarification on the need now(?) to compress Emailed reports from Outlook*

You wrote: "Notice: always compress (e.g. zip) reported emails before they
are sent to assp!"   The GUI says " It is also possible to send MS-outlook
'.msg' files (possibly zipped)."  Is it now required?

 I've always done "Forward as Attachment" in Outlook to report which
attaches the current message as a .msg formatted file (I previously
incorrectly wrote .eml) and seems to work (except for the analyze big that
you're attempting to squash). The vast majority of our staff do the same,
but requiring them to first save the .msg, then zip, then attach to report
might be asking too much.

While attached Outlook .msg files are binary, I don't >>think<< they're
compressed.  The .msg files always have made it to the corpus saved as
plain text files which seems right.  It was only the analyze report that
was failing.






*MaxBytesReports*

The gui talks about MaxBytesReports.

Any mail sent or forwarded by local/authenticated users to this username
will be interpreted as a spam report. Multiple attachments get truncated to
MaxBytesReports. Do not put the full address here, just the user part.

For example: asspspam . Use a fake domain like @assp.local or @
assp-nospam.org when you send the email- so the full address would be then
asspspam@assp.local.

You can sent multiple mails as attachments and/or zipped file(s). Each
attached email-file must have the extension defined in "maillogExt". In
this case only the attachments will be processed. To use this
multi-attachment-feature an installed Email::MIME module in PERL is needed.
It is also possible to send MS-outlook '.msg' files (possibly zipped). To
use this MS-outlook-feature in addition an installed
Email::Outlook::Message module in PERL is needed.


I don't see MaxBytesReports any where else in the code.  Is this supposed
to be MaxBytes?


Also, GUI correction  " You can sent multiple mails as attachments and/or
zipped file(s)" should be "send"




*Finding DKIM matches during rebuild:*


AddDKIMHeader needs to be on right?

Since it looks like the code is looking for the X-ASSP-DKIMIdentity line, I
think you should add a comment for the hidden DoRBWhite parameter and others
* that AddDKIMHeader in the GUI needs to be on for this to work. *


Speed -> more configuration choices necessary?

Great point about slowness in rebuild if this is all on and the
recommendation of potentially only turning it on periodically.


Since we're now doing more checks, does it make sense to have additional
hidden parameters to give more granular control?  I feel like I'll always
want to check for whitelisted in spam no matter how it matches, but other
might only want to consider DKIMWLAddress, while others don't want
DKIMWLAddresses and only matches to the actual whitelist.  How about
letting DoRBWhite be configured with different values like we have for
DoNoFrom?

Match:

0: nothing

1: whiteRe

2: npRe

4: whiteListedDomains

8: noProcessingDomains

16: whiteListedIPs

32: noProcessingIPs

64: DKIMWLAddresses

128: DKIMNPAddresses

and have those summed up?  Too complicated for not enough value?  Dunno,
thinking out loud here.  I'm cool with everything on, but maybe there are
others who would prefer to more granularly configure?



related: GUI mistake.  the AddDKIMHeader description still says that it
adds X-ASSP-DKIM: instead of "X-ASSP-DKIMidentity



*DoRBBlack removal of deny matches --  curiosity:*

For the new DoRBBlack, why is it checking denySMTPConnectionsFromAlways and
denySMTPConnectionsFrom?  Aren't additions made to that list after we've
collected what we've wanted (good or bad) from those IP's / emails which
would be good to have in the corpus?


*NWLI*

I'd like to rewrite the NWLI description at the bottom of the GUI, but I
need clarification first.  I'm sure NWLI functionality works in the code,
it's just not explained well in the GUI.

I see the revised language, but I'm still not sure that I follow.  When you
say 

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

More notes and questions that might be of interest based on what I've
witnessed:

Yesterday, I improved my script to update an included IP list file ONLY
when there is an IP change detected.   I ran the script fresh, then
restarted ASSP.   No main thread stuck warnings for several hours.  ASSP
wasn't saying that the files were modified, because they weren't.   At
10:30pm, one of the SPF records changed, so a single include file was
updated.  At 10:33 the main thread warning unable to transfer messages
start.

Before each hang (every 5 minutes), ASSP's bug would tell me that the
included file has changed.  I am guessing that the bug you found is ASSP
getting the file mod date at startup, but not recording the new date after
it detects a change.  At least we know why this happens every 5 minutes,
but I'm more interested in *why it's happening even once.*

To me, the main thread unable to transfer error indicates that there's
something going really slow when the rebuild of the regexes happens.
Because the external file changes, at least the group that uses that file
needs to be rebuilt.  Or, does the whole groups file get rebuilt and then
all rules that reference groups also gets rebuilt with the new regex?

Could my newly reformed DKIMWLAddress list that starts generally with [@|.]
be pushing me over the threshold, since each line is now an OR or does ASSP
handle optimizing that so it's a non-issue?

Whatever the case, not knowing how threading/workers actually work, why is
the regex rebuild seemingly using all workers and not just the maintenance
worker, leaving the other workers available?

thanks


On Wed, Oct 27, 2021 at 1:40 PM K Post  wrote:

>   > Apart from the fact that the MaintThread (1) does not store the
> file time correctly for 'Groups' included files (which will be fixed),
> Okay, at least that explains why it was trying to reload every 5 minutes.
>  Hopefully the fix in the ASSP code combined with mine (see below) to stop
> my script from updating the included files hourly, will stop the main
> thread from getting stuck so often during the reload.   Once an hour is
> bearable.
>
> >everything else you asked is answered in the GUI for an IT prof.
> I know you don't believe that I've ever read the GUI.  I'm a big proponent
> of RTFM, and always search and re-read sections before asking questions
> here.  Believe it or not, I write a whole lot of documentation.  It's the
> least favorite part of my "IT Pro" job, but I, as you do, recognize it as
> essential.  I cannot find anywhere in the ASSP GUI where it talks about
> an included external files being encrypted once it's read by ASSP.  This
> detail is either not in the GUI or isn't clear enough that I was able to
> find it.
>
>
> >Every security related content is encryped or not shown by assp.
> ASSP has no way of knowing if those external includes have passwords for
> LDAP in them, so I understand encrypting them, and that's fine.   Being
> that I was trying to ascertain why my config reloads were seemingly causing
> the main thread, and I didn't *see or read *anything (not that it's not
> there, I just don't see where), I asked about this encoding / encryption to
> make sure that this was expected and wasn't the cause of the issues I am
> seeing.
>
>
> >>1 33 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> >This is the last finished (debug) step (and the seconds since then - 33)
> in rereading the config, before the new regular expressions (IP) are build
> by each thread.
>
> That's very helpful information.  Could / should the status screen be
> updated to show processes as they start, instead of showing the last one
> that finished?  ConfigChangeTLS isn't really the current action as the
> status gui says, the regex rebuild has started.  I still don't follow why
> all of the workers are showing this previous step though.  Shouldn't only 1
> worker have been doing it, leaving the others available?  I have no idea,
> that's why I'm asking.
>
>
> Last night, I disabled the SPF update process then restarted ASSP (the bug
> you found seems to only be after a file changes.  If assp starts and the
> external files don't change after that, they're not detected as changed
> every time).  NO hangs all night.  I ran the script manually once just now
> and received the unable to transfer warning one assp picked up the
> changes.  So it sure seems like my issue is directly related to these
> external files.
>
> The IP lists that I generate from the SPF records are only 50kb combined
> in total, so by no means massive.  I was seeing the error when they were
> only 25kb total (before I discovered an issue with the URL I was getting
> yahoo ip's from).  Maybe the rebuild of those plus the other regexes just
> take

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

nerated by WriteFile-GetDomainIPSFromSPF.pl
#
157.56.232.0/21 FROMSPF: spf-a.outlook.com
157.56.240.0/20 FROMSPF: spf-a.outlook.com
207.46.198.0/25 FROMSPF: spf-a.outlook.com
207.46.4.128/25 FROMSPF: spf-a.outlook.com
157.56.24.0/25 FROMSPF: spf-a.outlook.com
(more)
157.55.9.128/25 FROMSPF:
40.92.0.0/15 FROMSPF: spf.protection.outlook.com
40.107.0.0/16 FROMSPF: spf.protection.outlook.com
52.100.0.0/14 FROMSPF: spf.protection.outlook.com
104.47.0.0/17 FROMSPF: spf.protection.outlook.com
(more)
157.55.2.0/25 FROMSPF: spf-a.hotmail.com
65.54.190.0/24 FROMSPF: spf-a.hotmail.com
65.54.51.64/26 FROMSPF: spf-a.hotmail.com
65.54.61.64/26 FROMSPF: spf-a.hotmail.com
65.55.111.0/24 FROMSPF: spf-a.hotmail.com
(more)
94.245.112.0/27 FROMSPF: _spf-ssg-c.microsoft.com
111.221.26.0/27 FROMSPF: _spf-ssg-c.microsoft.com
207.46.50.192/26 FROMSPF: _spf-ssg-c.microsoft.com
207.46.50.224 FROMSPF: _spf-ssg-c.microsoft.com



So WHY am I doing this?
Microsoft Outlook/Hotmail/365 IPs are often abused, but the vast majority
of users are legitimate.  There's tons of shared outgoing mail IP's in this
massive infrastructure.  I use the IP's that my script builds in:

noHelo - so one really bad office365 user doesn't manage to get the helo
for any microsoft SMTP server blocked for everyone else who is using it
noBlockingIPs - same concept.  Once bad actor shouldn't get a MS ip blocked
for the rest
noPB  - same
noPBWhite   also, don't make it so that good senders automatically make it
easier for bad senders to use Microsoft IP's
noExtremePB - probably not necessary since it's already in noPB
noDelay - stuff sent through MS IP's will be retried, so delaying is
pointless.  Might as well let legit senters come through right away.
noRBL - we've seen Microsoft ip's get on the DNSBL.

The same principles are used for other providers who send on behalf of many
organizations, including legitimate bulk emailers like constant contact who
sometimes are guilty of sending spam, but usually are good.

Is there a better way?  Is this a bad idea? It's worked well for me, with
the exception of ASSP giving the main thread stuck warnings occasionally at
reload.  I'd be happy to be able to stop using the script, but I don't know
of another way to accomplish what I'm looking for there.  What do you do to
make sure that a provider like Google or Microsoft doesn't have shared SMTP
ip's blocked because of a couple bad actors?

Thanks again for hearing me out!


On Wed, Oct 27, 2021 at 5:26 AM Thomas Eckardt 
wrote:

> Apart from the fact that the MaintThread (1) does not store the file
> time correctly for 'Groups' included files (which will be fixed),
> everything else you asked is answered in the GUI for an IT prof. Every
> security related content is encryped or not shown by assp.
>
> >I just saw this again in the statusassp gui
> 1 33 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
>
>
>
> This is the last finished (debug) step (and the seconds since then - 33)
> in rereading the config, before the new regular expressions (IP) are build
> by each thread.
>
> >For years, I've run an external script to query various providers' SPF
> records and get the IP addresses for them into a file.
>
> For what reason?
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:27.10.2021 08:45
> Betreff:Re: [Assp-test] Main_Thread is unable to transfer
> connection to any worker - try again
> --
>
>
>
>
> Looking more at these include files in my groups definition.  If I open
> them in the OS immediately after they're generated, it looks as expected.
> Something like:
> #
> # Generated by WriteFile-GetDomainIPSFromSPF.pl
> #
> *198.2.128.0/24* <http://198.2.128.0/24> FROMSPF: *spf.mandrillapp.com*
> <http://spf.mandrillapp.com/>
> *198.2.132.0/22* <http://198.2.132.0/22> FROMSPF: *spf.mandrillapp.com*
> <http://spf.mandrillapp.com/>
> *198.2.136.0/23* <http://198.2.136.0/23> FROMSPF: *spf.mandrillapp.com*
> <http://spf.mandrillapp.com/>
> *198.2.145.0/24* <http://198.2.145.0/24> FROMSPF: *spf.mandrillapp.com*
> <http://spf.mandrillapp.com/>
>
> Once ASSP loads them though, it goes into some kind of encoded/encrypted
> format, a single long line, all hexidecimal like:
> 2acc4d3156084a1a3edc250c1f32bc5 (continues)
>
> *Is this normal behavior?*  I haven't found any mention of it in the GUI.
> I'm not using configuration sharing.  The other config files don't seem to
> have this hexidecimal conversion, only the ones I generate with my script.
>
> I can edit the included files from the Groups section, by clicking the
> various "Edit included file "

[Assp-test] Analyze email report from spam report not fully processing headers from Outlook attached .eml

[K Post]

*SUMMARY*: Emailed analyze reports when in response to a forward as
attachment error report .eml from Outlook, doesn't seem to find the
subject, from, and potentially more.  The .eml is extracted and saved
properly in the error-spam/error-notspam folder though, with the
subject/from/etc.



For the longest time, any time someone forwards a message to the spam or
notspam reporting address by doing a "forward as attachment" from MS
Outlook, the resulting analyze report is broken.
I found an unanswered post from me from 2016 on this same problem.
Exchange was always in the mix here, but in 2016, it was a totally
different set up with the same symptoms.

Working as expected: The content of .eml file in the error report to
EmailHam/EmailSpam will appear in errors-spam or errors-notspam correctly
with the contents intact.  If I go into the log, find the entry with the
.rpt file and analyze that, it shows everything correctly including the
FROM line, the original subject, and information about the original DKIM
signature.   I'm confident that it'll be used for rebuild properly, so good
there.*✔*


However, the *analyze report *that is automatically sent when a message is
sent in as an attachment (either as an email to EmailHam, EmailSpam, or
EmailAnalyze) from MS Outlook as an .eml attachment shows:


General Hints:


m...@ourcharity.org has requested this analyze report
analyze is restricted to a maximum length of 10791 bytes  *<-- temporarily
set to 10k.  I've tried 25k too.  Doesn't matter *

*(separate question: is the 3k spam average still true?? most of mine are
20k+ and notpsm is 60k+ average.  Is 3000 still a recommended size for a
mature installation?  *

*related, would it be possible to consider only MaxBytes for bayesian, but
have bomb expressions search more of a message or would the be too slow /
cumbersome?)*

attachments will be fully analyzed using ASSP_AFC
attachments will be fully scanned for viruses
text processing uses unicode normalization
regular expression matches and results are truncated to 32 (RegExLength)
characters
removed all local X-ASSP- header lines for analysis

sender and reply addresses:
*MAIL FROM: r...@badsender.org<--- envelope from's
found.  envelope from is in the report file.  So at least some of the
header is exposed to analyze*

recipient addresses:
RCPT TO: ad...@ourcharity.org
using enhanced Originated IP detection for all except the most origin IP
addresses
•detected IP's on the mail routing way: 2603:10b6:a03:1e4:0:0:0:24(no PTR)
•detected source IP: 2603:10b6:a03:1e4:0:0:0:24


Subject: no subject found   *<-- no subject?  it's in the header.  I see it
in the .rpt file*
Feature Matching:

• DoNoFrom: detected (1) faults in scoring mode - last reason: missing
'From:' and 'Sender:' header tag ( DoNoFrom ) - penalty: 1 * 50 = 50  --
shouldn't be!!
• DKIM-check returned OK no domain to fetch policy for for identity ''  <--
of course that's a problem, if there's no from
• URIBL check: 'OK'
• RBLCacheCheck returned OK for 2603:10b6:a03:1e4:0:0:0:24: inserted as ok
at 2021-10-26 17:11:05
• domain ipv4depot.com (in Mail From:) has a valid MX record:
badsender-org.mail.protection.outlook.com
• domainMX ipv4depot-com.mail.protection.outlook.com has a valid A record:
104.47.57.110
• PTR record via DNS: status=no PTR
• RWLcheck returned OK for : status=unknown

Then the feature matching log is displayed, still complaining about no
from, bad DKIM.


I've spent the better part of 2 hours looking at the ConfigAnalyze
function.  I can't spot where the issue lies, if there is one, but I'm
hopeful it's an easy fix (or nudge in the right direction for me).


Thanks again for hearing me out on so much in the last couple of days


Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

Looking more at these include files in my groups definition.  If I open
them in the OS immediately after they're generated, it looks as expected.
Something like:

#
# Generated by WriteFile-GetDomainIPSFromSPF.pl
#
198.2.128.0/24 FROMSPF: spf.mandrillapp.com
198.2.132.0/22 FROMSPF: spf.mandrillapp.com
198.2.136.0/23 FROMSPF: spf.mandrillapp.com
198.2.145.0/24 FROMSPF: spf.mandrillapp.com

Once ASSP loads them though, it goes into some kind of encoded/encrypted
format, a single long line, all hexidecimal like:
2acc4d3156084a1a3edc250c1f32bc5 (continues)

*Is this normal behavior?*  I haven't found any mention of it in the GUI.
I'm not using configuration sharing.  The other config files don't seem to
have this hexidecimal conversion, only the ones I generate with my script.

I can edit the included files from the Groups section, by clicking the
various "Edit included file " button.  It looks normal, not hex
encoded.

Note: the bottom of the editor window says: "First line specifies text that
appears in the subject of report message. The remaining lines are the
report message body."   I'm guessing that's just an oversight.



On Tue, Oct 26, 2021 at 11:17 AM K Post  wrote:

> That's helpful Thomas, thank you.  I've clearly got more digging to do,
> and I will, but I could use more guidance on what I've found so far if you
> have the time.
>
> 1)  *ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)*
>
> I just saw this again in the statusassp gui
>
> 1 33 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 2 32 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 3 31 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 4 30 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 5 29 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 6 19 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 7 27 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 1 26 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
> 10001 39 s call to
> ConfigChangeTLSPorts->(TLStoProxyListenPorts,'',,Initializing)
>
>
> With worker logging set to diagnostic, I see saw a full screen of
>
>  Oct-26-21 10:14:06 Info: Main_Thread can't interrupt Worker_4 (2) at the
> moment - try next worker
> Oct-26-21 10:14:06 Info: Main_Thread is unable to interrupt any worker for
> new connection - wait for available worker (max 30 seconds)
> Oct-26-21 10:14:06 Info: try to interrupt worker Worker_3 (0) for new
> connection
> Oct-26-21 10:14:06 Info: Main_Thread can't interrupt Worker_3 (0) at the
> moment - try next worker
> Oct-26-21 10:14:06 Info: try to interrupt worker Worker_2 (0) for new
> connection
> Oct-26-21 10:14:06 Info: Main_Thread can't interrupt Worker_2 (0) at the
> moment - try next worker
> Oct-26-21 10:14:06 Info: try to interrupt worker Worker_5 (0) for new
> connection
> Oct-26-21 10:14:06 Info: Main_Thread can't interrupt Worker_5 (0) at the
> moment - try next worker
>
> ...many many lines of that in a row.
>
> What could be causing ConfigChangeTLSPorts to be initializing using all
> workers?  I've seen this before with POP3 configure, even though POP3 isn't
> used and isn't configured.
>
>
> 2) External config files always showing as changed???
>
> For years, I've run an external script to query various providers' SPF
> records and get the IP addresses for them into a file.  (provided a couple
> times here for community use). I then use that file in group definitions.
> For example, in the Groups file, I have:
> [GROUP-SENDGRID-IPS]
> # include IP-Lists/IPS-sendgrid.com.cfg
> # include IP-Lists/IPS-sendgrid.net.cfg
>
> Those 2 .cfg files are generated by the script doing a spf lookup for
> sendgrid.com  and another for sendgrid.net.  It recursively processes the
> records.  The script runs once every 3 hours to capture any DNS changes
> that providers might make.   The script isn't smart enough to know if
> anything has changed, it just re-writes the files every time.   Once ASSP
> reads them, it seems that they get saved encrypted.  I am able to view them
> fune using the GUI, but opening the .cfg files from the OS shows an
> encrypted long string.
>
> Even though my script only changes the files every 3 hours, I'm seeing
> entries in the log (with maintenance logging set to diagnostic) which
> indicates that ASSP thinks that they're changing every time, and it seems
> to be checking these files twice per scheduled reload.
>
>
> Oct-26-21 10:36:00 Worker_4 will sleep now
> Oct-26-21 10:36:03 Info: (re)scheduled ReloadOpti

Re: [Assp-test] Concept question: At rebuild, look at DKIMWLAddresses?

[K Post]

What about "only" checking corpus files for DKIMWLAddresses or
DKIMNPAddresses matches only if AddDKIMHeader is enabled?  It would just
need to match the regex against the X-ASSP-DKIMidentity.

notspam would be checked against DKIMNPAddresses and spam would be checked
against DKIM NP and WL.  If a server's processor can handle it at rebuild
time, wouldn't this be a good thing to have as an option?

I don't know why we'd need to check denySMTPConnectionsFromAlways or
denySMTPConnectionsFrom during the rebuild process.   blackListedDomains,
redRe would be nice, but again, we need to balance performance here

On Tue, Oct 26, 2021 at 3:36 AM Thomas Eckardt 
wrote:

> >*without rebuild taking too much of a performance hit?*
>
> fastest case: all in the file model
>
> slowest case:
> - no file model
> - checking whiteRe, whitelist, npRe, DKIMWLAddresses, DKIMNPAddresses,
> redRe - for assp/spam
> - checking denySMTPConnectionsFromAlways, denySMTPConnectionsFrom,
> blackListedDomains for assp/notspam
>
> The slowest  case is 12 times slower than the fastest.
>
> Thomas
>
>
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:24.10.2021 02:40
> Betreff:[Assp-test] Concept question: At rebuild, look at
> DKIMWLAddresses?
> --
>
>
>
>
> *Would it be possible to have ASSP consider DKIMWLAddress matches during
> rebuild, removing matching messages from spam, without rebuild taking too
> much of a performance hit? *
>
> During rebuild, ASSP runs rb_whitelisted against each message in the spam
> corpus, and if a match is found against the whitelist, that message is
> removed from spam.  (right?) It's a terrific way to help keep the corpus
> clean after a whitelist addition.
>
> I rely heavily on DKIMWLAddresses - it's super helpful to consider a
> message whitelisted *only* when the DKIM signature matches.
>
> *If it's realistically possible and not ill conceived, removal of messages
> from spam where there's a DKIMWLAddress match would further clean up spam,
> and lead to more accurate HMM/Bayesian detections. *
>
> What do you think?
>
> Along the same lines, what about considering messages that match no
> processing rules: the regexes and DKIMNPAddresses for messages in both spam
> and notspam?
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

vel-3-TLDlist via direct
> HTTP connection
> Oct-25-21 23:12:59 [Worker_1] Level-3-TLDlist download completed
> Oct-25-21 23:12:59 [Worker_1] Info: next TLDlist download in 1 day 5
> hours 58 minutes
> Oct-25-21 23:12:59 [Worker_1] Info: file c:/assp/files/URIBLCCTLDS.txt
> updated for URIBLCCTLDS
> Oct-25-21 23:13:00 [Main_Thread] Saving config
> Oct-25-21 23:13:00 [Main_Thread] Info: no configuration changes detected -
> nothing to save - file c:/assp/assp.cfg is unchanged
> Oct-25-21 23:13:00 [Main_Thread] Adminupdate: file
> 'c:/assp/files/URIBLCCTLDS.txt' for config 'URIBLCCTLDS' was changed
> Oct-25-21 23:13:00 [Main_Thread] Option list file:
> 'c:/assp/files/URIBLCCTLDS.txt' reloaded (URIBLCCTLDS) with 12,280 records
> Oct-25-21 23:13:02 [Worker_1] Worker_1 wakes up
> Oct-25-21 23:13:02 [Worker_5] Worker_5 wakes up
> Oct-25-21 23:13:02 [Worker_4] Worker_4 wakes up
> Oct-25-21 23:13:02 [Worker_3] Worker_3 wakes up
> Oct-25-21 23:13:02 [Worker_2] Worker_2 wakes up
> Oct-25-21 23:13:03 [Worker_1] Worker_1 finished reloading configuration
> Oct-25-21 23:13:03 [Worker_1] Worker_1 will sleep now
> Oct-25-21 23:13:04 [Worker_2] Worker_2 finished reloading configuration
> Oct-25-21 23:13:04 [Worker_2] Worker_2 will sleep now
> Oct-25-21 23:13:05 [Worker_3] Worker_3 finished reloading configuration
> Oct-25-21 23:13:05 [Worker_3] Worker_3 will sleep now
> Oct-25-21 23:13:06 [Worker_4] Worker_4 finished reloading configuration
> Oct-25-21 23:13:06 [Worker_4] Worker_4 will sleep now
> Oct-25-21 23:13:07 [Worker_5] Worker_5 finished reloading configuration
> Oct-25-21 23:13:07 [Worker_5] Worker_5 will sleep now
> Oct-25-21 23:13:08 [Worker_1] Worker_1 finished reloading
> configuration
> Oct-25-21 23:13:10 [Worker_10001] Worker_10001 finished reloading
> configuration
> Oct-25-21 23:13:29 [Worker_1] Downloading Extended Droplist via direct
> HTTP connection
> Oct-25-21 23:13:29 [Worker_1] Extended Droplist already up to date
> Oct-25-21 23:13:29 [Worker_1] Info: next droplist download in 7 hours
> 34 minutes
>
>
> debug may help
> for time related debugging, I (or some one who read the manual) would
> consider to use ConfigChangeSchedule
>
> notice: analyzing all the produced debug files (in general debug mode) is
> a very time consuming task
>
> Check your option files for bad (too greedy) regular expressions. Check,
> if there are other processes modifying assp files. Check that required
> services (DNS, SQL,ClamAV,. ) are responsive at this time.
>
> If (for any reason) it is expected, that the config reload takes 30
> seconds or longer - 'ConnectionTransferTimeOut' should be changed - or the 
> "Warning:
> Main_Thread is unable to transfer connection to any worker - try again!"
> should be ignored.
>
> If the reload takes X seconds for the MainThread. Within these X seconds
> all new connections are queued by the OS. After this time (the reload) the
> MainThread tries to transfer all these new connections within some
> (milli)seconds to the workers - this may overload the SMTP-workers for some
> time.
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:25.10.2021 17:20
> Betreff:Re: [Assp-test] Main_Thread is unable to transfer
> connection to any worker - try again
> --
>
>
>
> We are lucky to have a new (to us) faster server donated since this
> original May posting on the "unable to transfer connection to any worker"
> error.  However, with the new box with Windows 2019 installation, fully
> patched, MySQL latest community, and Strawberry perl 5.32 it's still
> happening in spurts.Not a heavy load, 16gb ram.  12 cores total.  ASSP
> uses about 1.7gb after running for a bit.  MySQL seems fast, ClamAV in use.
>
> Sometimes I get the warning just once in a day, often it's every *5
> minutes* for a while.  This morning, there were 3 occurrences with 10
> minutes in between each.  But always, when I look at the log, it's right
> after the "Saving config" process, always when there's no config changes.
>
> I tried to see where SaveConfig() is being called from every 5 minutes.  I
> >>think<< it's when the ReloadOptionFiles timer hits, but I'm not sure.
> ReloadOptionFiles is set to 300 (5 minutes)
> AutoReloadCfg is enabled, though I'm never modifying assp.cfg without
> using the GUI
> What else should I be looking at?
>
>
> From today:
> Oct-25-21 11:09:24 Saving config
> Oct-25-21 11:09:24 Info: no configuration changes detected - nothing to
> save - file d:/assp/assp.cfg is unchan

[Assp-test] NotifyRE matches whole log line, including timestamp

[K Post]

Question summary:

*In the mLog function where ASSP does the NotifyRe comparison, does it make
sense to consider having ASSP look at $comment instead of the full $m
variable which starts with the timestamp?*


For a long time, I've had the GUI example of

warning:

in my NotifyRe configuration.  It has worked great, alerting me via email,
whenever ASSP logs a warning.

Yesterday, I got hit with hundreds of notifications of warning throughout
the day, but not because ASSP was logging warnings, it's because we got
slammed with a scam email with a subject that contained "warning:" with the
colon in the email subject.


To avoid this in the future, I figured that I could just prefix warning:
with ^ to match the start of the line, but my testing shows that ASSP
compares NotifyRe against the entire log line, including the timestamp.

I accomplished what I'm looking for by changing the line in NotifyRe to be:

^[A-Za-z]{3}-\d\d-\d\d \d\d:\d\d:\d\d\swarning:

Might it make more sense to have ASSP look at $comment instead of the full
$m variable (includes timestamp) when it does the mLog function?  That way,
to match ASSP info, warning, and error lines, we wouldn't need to process
the more complicated regex compare on every line.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

We are lucky to have a new (to us) faster server donated since this
original May posting on the "unable to transfer connection to any worker"
error.  However, with the new box with Windows 2019 installation, fully
patched, MySQL latest community, and Strawberry perl 5.32 it's still
happening in spurts.Not a heavy load, 16gb ram.  12 cores total.  ASSP
uses about 1.7gb after running for a bit.  MySQL seems fast, ClamAV in use.

Sometimes I get the warning just once in a day, often it's every *5 minutes*
for a while.  This morning, there were 3 occurrences with 10 minutes in
between each.  But always, when I look at the log, it's right after the
"Saving config" process, always when there's no config changes.

I tried to see where SaveConfig() is being called from every 5 minutes.  I
>>think<< it's when the ReloadOptionFiles timer hits, but I'm not sure.
ReloadOptionFiles is set to 300 (5 minutes)
AutoReloadCfg is enabled, though I'm never modifying assp.cfg without using
the GUI
What else should I be looking at?


>From today:

Oct-25-21 11:09:24 Saving config
Oct-25-21 11:09:24 Info: no configuration changes detected - nothing to
save - file d:/assp/assp.cfg is unchanged
(a message processed)
Oct-25-21 11:09:57 Info: notification message queued to sent to
assp-not...@ourcharity.org
Oct-25-21 11:09:57 Warning: Main_Thread is unable to transfer connection to
any worker - try again!


assp acts normally for 10 minutes, then:


Oct-25-21 11:19:28 Saving config
Oct-25-21 11:19:28 Info: no configuration changes detected - nothing to
save - file d:/assp/assp.cfg is unchanged
Oct-25-21 11:20:00 Info: notification message queued to sent to
assp-not...@ourcharity.org
Oct-25-21 11:20:00 Warning: Main_Thread is unable to transfer connection to
any worker - try again!


assp resumes


Oct-25-21 11:29:32 Saving config
Oct-25-21 11:29:32 Info: no configuration changes detected - nothing to
save - file d:/assp/assp.cfg is unchanged
(a single message processed fine here)
Oct-25-21 11:30:04 Info: notification message queued to sent to
assp-not...@ourcharity.org
Oct-25-21 11:30:04 Warning: Main_Thread is unable to transfer connection to
any worker - try again!




On Mon, May 17, 2021 at 8:53 PM K Post  wrote:

> I'm desperate for help.  It seems that ASSP has thread problems frequently
> when it reloads the config.  I see warnings that the Main_Thread is unable
> to transfer connection to any worker, often every 5 minutes for hours.
>
> May-17-21 20:40:35 Saving config
> May-17-21 20:40:35 Info: no configuration changes detected - nothing to
> save - file c:/assp/assp.cfg is unchanged
> May-17-21 20:41:08 Info: notification message queued to sent to
> assp-not...@ourcharity.org
> May-17-21 20:41:08 Warning: Main_Thread is unable to transfer connection
> to any worker - try again!
>
> Even when there's no settings detected (line 2 above), it can hang.
>
> I can't seem to figure out why.
>
> Windows 2012 R2
> Strawberry Perl
> Latest ASSP
>
> Any guidance on where to start?
>
> Thank you
>
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Concept question: At rebuild, look at DKIMWLAddresses?

[K Post]

"only for spam (not for corrected spam) AND only if enabled (DoRBWhite ,
DoRBRed)"
Yes, only the spam corpus, and I have both DoRBWhite and DoRBRed set to 1
in CorrectASSPcfg.pm.  I forgot that this was a manual setting.

But back to my original question, do you think there's value in doing
something similar with message removals for DKWIMWLAddresses matches in
spam, and DKIMNPAddresses in both the spam and notspam folders?  If so, I
could try to cobble together some code if that's what you want - but again,
you've seen my coding...

On Sun, Oct 24, 2021 at 9:02 AM Thomas Eckardt 
wrote:

> >During rebuild, ASSP runs rb_whitelisted against each message in the
> spam corpus
>
> only for spam (not for corrected spam) AND only if enabled (DoRBWhite ,
> DoRBRed) !
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:24.10.2021 02:40
> Betreff:[Assp-test] Concept question: At rebuild, look at
> DKIMWLAddresses?
> --
>
>
>
>
> *Would it be possible to have ASSP consider DKIMWLAddress matches during
> rebuild, removing matching messages from spam, without rebuild taking too
> much of a performance hit? *
>
> During rebuild, ASSP runs rb_whitelisted against each message in the spam
> corpus, and if a match is found against the whitelist, that message is
> removed from spam.  (right?) It's a terrific way to help keep the corpus
> clean after a whitelist addition.
>
> I rely heavily on DKIMWLAddresses - it's super helpful to consider a
> message whitelisted *only* when the DKIM signature matches.
>
> *If it's realistically possible and not ill conceived, removal of messages
> from spam where there's a DKIMWLAddress match would further clean up spam,
> and lead to more accurate HMM/Bayesian detections. *
>
> What do you think?
>
> Along the same lines, what about considering messages that match no
> processing rules: the regexes and DKIMNPAddresses for messages in both spam
> and notspam?
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Concept question with sample code: DKIMWLAddresses single line for .domain.com and @domain.com?

[K Post]

(I believe I have this working, see code please)

I estimate that at least 90% of the time that I want to add an entry to
DKIMWLAddresses that I put two lines, one for @domain.com and the other to
match the subdomains, so .domain.com.

That's fine,and works well, but I'd like to see ASSP allow admins to use a
single line shorthand for signatures ending in both .domain.com and @
domain.com.  So instead of

@domain.com
.domain.com

just do the shorthand of

>>domain.com

The last time I asked for a new feature, you suggested that I try coding it
myself, so I did.  There's no pride in authorship here, I'm a LOUSY coder.
I'd love to see how you tackle this, provided that my concept's sound.

I picked the >> characters, since > is illegal in email addresses / domain
names.  Originally, I selected the + character, but that can be in the user
part of the email address.  >> is highly visible when scanning through
config files, a single > wasn't as easy for me to spot.

So,I tested out modifying the setDKIMNPAddressesRE and setDKIMWLAddressesRE
functions, from:
my $new=shift;
$new||=$neverMatch; # regexp that never matches
SetRE('DKIMWLAddressesRE',"(?:$new)\$",
  $regexMod,
  'DKIM whitelisted',$_[0]);

to:
my $new=shift;
$new||=$neverMatch; # regexp that never matches
*$new=~s/>>(.*)(\||$)/(\\\.|\\\@\)$1$2/go;*
SetRE('DKIMWLAddressesRE',"(?:$new)\$",
  $regexMod,
  'DKIM whitelisted',$_[0]);

If my hack coding is correct, this will take the existing string, which is
already a regex compiled by ASSP based on the non-regex DKIMWL/NPAddresses
entered by the user, look for

>>whatever| or >>whatever at the end of the string and change that to
(\@|\.)whever| or no | if it's the end of the string

In my rudimentary testing, that seems to work.

*What do you think?  *

That would cut down my DKIMWLAddresses down by close to 50%, and make
management much easier.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Concept question: At rebuild, look at DKIMWLAddresses?

[K Post]

*Would it be possible to have ASSP consider DKIMWLAddress matches during
rebuild, removing matching messages from spam, without rebuild taking too
much of a performance hit? *

During rebuild, ASSP runs rb_whitelisted against each message in the spam
corpus, and if a match is found against the whitelist, that message is
removed from spam.  (right?) It's a terrific way to help keep the corpus
clean after a whitelist addition.

I rely heavily on DKIMWLAddresses - it's super helpful to consider a
message whitelisted *only* when the DKIM signature matches.

*If it's realistically possible and not ill conceived, removal of messages
from spam where there's a DKIMWLAddress match would further clean up spam,
and lead to more accurate HMM/Bayesian detections. *

What do you think?

Along the same lines, what about considering messages that match no
processing rules: the regexes and DKIMNPAddresses for messages in both spam
and notspam?
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21293

[K Post]

Thanks for giving this more consideration.  I think these GUI changes will
help many!  The examples of NWLI are great and clarify a lot.

The only thing I don't see in the example is* the use of the + (only)
switch and how it differs from just having a N instead of N+ for example. *
I see where you wrote:

The NWLI conditions defined in a line are combined using a logical AND --
so N-W+ is combined to: NOT noprocessing AND whitelisted


That's clear enough, but what's the difference then between N-W+ and N-W
(without the +)?   Wouldn't that still be not noprocessing and
whitelisted?  Is the + optional?



On Wed, Oct 20, 2021 at 7:55 AM Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21293:
>
> - if a file for regular expressions contained an incomplete default
> definition for the !!!NWLI!!! directive, this directive was not applied to
> the regexes in the file
>
>
>
> changed:
>
> - some corrections and additions to the main help text in the GUI
>
> - the behavior of the 'NWLI' extension in regular expression definitions
> is enhanced
>
> The NWLI conditions defined in a line are combined using a logical AND --
> so N-W+ is combined to: NOT noprocessing AND whitelisted.
> In fact, the weight is skipped, if any of the defined NWLI options does
> not match for a mail. If multiple lines would match, the weight of the
> first matching line is used.
> This way you can define different weights for the same regular expression,
> but different mail states like in this example:
> (1) foo=>0:>NW - weight is zero if noprocessing AND whitelisted
> (2) foo=>0.5:>NW- - weight factor is 0.5 if noprocessing AND NOT
> whitelisted
> (3) foo=>1.5:>N-W - weight factor is 1.5 if NOT noprocessing AND
> whitelisted
> (4) foo=>55:>N-W- - weight is 55 if NOT noprocessing AND NOT whitelisted
> (5) foo=>2:>W - this line will not be processed, because line 1 or 3 would
> have matched before, depending on the noprocessing flag
> (6) foo=>2:>N- - this line will not be processed, because line 3 or 4
> would have matched before, depending on the whitelisted flag
>
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[K Post]

Sorry for making you angry, though I'm not sure how I have this time.  I'm
agreeing with you.  No need for your reply, I certainly don't want to upset
you or make it sound like I'm arguing.  I was simply suggesting a
language addition to the GUI (or the "manual" as you call it) to help add
clarity that NoProcessing does NOT override nonBaysian/HMM scoring by
default and to look at !!!NWLI!!! at the bottom of the gui for more
information.  That way, someone doesn't see "NoProcessing" and completely
incorrectly assume that it actually does not process the message at all.


On Mon, Oct 18, 2021 at 1:20 PM Thomas Eckardt 
wrote:

> >Note: Messages flagged as "no processing" will not contribute to the
> corpus
> there is no option to store noprocessing in the corpus folders
>
> >will not be scored based on Bayesian/HMM detection.
> not true - config option
>
>
> >However, by default, "Penalty Box" / Bomb scroing will still take place.
> NOT by default - EVERYTIME - nothing else is written in the manual
>
> See !!!NWLI!!! option to override this default behavior.
>
> nonsense
>
>
> THERE IS NO OTHER WAY to get knowledge about ASSP than reading the manual
> completely! I recommend to do this more than twice!
> It is useless to stumble around the configuration options and try to
> understand something.
>
> I'm angry  and I stop here - otherwise I would
> lose my way and my composure
>
>
>
> Thomas
>
> NWLI will get an improvement and a small fix in the next version.
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:18.10.2021 17:26
> Betreff:Re: [Assp-test] Does Message Score build from Bomb
> matches override a DKIMNP match??
> --
>
>
>
> Well that'll do it!  I incorrectly thought all these years that "no
> processing" actually meant NO processing.Looking back, the NWLI options
> have only been a choice for 11 years   I'm surprised I never caught
> this happening (correctly) before.
>
> I agree that it the gui doesn't say anywhere that noProcessing isn't
> processed by the penaltybox, but it's literally called NoProcessing, not
> less processing, etc, hence my confusion.
>
> GUI says:
> Mail solely to or from any of these addresses are proxied without
> processing. The envelope sender and recipients are checked. Like a more
> efficient version of Spam-Lovers & redlist combined. Accepts specific
> addresses (*u...@domain.com* ), user parts (user) or
> entire domains (@*domain.com* <http://domain.com/>). Wildcards are
> supported (fribo*@*domain.com* <http://domain.com/>). If you register TO
> addresses here, all recipients for a single mail must be marked as
> noprocessing to flag the mail as "noprocessing".
>
> My money says that 90%+ of the admins using ASSP have the same
> misperception as I do on this.  I suggest we add:
> Note: Messages flagged as "no processing" will not contribute to the
> corpus and will not be scored based on Bayesian/HMM detection.  However, by
> default, "Penalty Box" / Bomb scroing will still take place.  See !!!NWLI!!!
> option to override this default behavior.
>
>
>
>
>
>
>
> On Sun, Oct 17, 2021 at 4:25 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> >*(not sure why this line is in the log twice)*
>
> because the string was found twice
>
>
> >*msg rejected, even though no processing*
>
> I can't find anything in the manual, which states that 'noprocessing'
> mails are not processed by the penaltybox
>
> Scores are added by the bomb feature, because assp is configured to score
> noprocessing mails.
> bombReNP
> or
> =>NWLI is used (N at least for this regex)
>
> Thomas
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:17.10.2021 02:02
> Betreff:[Assp-test] Does Message Score build from Bomb matches
> override a DKIMNP match??
> --
>
>
>
> I had an inbound message rejected by ASSP, where the DKIM signature
> matched DKIMNP. I would have thought that if there's a DKIMNP match, that
> the message will just be passed and saved in discarded.
>
> Also, Senderbase is white for the network that it came from. so that
> should have reduced the score by a lot.
>
> There was a bombDataRE match. seemingly twice for the same line. and also
> in BombData.  I've got

Re: [Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[K Post]

I guess I should have written
. !!!N-W-L-I-!!! option to override this default behavior

On Mon, Oct 18, 2021 at 11:24 AM K Post  wrote:

> Well that'll do it!  I incorrectly thought all these years that "no
> processing" actually meant NO processing.Looking back, the NWLI options
> have only been a choice for 11 years   I'm surprised I never caught
> this happening (correctly) before.
>
> I agree that it the gui doesn't say anywhere that noProcessing isn't
> processed by the penaltybox, but it's literally called NoProcessing, not
> less processing, etc, hence my confusion.
>
> GUI says:
>
> Mail solely to or from any of these addresses are proxied without
> processing. The envelope sender and recipients are checked. Like a more
> efficient version of Spam-Lovers & redlist combined. Accepts specific
> addresses (u...@domain.com), user parts (user) or entire domains (@
> domain.com). Wildcards are supported (fribo*@domain.com). If you register
> TO addresses here, all recipients for a single mail must be marked as
> noprocessing to flag the mail as "noprocessing".
>
>
> My money says that 90%+ of the admins using ASSP have the same
> misperception as I do on this.  I suggest we add:
>
> Note: Messages flagged as "no processing" will not contribute to the
> corpus and will not be scored based on Bayesian/HMM detection.  However, by
> default, "Penalty Box" / Bomb scroing will still take place.  See !!!NWLI!!!
> option to override this default behavior.
>
>
>
>
>
>
>
>
> On Sun, Oct 17, 2021 at 4:25 AM Thomas Eckardt 
> wrote:
>
>> >*(not sure why this line is in the log twice)*
>>
>> because the string was found twice
>>
>>
>> >*msg rejected, even though no processing*
>>
>> I can't find anything in the manual, which states that 'noprocessing'
>> mails are not processed by the penaltybox
>>
>> Scores are added by the bomb feature, because assp is configured to score
>> noprocessing mails.
>> bombReNP
>> or
>> =>NWLI is used (N at least for this regex)
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:"K Post" 
>> An:"ASSP development mailing list" <
>> assp-test@lists.sourceforge.net>
>> Datum:17.10.2021 02:02
>> Betreff:[Assp-test] Does Message Score build from Bomb matches
>> override a DKIMNP match??
>> --
>>
>>
>>
>> I had an inbound message rejected by ASSP, where the DKIM signature
>> matched DKIMNP. I would have thought that if there's a DKIMNP match, that
>> the message will just be passed and saved in discarded.
>>
>> Also, Senderbase is white for the network that it came from. so that
>> should have reduced the score by a lot.
>>
>> There was a bombDataRE match. seemingly twice for the same line. and also
>> in BombData.  I've got Dear Friend, in both files by mistake, that'll be
>> fixed, but that pushed the score above 50, so it was rejected.  Shouldn't
>> DKIMNP override the rejection though?
>>
>> Here's the log, with my notes:
>>
>> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
>> > to: u...@ourchairty.org 
>> *DKIM-Signature
>> found*
>> Info: enhanced Originated IP detection ignored IP's: 102.xxx.yyy.85
>> (connected IP) , 10.11.74.34
>> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
>> > to: u...@ourchairty.org info:
>> found DKIM signature identity '@*bounce.TheirDomain.com*
>> <http://bounce.theirdomain.com/>'
>> @*bounce.TheirDomain.com* <http://bounce.theirdomain.com/> @
>> *bounce.TheirDomain.com* <http://bounce.theirdomain.com/>
>> ,u...@ourchairty.org matches *.TheirDomain.com in DKIMNPAddresses*
>> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
>> > to: u...@ourchairty.org [scoring]
>> DKIM signature verified-OK - header-passed - identity is: @
>> *bounce.TheirDomain.com* <http://bounce.theirdomain.com/> - sender
>> policy is: neutral - author policy s: neutral - *state changed to:
>> noprocessing*
>> Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
>> 'dear friend,' - weight is 0.5   *<-- we get a lot of Dear Friend,
>> garbage, so I have it in BombData with a 50% score*
>> Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
>> 'dear friend,' - weight is 0.5*(not sure why this line is in the log
>> twice)*
>> msg11890-19574 102.xxx.yyy.85 <

Re: [Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[K Post]

Well that'll do it!  I incorrectly thought all these years that "no
processing" actually meant NO processing.Looking back, the NWLI options
have only been a choice for 11 years   I'm surprised I never caught
this happening (correctly) before.

I agree that it the gui doesn't say anywhere that noProcessing isn't
processed by the penaltybox, but it's literally called NoProcessing, not
less processing, etc, hence my confusion.

GUI says:

Mail solely to or from any of these addresses are proxied without
processing. The envelope sender and recipients are checked. Like a more
efficient version of Spam-Lovers & redlist combined. Accepts specific
addresses (u...@domain.com), user parts (user) or entire domains (@
domain.com). Wildcards are supported (fribo*@domain.com). If you register
TO addresses here, all recipients for a single mail must be marked as
noprocessing to flag the mail as "noprocessing".


My money says that 90%+ of the admins using ASSP have the same
misperception as I do on this.  I suggest we add:

Note: Messages flagged as "no processing" will not contribute to the corpus
and will not be scored based on Bayesian/HMM detection.  However, by
default, "Penalty Box" / Bomb scroing will still take place.  See !!!NWLI!!!
option to override this default behavior.








On Sun, Oct 17, 2021 at 4:25 AM Thomas Eckardt 
wrote:

> >*(not sure why this line is in the log twice)*
>
> because the string was found twice
>
>
> >*msg rejected, even though no processing*
>
> I can't find anything in the manual, which states that 'noprocessing'
> mails are not processed by the penaltybox
>
> Scores are added by the bomb feature, because assp is configured to score
> noprocessing mails.
> bombReNP
> or
> =>NWLI is used (N at least for this regex)
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:17.10.2021 02:02
> Betreff:[Assp-test] Does Message Score build from Bomb matches
> override a DKIMNP match??
> --
>
>
>
> I had an inbound message rejected by ASSP, where the DKIM signature
> matched DKIMNP. I would have thought that if there's a DKIMNP match, that
> the message will just be passed and saved in discarded.
>
> Also, Senderbase is white for the network that it came from. so that
> should have reduced the score by a lot.
>
> There was a bombDataRE match. seemingly twice for the same line. and also
> in BombData.  I've got Dear Friend, in both files by mistake, that'll be
> fixed, but that pushed the score above 50, so it was rejected.  Shouldn't
> DKIMNP override the rejection though?
>
> Here's the log, with my notes:
>
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org *DKIM-Signature
> found*
> Info: enhanced Originated IP detection ignored IP's: 102.xxx.yyy.85
> (connected IP) , 10.11.74.34
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org info:
> found DKIM signature identity '@*bounce.TheirDomain.com*
> <http://bounce.theirdomain.com/>'
> @*bounce.TheirDomain.com* <http://bounce.theirdomain.com/> @
> *bounce.TheirDomain.com* <http://bounce.theirdomain.com/>
> ,u...@ourchairty.org matches *.TheirDomain.com in DKIMNPAddresses*
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org [scoring]
> DKIM signature verified-OK - header-passed - identity is: @
> *bounce.TheirDomain.com* <http://bounce.theirdomain.com/> - sender policy
> is: neutral - author policy s: neutral - *state changed to: noprocessing*
> Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
> 'dear friend,' - weight is 0.5   *<-- we get a lot of Dear Friend,
> garbage, so I have it in BombData with a 50% score*
> Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
> 'dear friend,' - weight is 0.5*(not sure why this line is in the log
> twice)*
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org spambomb
> Regex: bombDataRe 'PB 18: for Dear Friend,'
> msg11890-19574 [BombData] 102.xxx.yyy.85 <
> *bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org [scoring]
> (BombData 'Dear Friend,')
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.theirdomain.com*
> > to: u...@ourchairty.org
> Message-Score: added 18 for Regex: bombDataRe 'PB 18: for Dear Friend,'
> BombData: 'Dear Friend,', total score for this message is now 18
> msg11890-19574 102.xxx.yyy.85 <*bounce_ab...@bounce.their

[Assp-test] Does Message Score build from Bomb matches override a DKIMNP match??

[K Post]

I had an inbound message rejected by ASSP, where the DKIM signature matched
DKIMNP. I would have thought that if there's a DKIMNP match, that the
message will just be passed and saved in discarded.

Also, Senderbase is white for the network that it came from. so that should
have reduced the score by a lot.

There was a bombDataRE match. seemingly twice for the same line. and also
in BombData.  I've got Dear Friend, in both files by mistake, that'll be
fixed, but that pushed the score above 50, so it was rejected.  Shouldn't
DKIMNP override the rejection though?

Here's the log, with my notes:

msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org *DKIM-Signature found*
Info: enhanced Originated IP detection ignored IP's: 102.xxx.yyy.85
(connected IP) , 10.11.74.34
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org info: found DKIM signature identity '@
bounce.TheirDomain.com'
@bounce.TheirDomain.com @bounce.TheirDomain.com,u...@ourchairty.org
matches *.TheirDomain.com
in DKIMNPAddresses*
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org [scoring] DKIM signature verified-OK - header-passed -
identity is: @bounce.TheirDomain.com - sender policy is: neutral - author
policy s: neutral - *state changed to: noprocessing*
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
'dear friend,' - weight is 0.5   *<-- we get a lot of Dear Friend, garbage,
so I have it in BombData with a 50% score*
Info: weighted regex (bombDataRe) result found for 'Dear Friend,' - with
'dear friend,' - weight is 0.5*(not sure why this line is in the log
twice)*
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org spambomb Regex: bombDataRe 'PB 18: for Dear Friend,'
msg11890-19574 [BombData] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [scoring]
(BombData 'Dear Friend,')
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org Message-Score: added 18 for Regex: bombDataRe 'PB 18:
for Dear Friend,' BombData: 'Dear Friend,', total score for this message is
now 18
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org spambomb Regex: bombRe 'PB 35: for Dear Friend'
msg11890-19574 [BombData][bombRe] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [scoring]
(bombRe 'Dear Friend')
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org Message-Score: added 35 for Regex: bombRe 'PB 35: for
Dear Friend' bombRe: 'Dear Friend', total score for this message is now 53
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org deleting spamming safelisted tuplet: (102.xxx.yyy.0,
bounce.TheirDomain.com) age: 1s
msg11890-19574 [MessageLimit] 102.xxx.yyy.85 <
bounce_ab...@bounce.theirdomain.com> to: u...@ourchairty.org [spam
found] (*MessageScore
53, limit 50*) [Our  Newsletter October 15th 2021] ->
messages/discarded/Our__Newsletter_October_15th_2021--254778.txt;
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org* [SMTP Error] 554 5.7.1* [PE] rejected msg [PR]
[msg11890-19574 212EA668]  *<-- msg rejected, even though no processing*
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org info: PB-IP-Score for '102.xxx.yyy.0' is 53, added 53
in this session
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org finished message - received DATA size: 138.82 kByte -
sent DATA size: 0 Byte
msg11890-19574 102.xxx.yyy.85  to:
u...@ourchairty.org disconnected: session:212EA668 102.xxx.yyy.85 -
processing time 2 seconds
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21287

[K Post]

Very shortly after startup, I received:

 BerkeleyDB-ERROR: in start rebuildAddCorrections - syntax error at (eval
679) line 5, near "$main::$BDBerrLog "

Global symbol "$BDBerrLog" requires explicit package name (did you forget
to declare "my $BDBerrLog"?) at (eval 679) line 5.

 - BDB:

On Fri, Oct 15, 2021 at 12:51 PM K Post  wrote:

> Again, thanks.
> So are you recommending that $BDBerrLog stay at the default of 0 under
> normal circumstances, and only be changed to 1 if there appears to be
> something awry with one or more BDB actions?
>
> I've never seen anything written to any of the BDBError.txt files, but I
> certainly could have missed errors there - it's just not something I've
> monitored.  I made the mistake of assuming, yes, assuming that errors would
> also go to maillog.txt - but as you pointed out, it's the BDB driver that
> would erroring, not ASSP.   Is there a way / does it make sense / have you
> considered having ASSP use some magic to trap any BDB error or warning that
> may occur?
>
> On Fri, Oct 15, 2021 at 5:35 AM Thomas Eckardt 
> wrote:
>
>> >If msvcrt is being used, would 512 open files ever not be enough?
>>
>> No, not in every case. This depends on the configuration and the workload.
>>
>> > $winSetMaxIO_DLL and $winSetMaxIO I can understand how to set them
>> for my set up.
>>
>> don't change them
>>
>> >Do you know if the Strawberry Perl installations-DUSE_PERLIO?
>>
>> yes it is  ...
>>
>> > I tried looking it up, but I'm coming up empty
>> .. # Notice: PERLIO (perl compiled with -DUSE_PERLIO - *check with
>> :>perl -V*)
>>
>>  :>perl -V
>> or
>> read perl/lib/Config_heavy.pl
>>
>>
>> >With BDBErrLog set to 0, I assume that any error with BDB files would
>> still be spit out to the maillog.txt file so we can be alerted that
>> something's wrong?
>>
>> assume ?? . Read the perl POD for BekeleyDB.pm and the oracle
>> documentation for BerkeleyDB.
>> Who would need BDB-ENV -errfile if such errors could be catched elsewhere
>> easely?
>>
>> After (e.g.) a HASH %bar is tied to : memory, file, orderedtie,
>> BerkeleyDB, ODBC, ADO or any native RDBM
>> (oracle,db2,mysql,mariadb,mssql,postgre .) - a simple call like
>>
>> $bar{$foo}
>>
>> accesses totaly different code (the driver). It is impossible to catch
>> all possible errors for all cases for all tied mechanism, after such a
>> call, to write them to maillog.txt. ASSP tries to do its best to catch as
>> much of the errors as possible and to recover from error conditions
>> automatically.
>> But errors may occure at software layers, which can't be accessed by assp.
>> ASSP catches all errors at init-time of BerkeleyDB (and recovers if
>> possible). If there occure errors at runtime for BerkeleyDB, someone can
>> enable 'BDBErrLog' to get the runtime errors recorded.
>>
>> Thomas
>>
>>
>>
>> Von:"K Post" 
>> An:"ASSP development mailing list" <
>> assp-test@lists.sourceforge.net>
>> Datum:14.10.2021 20:25
>> Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
>> build 21287
>> --
>>
>>
>>
>> Whew you've been busy! Thank you.
>>
>>- If msvcrt is being used, would 512 open files ever not be enough?
>>I feel like I was getting the file issues when many links to BDB-error.txt
>>files were getting stuck open, so exceeding 512, but that was ultimately
>>because of my stupid griplist directory misconfiguration combined with bad
>>TLS early talkers.  I guess I'm trying to understand why
>>$winSetMaxIO_DLL and $winSetMaxIO hidden params were necessary, so I can
>>understand how to set them for my set up.
>>- Do you know if the Strawberry Perl installations at
>>*https://strawberryperl.com/releases.html*
>><https://strawberryperl.com/releases.html> compiled with DUSE_PERLIO?
>>  I tried looking it up, but I'm coming up empty
>>
>> With BDBErrLog set to 0, I assume that any error with BDB files would
>> still be spit out to the maillog.txt file so we can be alerted that
>> something's wrong?
>>
>>
>>
>> On Thu, Oct 14, 2021 at 9:52 AM Thomas Eckardt <
>> *thomas.ecka...@thockar.com* > wrote:
>> Hi all,
>>
>> fixed in assp 2.6.6 *SPAM-Evaporator* build 21287:
>>
>> - If a folder was defined for the parameter 'griplist' (e.g.
>> grip/griplist) and 

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21287

[K Post]

Again, thanks.
So are you recommending that $BDBerrLog stay at the default of 0 under
normal circumstances, and only be changed to 1 if there appears to be
something awry with one or more BDB actions?

I've never seen anything written to any of the BDBError.txt files, but I
certainly could have missed errors there - it's just not something I've
monitored.  I made the mistake of assuming, yes, assuming that errors would
also go to maillog.txt - but as you pointed out, it's the BDB driver that
would erroring, not ASSP.   Is there a way / does it make sense / have you
considered having ASSP use some magic to trap any BDB error or warning that
may occur?

On Fri, Oct 15, 2021 at 5:35 AM Thomas Eckardt 
wrote:

> >If msvcrt is being used, would 512 open files ever not be enough?
>
> No, not in every case. This depends on the configuration and the workload.
>
> > $winSetMaxIO_DLL and $winSetMaxIO I can understand how to set them
> for my set up.
>
> don't change them
>
> >Do you know if the Strawberry Perl installations-DUSE_PERLIO?
>
> yes it is  ...
>
> > I tried looking it up, but I'm coming up empty
> .. # Notice: PERLIO (perl compiled with -DUSE_PERLIO - *check with
> :>perl -V*)
>
>  :>perl -V
> or
> read perl/lib/Config_heavy.pl
>
>
> >With BDBErrLog set to 0, I assume that any error with BDB files would
> still be spit out to the maillog.txt file so we can be alerted that
> something's wrong?
>
> assume ?? . Read the perl POD for BekeleyDB.pm and the oracle
> documentation for BerkeleyDB.
> Who would need BDB-ENV -errfile if such errors could be catched elsewhere
> easely?
>
> After (e.g.) a HASH %bar is tied to : memory, file, orderedtie,
> BerkeleyDB, ODBC, ADO or any native RDBM
> (oracle,db2,mysql,mariadb,mssql,postgre .) - a simple call like
>
> $bar{$foo}
>
> accesses totaly different code (the driver). It is impossible to catch all
> possible errors for all cases for all tied mechanism, after such a call, to
> write them to maillog.txt. ASSP tries to do its best to catch as much of
> the errors as possible and to recover from error conditions automatically.
> But errors may occure at software layers, which can't be accessed by assp.
> ASSP catches all errors at init-time of BerkeleyDB (and recovers if
> possible). If there occure errors at runtime for BerkeleyDB, someone can
> enable 'BDBErrLog' to get the runtime errors recorded.
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:14.10.2021 20:25
> Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
> build 21287
> --
>
>
>
> Whew you've been busy! Thank you.
>
>- If msvcrt is being used, would 512 open files ever not be enough?  I
>feel like I was getting the file issues when many links to BDB-error.txt
>files were getting stuck open, so exceeding 512, but that was ultimately
>because of my stupid griplist directory misconfiguration combined with bad
>TLS early talkers.  I guess I'm trying to understand why
>$winSetMaxIO_DLL and $winSetMaxIO hidden params were necessary, so I can
>understand how to set them for my set up.
>- Do you know if the Strawberry Perl installations at
>*https://strawberryperl.com/releases.html*
><https://strawberryperl.com/releases.html> compiled with DUSE_PERLIO?
>  I tried looking it up, but I'm coming up empty
>
> With BDBErrLog set to 0, I assume that any error with BDB files would
> still be spit out to the maillog.txt file so we can be alerted that
> something's wrong?
>
>
>
> On Thu, Oct 14, 2021 at 9:52 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21287:
>
> - If a folder was defined for the parameter 'griplist' (e.g.
> grip/griplist) and this folder was not extisting, all griplist functions
> were not working.
>   If a folder is now defined, it is created by assp.
>
> - If 'ConfigChangeSchedule' was used to change a hidden configuration
> parameter, only the main thread (not any worker) was aware of the change.
>
> - If a mail subject contained a questionmark '?' in its text and the
> subject header line was encoded 'Quoted Printable' and the questionmark was
> not right MIME encoded
>   (instead it was written as '?') all internal functions related to the
> mail subject were not working correctly
>
>
> changed:
>
> - BerkeleyDB error logs (BDB-error.txt) are no longer permanently created
> and locked
>   Instead there is a new hidden parameter 'BDBerrLog', which ca

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21287

[K Post]

Whew you've been busy! Thank you.


   - If msvcrt is being used, would 512 open files ever not be enough?  I
   feel like I was getting the file issues when many links to BDB-error.txt
   files were getting stuck open, so exceeding 512, but that was ultimately
   because of my stupid griplist directory misconfiguration combined with bad
   TLS early talkers.  I guess I'm trying to understand why
   $winSetMaxIO_DLL and $winSetMaxIO hidden params were necessary, so I can
   understand how to set them for my set up.

   -
   - Do you know if the Strawberry Perl installations at
   https://strawberryperl.com/releases.html compiled with DUSE_PERLIO?I
   tried looking it up, but I'm coming up empty

   - With BDBErrLog set to 0, I assume that any error with BDB files would
   still be spit out to the maillog.txt file so we can be alerted that
   something's wrong?



On Thu, Oct 14, 2021 at 9:52 AM Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21287:
>
> - If a folder was defined for the parameter 'griplist' (e.g.
> grip/griplist) and this folder was not extisting, all griplist functions
> were not working.
>   If a folder is now defined, it is created by assp.
>
> - If 'ConfigChangeSchedule' was used to change a hidden configuration
> parameter, only the main thread (not any worker) was aware of the change.
>
> - If a mail subject contained a questionmark '?' in its text and the
> subject header line was encoded 'Quoted Printable' and the questionmark was
> not right MIME encoded
>   (instead it was written as '?') all internal functions related to the
> mail subject were not working correctly
>
>
> changed:
>
> - BerkeleyDB error logs (BDB-error.txt) are no longer permanently created
> and locked
>   Instead there is a new hidden parameter 'BDBerrLog', which can be set to
> 1 to monitor BDB-problems.
>
> our $BDBerrLog = 0; # (0/1) log BerkeleyDB errors in the related BDB-ENV
> -errfile .../BDB-error.txt (default = 0)
>
> - The GUI-help text for 'noGriplistUpload', 'noGriplistDownload' and
> 'gripValencePB' are updated - griplist functions are not changed
>
>
>
> added:
> - If windows systems are running out of available open file descriptors
> and the used perl installation is not compiled using the -DUSE_PERLIO
> switch,
>   the following parameters can be used to increase the available file
> descriptors for the assp process
>
> our $winSetMaxIO_DLL = 'msvcrt'; # the name of the microsoft
> C-runtime-library used by perl and/or perl-modules (Win32 only !!!) -
> default is msvcrt
>  # If your perl uses (is compiled
> against) any other msvcrtXXX (for example: msvcrt160 or msvcrt100) - change
> this value, if
>  # you want to set the maximum
> open files limit in the msvcrtXXX.
>  # This value is ONLY used for the
> below purpose ($winSetMaxIO), it has no other effect !
>
> our $winSetMaxIO = 0;# (0/1/ 512 * 2**N) set the
> maximum open files limit (Win32 only !!!) in ($winSetMaxIO_DLL) msvcrt.dll
> (_getmaxstdio , _setmaxstdio)
>  # 0 - use the default setting in
> msvcrt.dll (normaly set to 512)
>  # 1 - find the maximum allowed
> value between 512 and 8192 and set it
>  # 512 * 2**N - try to set the
> value as high as possible up to the given maximum (min 512 , max 8192, in
> 512 * 2**N [N=0..4])
>  #  if the defined value
> is less than the current maximum, the setting will not be changed
>  # Notice: PERLIO (perl compiled
> with -DUSE_PERLIO - check with :>perl -V) may define a different max open
> file limit for its
>  # IO's (defaults to 2048
> because PERLIO_MAX_REFCOUNTABLE_FD=2048)
>  # - this limit is not
> affected by this value
>
>
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Too many open files - Windows

[K Post]

Friends,
The crashing was ALL caused by a griplist database location
misconfiguration.  I had a bad path in the config, which made the berkeley
DB module leave too many connections to the griplist dbd-error.txt file
open, ultimately crashing ASSP.
Lesson learned: don't be a dope and have a bad path to the griplist!!!
Thank you, as always, to Thomas for helping figure this one out!!


On Wed, Oct 6, 2021 at 1:35 PM K Post  wrote:

> Looking way better Thomas. Thank you.
>
> Connected: session:23D15F00 154.21.28.20:60788 > (assp internal ip):25 >
> (smtp internal ip):25
> 154.21.28.20 warning: got an unexpected TLSv1_2 handshake
> Client-Helo-Frame of version (3.3) from IP '154.21.28.20' at local IP
> '(assp internal ip)' and Port '25' - the connection will be closed
> 154.21.28.20 Message-Score: added 25 (etValencePB) for EarlyTalker, total
> score for this message is now 25
> 154.21.28.20 info: PB-IP-Score for '154.21.28.0' is 50, added 25 in this
> session
> 154.21.28.20 disconnected: session:23D15F00 154.21.28.20 - processing time
> 0 seconds
> Connected: session:223748B0 154.21.28.20:60791 > (assp internal ip):25 >
> (smtp internal ip):25
> Error: Worker_3 accept_SSL to client 154.21.28.20 denied - the client
> failed before on SSL/TLS
> Error: Worker_3 accept_SSL to client 154.21.28.20 denied - the client
> failed before on SSL/TLS
> Connected: session:297ED5F8 154.21.28.20:60795 > (assp internal ip):25 >
> (smtp internal ip):25
> 154.21.28.20 disconnected: session:223748B0 154.21.28.20 - processing time
> 1 seconds
>
>
> This group of pesky servers (seems like a big range that keeps trying to
> connect) keeps throwing SSL errors, but ASSP is now handling the errors
> much more gracefully.  No more bad file descriptor errors!!!
>
> It seems like a decent sized SMTP farm, I assume spamming. * Do you think
> I should add the IP block to noTLSIP?   *Will that force them not to use
> SSL or are they just sending a handshake way too early for that to work?
>
>
>
>
>
>
>
>
>
>
> On Wed, Oct 6, 2021 at 10:08 AM Thomas Eckardt 
> wrote:
>
>> try https://sourceforge.net/p/assp/svn/HEAD/tree/assp2/trunk/test/
>>
>> tell me if and how it works for you
>>
>> Thomas
>>
>>
>>
>>
>> Von:"K Post" 
>> An:"ASSP development mailing list" <
>> assp-test@lists.sourceforge.net>
>> Datum:05.10.2021 16:30
>> Betreff:Re: [Assp-test] Too many open files - Windows
>> --
>>
>>
>>
>> I had originally started a new thread on the below, but I wonder if stuck
>> open sockets might be the cause of the Too many open files.  Bad SSL
>> connections seem to be failing to close "close failed on
>> IO::Socket::SSL=GLOB(0x2b7a5cd0) : Bad file descriptor"  That error happens
>> a lot when a SMTP server is persistent, even when we kick it for previous
>> failed SSL attempts.
>>
>> I'm aware that 21277 offers "Improved error handling in case a client or
>> server connects to the default SMTP-listener (25) using SSL."
>>
>> I saw:
>> got an unexpected TLSv1_2 handshake Client-Helo-Frame of version (3.3)
>> from IP '154.21.28.74' at local IP '(my ip here)' and Port '25' - this
>> frame is ignored
>> and in the 21277 release thread, you said that's the new code throwing
>> the warning
>>
>> I want to make sure something isn't awry though.  It might just be
>> coincidental, but with 21277, there's a couple smtp servers throwing
>> warnings now and then close errors.  Maybe this was always a problem with
>> bad SMTP servers, just not warned about before.  Most of the IP's I've seen
>> have a poor Senderbase reputation, but aren't blacklisted.  I've seen a lot
>> from one particular IP, but there are others.Poor Reputation for the
>> whole block of servers:
>> *https://talosintelligence.com/reputation_center/lookup?search=154.21.114.200*
>> <https://talosintelligence.com/reputation_center/lookup?search=154.21.114.200>
>>
>>
>> Here's an excerpt from the log, which complains about Bad file descriptor
>> and sockets that can't close..  Could I have something misconfigured??
>> Could the new SSL error handling be causing this, leaving things open, and
>> then causing the Too many open files problem??  Is this a really bad
>> behaving set of SMTP servers that ASSP is having trouble with and leaving
>> things open?  Or maybe it's completely unrelated to the too many open files
>> problem?  The below excerpt is all in the period of 2 seconds.  And I see
>> the same ki

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

Friends,
The crashing was ALL caused by a griplist database location
misconfiguration.  I had a bad path in the config, which made the berkeley
DB module leave too many connections to the griplist dbd-error.txt file
open, ultimately crashing ASSP.
Lesson learned: don't be a dope and have a bad path to the griplist!!!
Thank you, as always, to Thomas for helping figure this one out!!



On Fri, Oct 8, 2021 at 8:34 PM K Post  wrote:

> So this is odd running handle on perl.exe gives me stuff I'd expect
> (though I don't know what it all is)
>
>78: File  (RW-)
> C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.2213_none_de6ea00a534da176
>   18C: File  (R-D)   C:\Windows\System32\en-US\KernelBase.dll.mui
>   1B0: File  (RW-)
> d:\StrawberryPerl\perl\site\lib\Net\DNS\Resolver\Base.pm
>   280: File  (RW-)   d:\assp
>   2DC: File  (R-D)   C:\Windows\System32\en-US\mswsock.dll.mui
>   678: File  (RW-)   L:\ASSP Logs\maillog.txt   <-- link of d:\assp\logs
> to another drive
>   67C: File  (RW-)   L:\ASSP Logs\bmaillog.txt  <-- link of d:\assp\logs
> to another drive
>   704: Section   \BaseNamedObjects\__ComCatalogCache__
>   718: Section   \BaseNamedObjects\__ComCatalogCache__
>   71C: Section   \RPC Control\DSECFAC
>   7D8: File  (RW-)   d:\assp\pid
>
> and then about *1300* additional identical entries (with different
> handles), that are:
>
> xxx: *File  (RW-)   d:\assp\tmpDB\Griplist\BDB-error.txt*
>
> And those 1300 idential lines is with ASSP trucking along quite nicely.
> No errors in the 15 hours or so since it crashed last.  I can't imagine
> that 1300 open handles to DBD-error.txt is normal though.   And I have no
> idea what it shows when it's crashing.  I'm not sure how to find that out
> either, as once I'm alerted to SMTP being down, it's already too late -
> ASSP has crashed.
>
> DBD-error.txt is 0 bytes.  In the Griplist folder.  It appears to have
> been "modified" about a minute ago, but still 0bytes.
>
>  I also see:
> __db.001 888kb
> __db.002 120kb
> __db.003 648kb
>
> I don't believe that I use the griplist.  noGriplistUpload and
> noGriplistDownload are both checked.  Due to our charity privacy policy :(
>
> I just restarted the ASSP service in Windows and see 10 of the
> griplist\DBD-error.txt handles.  Is >>that<< normal?  I temporarily
> reverted to 21218 (I happened to have that old version on hand) and seems
> to have 9 of the griplist\DBD-error.txt handles, so having a bunch doesn't
> seem to be a new problem, if it's a problem at all.
>
> I feel like some loop happens when there TLS errors with the new versions
> though that somehow gets more of these griplist\dbd-error.txt handels to
> open until there no more file handles left
>
>
>
>
>
>
>
>
>
> On Fri, Oct 8, 2021 at 7:27 PM K Post  wrote:
>
>> That's funny Bob, I was just looking at Sysinternals to see if there was
>> such a tool.  Also looking to see if there's anything like this:
>>
>> https://stackoverflow.com/questions/8845949/how-to-find-open-global-filehandles-in-a-perl-program
>> that would work for Windows.
>>
>>
>>
>> On Fri, Oct 8, 2021 at 1:56 PM Robert K Coffman Jr. -Info From Data Corp.
>>  wrote:
>>
>>> I'm curious what handle (Sysinternals tool) says when you hit the file
>>> limit...
>>>
>>> - Bob
>>>
>>> On 10/8/2021 10:47 AM, K Post wrote:
>>> > And a similar thing just happened again with 21280.  ASSP gets to the
>>> > point where it can't open any more files, griplist can't be opened,
>>> and
>>> > it goes into a shutdown process.
>>>
>>>
>>>
>>> ___
>>> Assp-test mailing list
>>> Assp-test@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>>
>>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Multiple topics: Griplist vs GripList, stuck open handles to dbd-error on windows, more.

[K Post]

(our last 2 messages crossed).
"So, if assp tries (and retries for ever) to use a BerkeleyDB - but the
folder not extsts - and BerkeleyDB.pm does for what ever reason not close
files/filehandles/... - this will lead in to a 'too many opened files'
error."
YES! YES! YES!  Explains exactly how I was crashing.  I'm going to add that
comment on the 21280 thread in case anyone else is following there.

On Mon, Oct 11, 2021 at 9:28 AM Thomas Eckardt 
wrote:

> The count of handles seems not to be the problem!
> But, there is a limit for available filedescrptors per process on windows
> (msvcrt.dll) - 2048 !
>
> I made a test with the following code
>
> #
> use Tie::File;
> our %k;
> for (0...1) {
> $k{$_} = [];
> }
>
> for (sort {$a <=> $b} keys %k) {
>tie @{$k{$_}}, 'Tie::File', "c:/tmp/test/$_";
>push @{$k{$_}}, "$_";
>print "$_\n";
> }
> 
>
>
> this creates 2044 files (not the expected 10001). 2044 + STDOUT + STDIN +
> STDERR + script = 2048
>
> So, if assp tries (and retries for ever) to use a BerkeleyDB - but the
> folder not extsts - and BerkeleyDB.pm does for what ever reason not close
> files/filehandles/... - this will lead in to a 'too many opened files'
> error.
>
>
>
>
> Thomas
>
>
>
>
>
> Von:"Thomas Eckardt" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:11.10.2021 09:58
> Betreff:Re: [Assp-test] Multiple topics: Griplist vs GripList,
> stuck open handles to dbd-error on windows, more.
> --
>
>
>
> 1) - this will be fixed
>
> 2) - not default settings needs to be checked by the admin
>
> 3) - BDB-error.txt files are used to record BerkeleyDB errors - a handle
> is keeped opened by each thread for each BDB-environment (db) as long as
> the database-env is opened by the thread
>  most times such handles are opened until the thread is stopped (assp ends)
>
> 4)
> a) yes
>
> b) yes
>
> c)
> Scoring is done using gripValencePB and the resulting message/ip - score
> is calculated as follows:
>
> if the grip value is < 0.3 : -int(((0.3 - grip value) / 0.3) *
> gripValencePB)
> if the grip value is > 0.7 : int(((grip value - 0.7) / 0.3) *
> gripValencePB)
> grip values between 0.3 and 0.7 are ignored.
>
>
> windows handles (IMHO):
>
> system-max handles : ~ 2**24 (>16.000.000)
> process/thread max handles : ~ 10.000 (configurable in the registry -
> hex(2710))
>
> use Testlimit.exe / Testlimit64.exe -h
>
> How ever - the c-library used by the process sets the handle (and other)
> limits! If a valid but not-default-system c-lib is found in the PATH, it
> will be used (with there internal limit settings).
>
> assp never uses more than 2000 handles (typical less than 1000) on any
> system watched by me
> most handles (~ 1.000.000) are used permanently by mysqld on windows
>
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:09.10.2021 16:21
> Betreff:[Assp-test] Multiple topics: Griplist vs GripList, stuck
> open handles to dbd-error on windows, more.
> --
>
>
>
> Several related items here:
>
>1. Bug? Rebuild process still uploading griplist, even if disabled,
>due possibly to case error in code.
>2. ASSP not checking for valid griplist, if an invalid folder name is
>entered
>3. On windows, Rebuild process leaving \tmpDB\rebuildDB\BDB-error.txt
>handle open  (wider issue with DBD-error.txt files getting stuck open on
>Windows?)
>4. Griplist clarification request
>
>
> 1)  I've got noGriplistUpload and noGriplistDownload both checked in the
> GUI.  But I noticed that ,at the end of my rebuild log, it's still doing
> the upload
>
> Uploading Griplist via Direct Connection
>
>
> *rebuildspamdb.pm* <http://rebuildspamdb.pm/> has
> return if $main::noGripListUpload;
> before the upload happens, but I think there's a case mistake there.
>
> I believe it should be Griplist, with a lower case L.
>
>
> 2) In trying to figure out my ASSP on Windows crashes due to too many open
> files, I've started using Sysinternals Handle utility to look at open
> handles.  Yesterday I saw 1300+ handles open to
> d:\assp\tmpDB\Griplist\BDB-error.txt
>
> I believe that had something to do with me having an invalid griplist
> database name, "d/griplist" but the d folder didn't exist on my new server
> config!!  My fault there for 

Re: [Assp-test] Multiple topics: Griplist vs GripList, stuck open handles to dbd-error on windows, more.

[K Post]

) / 0.3) *
> gripValencePB)
>  if the grip value is > 0.7 : int(((grip value - 0.7) / 0.3) *
> gripValencePB)
>  grip values between 0.3 and 0.7 are ignored.
>
>
> windows handles (IMHO):
>
> system-max handles : ~ 2**24 (>16.000.000)
> process/thread max handles : ~ 10.000 (configurable in the registry -
> hex(2710))
>
> use Testlimit.exe / Testlimit64.exe -h
>
> How ever - the c-library used by the process sets the handle (and other)
> limits! If a valid but not-default-system c-lib is found in the PATH, it
> will be used (with there internal limit settings).
>
> assp never uses more than 2000 handles (typical less than 1000) on any
> system watched by me
> most handles (~ 1.000.000) are used permanently by mysqld on windows
>
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:09.10.2021 16:21
> Betreff:[Assp-test] Multiple topics: Griplist vs GripList, stuck
> open handles to dbd-error on windows, more.
> --
>
>
>
> Several related items here:
>
>1. Bug? Rebuild process still uploading griplist, even if disabled,
>due possibly to case error in code.
>2. ASSP not checking for valid griplist, if an invalid folder name is
>entered
>3. On windows, Rebuild process
>leaving \tmpDB\rebuildDB\BDB-error.txt handle open  (wider issue with
>DBD-error.txt files getting stuck open on Windows?)
>4. Griplist clarification request
>
>
> 1)  I've got noGriplistUpload and noGriplistDownload both checked in the
> GUI.  But I noticed that ,at the end of my rebuild log, it's still doing
> the upload
>
> Uploading Griplist via Direct Connection
>
>
> *rebuildspamdb.pm* <http://rebuildspamdb.pm/> has
> return if $main::noGripListUpload;
> before the upload happens, but I think there's a case mistake there.
>
> I believe it should be Griplist, with a lower case L.
>
>
> 2) In trying to figure out my ASSP on Windows crashes due to too many open
> files, I've started using Sysinternals Handle utility to look at open
> handles.  Yesterday I saw 1300+ handles open
> to d:\assp\tmpDB\Griplist\BDB-error.txt
>
> I believe that had something to do with me having an invalid griplist
> database name, "d/griplist" but the d folder didn't exist on my new server
> config!!  My fault there for sure, not sure how d/ prefixed it, but it
> appears ASSP didn't check to see that the folder existed.  Shame on me for
> missing my config file, but it might be good for ASSP to warn or create the
> folder if it doesn't exist.
>
> I cleared out the griplist database file to test and restarted. I don't
> see any more open griplist\dbd-error.txt handles, even though I'm getting
> many many bad SMTP servers connecting with early TLS.  Good.  I'll probably
> put the griplist database back (obviously with a valid filename!!)See
> #3
>
>
> 3) Windows (both 2012 and 2019) might not be closing Berkely error files
> correctly in general.
>
> When the new version started logging the new SSL errors, I >believe<
> that's when it started trying to access the non-existent "d/" folder config
> error in m assp.cfg.  Every time an early TLS line was caught, BerkeleyDB
> would keep the error file open.  Strangely, the error file was 0kb.
>
> Now that I've cleared out the griplist entry, I don't get those
> /tmpdb/griplist/dbd-error.txt open handles.  However, after my rebuild over
> night, I see and open handle to:
> tmpDB\rebuildDB\BDB-error.txt
> about 9+ hours after rebuild.
>
> Is that normal?  Might Windows not be closing handles correctly in general?
>
> 4) While I'm at it, could some clarification be provided as to the
> function of the griplist?  (and please correct me if anything I say here is
> incorrect!!)  I've searched like crazy over the last couple of days, but
> can't find the answers I'm looking for.
>
> The griplist is an ip scoring database correct?  I know it was originally
> called the grey-ip-list, 15+ years ago, but then greylisting became common
> language for delaying, so the original grayiplist started being called the
> "griplist" to avoid confusion.
>
> Note: the gui still says:
> GreyIPlist Database (griplist)
> The file with the current Grey-IP-List database -- make this blank if you
> don't use it.
> If I'm understanding griplist correctly, I think the gui should be
> reworded.  Maybe some explanation added too??
>
> There's also the optional upload and download concept of the griplist.
> This appears to send the local griplist to sourceforge, it gets process

Re: [Assp-test] Multiple topics: Griplist vs GripList, stuck open handles to dbd-error on windows, more.

[K Post]

And I'm noticing that each time rebuildspamdb runs, at least one more (it
>think< it may be two every time) handle to
\tmpDB\rebuildDB\BDB-error.txt is left open.

On Sat, Oct 9, 2021 at 4:35 PM K Post  wrote:

> answering part of one of my own questions - I hope correctly.
>
>
>- *And in the end, how is a match on the griplist scored?  Let's say
>an IP is on the griplist as being a really bad IP.  What score does a
>message get?  Is that configurable? *
>
> gripValancePB (default of 5) is added or subtracted from the message
> score.  If the grip value is more than .7 the score gets added, if less
> than .3, the score gets subtracted.
>
> Under noGriplistUpload it says:
>
> Check this to disable the Griplist upload. *The Griplist contains IPs and
> their value between 0 and 1, lower is less spammy, higher is more spammy.
> This value is called the grip value.*
>
>
> *I feel like that description is in the wrong place.  *My suggestion:
> move the bolded part of the GUI entry above to the griplist entry.  Or at
> least also have it there.  If you've got an entry in griplist, you're using
> the griplist, so I believe that's where the explanation should go, instead
> of in the section where you decide to share and use a shared griplist.
> (I still can't figure out if you do share and download the griplist it
> that download replaces your local griplist or what)
>
>
>
>
> On Sat, Oct 9, 2021 at 10:19 AM K Post  wrote:
>
>> Several related items here:
>>
>>1. Bug? Rebuild process still uploading griplist, even if disabled,
>>due possibly to case error in code.
>>
>>2. ASSP not checking for valid griplist, if an invalid folder name is
>>entered
>>
>>3. On windows, Rebuild process
>>leaving \tmpDB\rebuildDB\BDB-error.txt handle open  (wider issue with
>>DBD-error.txt files getting stuck open on Windows?)
>>
>>4. Griplist clarification request
>>
>>
>> 1)  I've got noGriplistUpload and noGriplistDownload both checked in the
>> GUI.  But I noticed that ,at the end of my rebuild log, it's still doing
>> the upload
>>
>> Uploading Griplist via Direct Connection
>>
>>
>> rebuildspamdb.pm has
>>
>> return if $main::noGripListUpload;
>>
>> before the upload happens, but I think there's a case mistake there.
>>
>>
>> I believe it should be Griplist, with a lower case L.
>>
>>
>> 2) In trying to figure out my ASSP on Windows crashes due to too many
>> open files, I've started using Sysinternals Handle utility to look at open
>> handles.  Yesterday I saw 1300+ handles open
>> to d:\assp\tmpDB\Griplist\BDB-error.txt
>>
>> I believe that had something to do with me having an invalid griplist
>> database name, "d/griplist" but the d folder didn't exist on my new server
>> config!!  My fault there for sure, not sure how d/ prefixed it, but it
>> appears ASSP didn't check to see that the folder existed.  Shame on me for
>> missing my config file, but it might be good for ASSP to warn or create the
>> folder if it doesn't exist.
>>
>> I cleared out the griplist database file to test and restarted. I don't
>> see any more open griplist\dbd-error.txt handles, even though I'm getting
>> many many bad SMTP servers connecting with early TLS.  Good.  I'll probably
>> put the griplist database back (obviously with a valid filename!!)See
>> #3
>>
>>
>> 3) Windows (both 2012 and 2019) might not be closing Berkely error files
>> correctly in general.
>>
>> When the new version started logging the new SSL errors, I >believe<
>> that's when it started trying to access the non-existent "d/" folder config
>> error in m assp.cfg.  Every time an early TLS line was caught, BerkeleyDB
>> would keep the error file open.  Strangely, the error file was 0kb.
>>
>> Now that I've cleared out the griplist entry, I don't get those
>> /tmpdb/griplist/dbd-error.txt open handles.  However, after my rebuild over
>> night, I see and open handle to:
>>
>> tmpDB\rebuildDB\BDB-error.txt
>>
>> about 9+ hours after rebuild.
>>
>> Is that normal?  Might Windows not be closing handles correctly
>> in general?
>>
>> 4) While I'm at it, could some clarification be provided as to the
>> function of the griplist?  (and please correct me if anything I say here is
>> incorrect!!)  I've searched like crazy over the last couple of days, but
>> can't find the answers I'm looking for.
>>
>> The griplist is an ip scoring database correct?  I know it w

Re: [Assp-test] Multiple topics: Griplist vs GripList, stuck open handles to dbd-error on windows, more.

[K Post]

answering part of one of my own questions - I hope correctly.


   - *And in the end, how is a match on the griplist scored?  Let's say an
   IP is on the griplist as being a really bad IP.  What score does a message
   get?  Is that configurable? *

gripValancePB (default of 5) is added or subtracted from the message
score.  If the grip value is more than .7 the score gets added, if less
than .3, the score gets subtracted.

Under noGriplistUpload it says:

Check this to disable the Griplist upload. *The Griplist contains IPs and
their value between 0 and 1, lower is less spammy, higher is more spammy.
This value is called the grip value.*


*I feel like that description is in the wrong place.  *My suggestion: move
the bolded part of the GUI entry above to the griplist entry.  Or at least
also have it there.  If you've got an entry in griplist, you're using the
griplist, so I believe that's where the explanation should go, instead of
in the section where you decide to share and use a shared griplist.
(I still can't figure out if you do share and download the griplist it that
download replaces your local griplist or what)




On Sat, Oct 9, 2021 at 10:19 AM K Post  wrote:

> Several related items here:
>
>1. Bug? Rebuild process still uploading griplist, even if disabled,
>due possibly to case error in code.
>
>2. ASSP not checking for valid griplist, if an invalid folder name is
>entered
>
>3. On windows, Rebuild process
>leaving \tmpDB\rebuildDB\BDB-error.txt handle open  (wider issue with
>DBD-error.txt files getting stuck open on Windows?)
>
>4. Griplist clarification request
>
>
> 1)  I've got noGriplistUpload and noGriplistDownload both checked in the
> GUI.  But I noticed that ,at the end of my rebuild log, it's still doing
> the upload
>
> Uploading Griplist via Direct Connection
>
>
> rebuildspamdb.pm has
>
> return if $main::noGripListUpload;
>
> before the upload happens, but I think there's a case mistake there.
>
>
> I believe it should be Griplist, with a lower case L.
>
>
> 2) In trying to figure out my ASSP on Windows crashes due to too many open
> files, I've started using Sysinternals Handle utility to look at open
> handles.  Yesterday I saw 1300+ handles open
> to d:\assp\tmpDB\Griplist\BDB-error.txt
>
> I believe that had something to do with me having an invalid griplist
> database name, "d/griplist" but the d folder didn't exist on my new server
> config!!  My fault there for sure, not sure how d/ prefixed it, but it
> appears ASSP didn't check to see that the folder existed.  Shame on me for
> missing my config file, but it might be good for ASSP to warn or create the
> folder if it doesn't exist.
>
> I cleared out the griplist database file to test and restarted. I don't
> see any more open griplist\dbd-error.txt handles, even though I'm getting
> many many bad SMTP servers connecting with early TLS.  Good.  I'll probably
> put the griplist database back (obviously with a valid filename!!)See
> #3
>
>
> 3) Windows (both 2012 and 2019) might not be closing Berkely error files
> correctly in general.
>
> When the new version started logging the new SSL errors, I >believe<
> that's when it started trying to access the non-existent "d/" folder config
> error in m assp.cfg.  Every time an early TLS line was caught, BerkeleyDB
> would keep the error file open.  Strangely, the error file was 0kb.
>
> Now that I've cleared out the griplist entry, I don't get those
> /tmpdb/griplist/dbd-error.txt open handles.  However, after my rebuild over
> night, I see and open handle to:
>
> tmpDB\rebuildDB\BDB-error.txt
>
> about 9+ hours after rebuild.
>
> Is that normal?  Might Windows not be closing handles correctly in general?
>
> 4) While I'm at it, could some clarification be provided as to the
> function of the griplist?  (and please correct me if anything I say here is
> incorrect!!)  I've searched like crazy over the last couple of days, but
> can't find the answers I'm looking for.
>
> The griplist is an ip scoring database correct?  I know it was originally
> called the grey-ip-list, 15+ years ago, but then greylisting became common
> language for delaying, so the original grayiplist started being called the
> "griplist" to avoid confusion.
>
> Note: the gui still says:
>
> GreyIPlist Database (griplist)
> The file with the current Grey-IP-List database -- make this blank if you
> don't use it.
>
> If I'm understanding griplist correctly, I think the gui should be
> reworded.  Maybe some explanation added too??
>
> There's also the optional upload and download concept of the griplist.
> This appears to send the local griplist to sourceforge, i

[Assp-test] Multiple topics: Griplist vs GripList, stuck open handles to dbd-error on windows, more.

[K Post]

Several related items here:

   1. Bug? Rebuild process still uploading griplist, even if disabled, due
   possibly to case error in code.

   2. ASSP not checking for valid griplist, if an invalid folder name is
   entered

   3. On windows, Rebuild process
   leaving \tmpDB\rebuildDB\BDB-error.txt handle open  (wider issue with
   DBD-error.txt files getting stuck open on Windows?)

   4. Griplist clarification request


1)  I've got noGriplistUpload and noGriplistDownload both checked in the
GUI.  But I noticed that ,at the end of my rebuild log, it's still doing
the upload

Uploading Griplist via Direct Connection


rebuildspamdb.pm has

return if $main::noGripListUpload;

before the upload happens, but I think there's a case mistake there.


I believe it should be Griplist, with a lower case L.


2) In trying to figure out my ASSP on Windows crashes due to too many open
files, I've started using Sysinternals Handle utility to look at open
handles.  Yesterday I saw 1300+ handles open
to d:\assp\tmpDB\Griplist\BDB-error.txt

I believe that had something to do with me having an invalid griplist
database name, "d/griplist" but the d folder didn't exist on my new server
config!!  My fault there for sure, not sure how d/ prefixed it, but it
appears ASSP didn't check to see that the folder existed.  Shame on me for
missing my config file, but it might be good for ASSP to warn or create the
folder if it doesn't exist.

I cleared out the griplist database file to test and restarted. I don't see
any more open griplist\dbd-error.txt handles, even though I'm getting many
many bad SMTP servers connecting with early TLS.  Good.  I'll probably put
the griplist database back (obviously with a valid filename!!)See #3


3) Windows (both 2012 and 2019) might not be closing Berkely error files
correctly in general.

When the new version started logging the new SSL errors, I >believe< that's
when it started trying to access the non-existent "d/" folder config error
in m assp.cfg.  Every time an early TLS line was caught, BerkeleyDB would
keep the error file open.  Strangely, the error file was 0kb.

Now that I've cleared out the griplist entry, I don't get those
/tmpdb/griplist/dbd-error.txt open handles.  However, after my rebuild over
night, I see and open handle to:

tmpDB\rebuildDB\BDB-error.txt

about 9+ hours after rebuild.

Is that normal?  Might Windows not be closing handles correctly in general?

4) While I'm at it, could some clarification be provided as to the function
of the griplist?  (and please correct me if anything I say here is
incorrect!!)  I've searched like crazy over the last couple of days, but
can't find the answers I'm looking for.

The griplist is an ip scoring database correct?  I know it was originally
called the grey-ip-list, 15+ years ago, but then greylisting became common
language for delaying, so the original grayiplist started being called the
"griplist" to avoid confusion.

Note: the gui still says:

GreyIPlist Database (griplist)
The file with the current Grey-IP-List database -- make this blank if you
don't use it.

If I'm understanding griplist correctly, I think the gui should be
reworded.  Maybe some explanation added too??

There's also the optional upload and download concept of the griplist.
This appears to send the local griplist to sourceforge, it gets processed
by whatever you've got running on the backend, and then I can (also
optionally) download another griplist which is based on all ASSP user
data.  If you don't upload, you can't download, and that's fair - you need
to contribute to benefit from the group.

The charity that I work for has a pretty poorly thought out privacy policy
that requires me to jump through all kinds of hoops when sharing >any<
information.  It's frustrating for sure. That means I cannot upload our
griplist without petitioning an internal committee.  I'm thinking I want to
do that, but need to fully understand the griplist first.


   - If we don't upload/download, the griplist is stlil maintained locally,
   just only with my data right?
   - If I get approval to share the ip data and download the griplist, that
   downloaded griplist is merged with my local griplist?
   - And in the end, how is a match on the griplist scored?  Let's say an
   IP is on the griplist as being a really bad IP.  What score does a message
   get?  Is that configurable?


As always, thank you
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

So this is odd running handle on perl.exe gives me stuff I'd expect
(though I don't know what it all is)

   78: File  (RW-)
C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.2213_none_de6ea00a534da176
  18C: File  (R-D)   C:\Windows\System32\en-US\KernelBase.dll.mui
  1B0: File  (RW-)
d:\StrawberryPerl\perl\site\lib\Net\DNS\Resolver\Base.pm
  280: File  (RW-)   d:\assp
  2DC: File  (R-D)   C:\Windows\System32\en-US\mswsock.dll.mui
  678: File  (RW-)   L:\ASSP Logs\maillog.txt   <-- link of d:\assp\logs to
another drive
  67C: File  (RW-)   L:\ASSP Logs\bmaillog.txt  <-- link of d:\assp\logs to
another drive
  704: Section   \BaseNamedObjects\__ComCatalogCache__
  718: Section   \BaseNamedObjects\__ComCatalogCache__
  71C: Section   \RPC Control\DSECFAC
  7D8: File  (RW-)   d:\assp\pid

and then about *1300* additional identical entries (with different
handles), that are:

xxx: *File  (RW-)   d:\assp\tmpDB\Griplist\BDB-error.txt*

And those 1300 idential lines is with ASSP trucking along quite nicely.  No
errors in the 15 hours or so since it crashed last.  I can't imagine that
1300 open handles to DBD-error.txt is normal though.   And I have no idea
what it shows when it's crashing.  I'm not sure how to find that out
either, as once I'm alerted to SMTP being down, it's already too late -
ASSP has crashed.

DBD-error.txt is 0 bytes.  In the Griplist folder.  It appears to have been
"modified" about a minute ago, but still 0bytes.

 I also see:
__db.001 888kb
__db.002 120kb
__db.003 648kb

I don't believe that I use the griplist.  noGriplistUpload and
noGriplistDownload are both checked.  Due to our charity privacy policy :(

I just restarted the ASSP service in Windows and see 10 of the
griplist\DBD-error.txt handles.  Is >>that<< normal?  I temporarily
reverted to 21218 (I happened to have that old version on hand) and seems
to have 9 of the griplist\DBD-error.txt handles, so having a bunch doesn't
seem to be a new problem, if it's a problem at all.

I feel like some loop happens when there TLS errors with the new versions
though that somehow gets more of these griplist\dbd-error.txt handels to
open until there no more file handles left









On Fri, Oct 8, 2021 at 7:27 PM K Post  wrote:

> That's funny Bob, I was just looking at Sysinternals to see if there was
> such a tool.  Also looking to see if there's anything like this:
>
> https://stackoverflow.com/questions/8845949/how-to-find-open-global-filehandles-in-a-perl-program
> that would work for Windows.
>
>
>
> On Fri, Oct 8, 2021 at 1:56 PM Robert K Coffman Jr. -Info From Data Corp. <
> bcoff...@infofromdata.com> wrote:
>
>> I'm curious what handle (Sysinternals tool) says when you hit the file
>> limit...
>>
>> - Bob
>>
>> On 10/8/2021 10:47 AM, K Post wrote:
>> > And a similar thing just happened again with 21280.  ASSP gets to the
>> > point where it can't open any more files, griplist can't be opened, and
>> > it goes into a shutdown process.
>>
>>
>>
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

That's funny Bob, I was just looking at Sysinternals to see if there was
such a tool.  Also looking to see if there's anything like this:
https://stackoverflow.com/questions/8845949/how-to-find-open-global-filehandles-in-a-perl-program
that would work for Windows.



On Fri, Oct 8, 2021 at 1:56 PM Robert K Coffman Jr. -Info From Data Corp. <
bcoff...@infofromdata.com> wrote:

> I'm curious what handle (Sysinternals tool) says when you hit the file
> limit...
>
> - Bob
>
> On 10/8/2021 10:47 AM, K Post wrote:
> > And a similar thing just happened again with 21280.  ASSP gets to the
> > point where it can't open any more files, griplist can't be opened, and
> > it goes into a shutdown process.
>
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

And a similar thing just happened again with 21280.  ASSP gets to the point
where it can't open any more files, griplist can't be opened, and it goes
into a shutdown process.

On Thu, Oct 7, 2021 at 2:50 PM K Post  wrote:

> Here's excerpts from my log.  Odd stuff and concerning.
>
>
> ASSP had been running for 24+ hours on 21279 without issue.  Saw a bunch
> of the   got an unexpected TLSv1_2 handshake Client-Helo-Frame warnings
> during that time, which ASSP seems to handle much better than with the
> previous version.  The transactions I saw are not dissimilar this this
> sequence which happened just before things headed south:
>
>
> Oct-07-21 11:52:03 Connected: session:AC2E2C40 212.102.59.230:53720 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:03 212.102.59.230 *warning: got an unexpected TLSv1_2
> handshake Client-Helo-Frame of version (3.3)* from IP '212.102.59.230' at
> local IP '(assp internal ip)' and Port '25' - the connection will be closed
> Oct-07-21 11:52:03 212.102.59.230 Message-Score: added 25 (etValencePB)
> for EarlyTalker, total score for this message is now 25
> Oct-07-21 11:52:03 212.102.59.230 info: PB-IP-Score for '212.102.59.0' is
> 200, added 25 in this session
> Oct-07-21 11:52:03 212.102.59.230 disconnected: session:AC2E2C40
> 212.102.59.230 - processing time 0 seconds
> Oct-07-21 11:52:03 Connected: session:71BAB808 212.102.59.230:53722 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:03 *Error: Worker_5 accept_SSL to client 212.102.59.230
> denied - the client failed before on SSL/TLS*
> Oct-07-21 11:52:03 Error: Worker_5 accept_SSL to client 212.102.59.230
> denied - the client failed before on SSL/TLS
> Oct-07-21 11:52:03 212.102.59.230 disconnected: session:71BAB808
> 212.102.59.230 - processing time 0 seconds
> Oct-07-21 11:52:03 Connected: session:4DB26EA0 212.102.59.230:53726 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:04 Connected: session:3050E3F0 212.102.59.230:53728 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:04 212.102.59.230 disconnected: session:4DB26EA0
> 212.102.59.230 - processing time 1 seconds
> Oct-07-21 11:52:05 Error: Worker_1 accept_SSL to client 212.102.59.230
> denied - the client failed before on SSL/TLS
> Oct-07-21 11:52:05 212.102.59.230 disconnected: session:3050E3F0
> 212.102.59.230 - processing time 1 seconds
> Oct-07-21 11:52:05 Error: Worker_1 accept_SSL to client 212.102.59.230
> denied - the client failed before on SSL/TLS
> Oct-07-21 11:52:07 Connected: session:71BAB808 52.207.41.187:45398 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:08 Connected: session:52815DB8 52.207.41.187:45442 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:08 52.207.41.187 disconnected: session:71BAB808
> 52.207.41.187 - processing time 1 seconds
> Oct-07-21 11:52:08 52.207.41.187 info: got STARTTLS request from
> 52.207.41.187
> Oct-07-21 11:52:09 52.207.41.187 disconnected: session:52815DB8
> 52.207.41.187 - processing time 1 seconds
> Oct-07-21 11:52:10 Connected: session:A91752E8 172.241.24.83:58278 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:10 172.241.24.83 disconnected: session:A91752E8
> 172.241.24.83 - processing time 0 seconds
> Oct-07-21 11:52:10 Connected: session:8B034AF0 172.241.24.83:58392 >
> (assp internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:11 172.241.24.83 info: got STARTTLS request from
> 172.241.24.83
> Oct-07-21 11:52:12 172.241.24.83 disconnected: session:8B034AF0
> 172.241.24.83 - processing time 2 seconds
> Oct-07-21 11:52:25 Connected: session:52C3B6D0 23.239.7.4:49106 > (assp
> internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:25 23.239.7.4 disconnected: session:52C3B6D0 23.239.7.4 -
> processing time 0 seconds
> Oct-07-21 11:52:25 Connected: session:874B2E68 23.239.7.4:49688 > (assp
> internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:25 23.239.7.4 info: got STARTTLS request from 23.239.7.4
> Oct-07-21 11:52:26 Connected: session:886EF9A0 192.82.209.81:9208 > (assp
> internal ip):25 > (smtp internal ip):25
> Oct-07-21 11:52:26 192.82.209.81 info: got STARTTLS request from
> 192.82.209.81
> Oct-07-21 11:52:26 23.239.7.4 disconnected: session:874B2E68 23.239.7.4 -
> processing time 1 seconds
>
>
> Then, a bunch of email is received normally, then this:
>
>
> Oct-07-21 11:54:01 179.26.113.249 info: injected STARTTLS request to (smtp
> internal ip)*<-- injected STARTTTLS request? Haven't noticed that
> before*
> Oct-07-21 11:54:02 msg25642-16992 179.26.113.249 
> Message-Score: added 5 (fiphValencePB) for Suspicious HELO - conta

Re: [Assp-test] Crashed ASSP with fat-fingered Regex mistake

[K Post]

Thanks for the fix in 21280!
" the definition of an invalid regular expression in 'NotifyRe' may caused
a crash of the assp process"

On Wed, Oct 6, 2021 at 1:53 PM K Post  wrote:

> Nothing urgent, but I managed to completely crash assp by entering an
> invalid regex.
>
> In trying to keep on eye on the annoying failed TLS smtp server pool, I
> intended to put a NotifyRE together like:
>
> connected: session:(.){8} 154\.21\.
>
>
> To start, I entered, I was going to first try
>
> Connected: session:(.){8}
>
> then add the start of the ip once I knew the always matching session
> connection would trigger the notice.
>
> Well, I carelessly entered
>
> Connected: session:(.){8)
>
> with the mistake of ending the test line with ) instead of }
>
> ASSP crashed immediately.
>
> Info: shutdown reason was: try restarting ASSP on exception
> ASSP finished work
>
>
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

/assp/assp.cfg.tmp to
d:/assp/assp.cfg - No such file or directory
Oct-07-21 12:00:03 Finished saving config
Oct-07-21 12:00:03 BerkeleyDB-ENV-ERROR Griplist: Cannot open file
d:/assp/tmpDB/Griplist/BDB-error.txt: Too many open files
at (eval 3745) line 1.
- BDB:BDB0062 Successful return: 0
Oct-07-21 12:00:03 Initializing shutdown sequence
Oct-07-21 12:00:03 Cleaning whitelist database finished: keys
before=161004, deleted=24
Oct-07-21 12:00:03 Consolidate whitelist database ...
Oct-07-21 12:00:03 Info: removing all SMTP and Proxy listeners
\Oct-07-21 12:00:03 Worker_6 finished
Oct-07-21 12:00:03 ClamAV Up
Oct-07-21 12:00:03 Waiting for all SMTP-Workers to be finished
Oct-07-21 12:00:03 Worker_2 finished

( one last message is received)

Oct-07-21 12:00:04 Worker_1 finished
Oct-07-21 12:00:04 SMTP Workers finished
Oct-07-21 12:00:04 Waiting for high Workers to be finished
Oct-07-21 12:00:05 Consolidate whitelist database finished: keys
before=203956, deleted=0
Oct-07-21 12:00:05 Worker_1 finished
Oct-07-21 12:00:05 Info: RebuildSpamdb Scheduler stopped
Oct-07-21 12:00:05 Worker_10001 finished
Oct-07-21 12:00:06 High workers finished work
Oct-07-21 12:00:06 Saving ldaplist
Oct-07-21 12:00:06 Info: saving Stats in file asspstats.sav
Oct-07-21 12:00:06 Info: saving ScoreStats in file asspscorestats.sav
Oct-07-21 12:00:06 Info: saving confidence graphical Stats in file
d:/assp/logs/confidenceGraphStats-2021-10.txt
Oct-07-21 12:00:06 Info: saving internal Caches in to folder
d:/assp/tmpDB/files
Oct-07-21 12:00:06 Closing all databases
Oct-07-21 12:00:06 Info: removing all WEB listeners
Oct-07-21 12:00:06 *Info: shutdown reason was: try restarting ASSP on
exception*
Oct-07-21 12:00:06 ASSP finished work

On Thu, Oct 7, 2021 at 1:53 PM K Post  wrote:

> Uh oh.   I haven't moved away from the test version you provided yesterday
> yet, but I just got a complete ASSP crash, which was preceded by:
>
> error: Worker_1 accept to client failed IO::Socket::INET=GLOB(0x881a0c40)
> (timeout: 2 s) : Too many open files
>
> error: Worker_2 accept to client failed IO::Socket::INET=GLOB(0x82c3f108)
> (timeout: 2 s) : Too many open files
>
> error: unable to close Socket IO::Socket::INET=GLOB(0xd872058) -  - Bad
> file descriptor
>
> ERROR:  no answering DNS-SERVER found    <--- all 3 of my internal
> dns servers are operational
>
> error: couldn't create server socket to (inernal smtp ip) --
> abortion connection
>
> error: Worker_2 accept to client failed IO::Socket::INET=GLOB(0x7e56c288)
> (timeout: 2 s) : Too many open files
>
>
> I was able to restart the ASSP service, but immediately started getting
> the same error messages as before.
>
> I've restarted Windows, working on getting it to start now, but I wanted
> to post ASAP just in case this is related to the test version / new version
> you just released.
>
>
>
>
>
> On Thu, Oct 7, 2021 at 12:05 PM Thomas Eckardt 
> wrote:
>
>> Hi all,
>>
>> fixed in assp 2.6.6 *SPAM-Evaporator* build 21280:
>>
>>
>> - if $fakeAUTHsuccess was set, the collected .eml files contained only
>> the X-Assp headers - not the spam mail data
>>
>> - build 21277 caused an error 'too many opened files' on windows, if
>> there were too many SSL-connection at a plain port (25)
>>
>> - the definition of an invalid regular expression in 'NotifyRe' may
>> caused a crash of the assp process
>>
>>
>>
>>
>> changed:
>>
>> - The default value for
>>
>> $ignoreEarlySSLClientHelo
>> # (0/1) 1 - unexpected early SSLv23/TLS handshake Client-Helo-Frames are
>> ignored , 0 - unexpected early SSLv23/TLS handshake Client-Helo-Frames are
>> NOT ignored and the connection will be closed
>>
>> is changed from 1 to 0 in assp.pl
>>
>> to recover the old setting, you may change assp.pl or you can set
>> $main::ignoreEarlySSLClientHelo = 1; in lib/CorrectASSPcfg.pm sub set{}
>> The setting '1' was used as default to ignore early SSL connections from
>> local clients.
>>
>>
>> - mails which are catched by 'fakeAUTHsuccess' are now counted for the
>> statistics in STATS:msgMaxErrors and SCORESTATS:MaxErrors
>>
>>
>> Thomas
>>
>> DISCLAIMER:
>> ***
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> ***
>>
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21280

[K Post]

Uh oh.   I haven't moved away from the test version you provided yesterday
yet, but I just got a complete ASSP crash, which was preceded by:

error: Worker_1 accept to client failed IO::Socket::INET=GLOB(0x881a0c40)
(timeout: 2 s) : Too many open files

error: Worker_2 accept to client failed IO::Socket::INET=GLOB(0x82c3f108)
(timeout: 2 s) : Too many open files

error: unable to close Socket IO::Socket::INET=GLOB(0xd872058) -  - Bad
file descriptor

ERROR:  no answering DNS-SERVER found    <--- all 3 of my internal
dns servers are operational

error: couldn't create server socket to (inernal smtp ip) --
abortion connection

error: Worker_2 accept to client failed IO::Socket::INET=GLOB(0x7e56c288)
(timeout: 2 s) : Too many open files


I was able to restart the ASSP service, but immediately started getting the
same error messages as before.

I've restarted Windows, working on getting it to start now, but I wanted to
post ASAP just in case this is related to the test version / new version
you just released.





On Thu, Oct 7, 2021 at 12:05 PM Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21280:
>
>
> - if $fakeAUTHsuccess was set, the collected .eml files contained only the
> X-Assp headers - not the spam mail data
>
> - build 21277 caused an error 'too many opened files' on windows, if there
> were too many SSL-connection at a plain port (25)
>
> - the definition of an invalid regular expression in 'NotifyRe' may caused
> a crash of the assp process
>
>
>
>
> changed:
>
> - The default value for
>
> $ignoreEarlySSLClientHelo
> # (0/1) 1 - unexpected early SSLv23/TLS handshake Client-Helo-Frames are
> ignored , 0 - unexpected early SSLv23/TLS handshake Client-Helo-Frames are
> NOT ignored and the connection will be closed
>
> is changed from 1 to 0 in assp.pl
>
> to recover the old setting, you may change assp.pl or you can set
> $main::ignoreEarlySSLClientHelo = 1; in lib/CorrectASSPcfg.pm sub set{}
> The setting '1' was used as default to ignore early SSL connections from
> local clients.
>
>
> - mails which are catched by 'fakeAUTHsuccess' are now counted for the
> statistics in STATS:msgMaxErrors and SCORESTATS:MaxErrors
>
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Crashed ASSP with fat-fingered Regex mistake

[K Post]

Nothing urgent, but I managed to completely crash assp by entering an
invalid regex.

In trying to keep on eye on the annoying failed TLS smtp server pool, I
intended to put a NotifyRE together like:

connected: session:(.){8} 154\.21\.


To start, I entered, I was going to first try

Connected: session:(.){8}

then add the start of the ip once I knew the always matching session
connection would trigger the notice.

Well, I carelessly entered

Connected: session:(.){8)

with the mistake of ending the test line with ) instead of }

ASSP crashed immediately.

Info: shutdown reason was: try restarting ASSP on exception
ASSP finished work
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Too many open files - Windows

[K Post]

Looking way better Thomas. Thank you.

Connected: session:23D15F00 154.21.28.20:60788 > (assp internal ip):25 >
(smtp internal ip):25
154.21.28.20 warning: got an unexpected TLSv1_2 handshake Client-Helo-Frame
of version (3.3) from IP '154.21.28.20' at local IP '(assp internal ip)'
and Port '25' - the connection will be closed
154.21.28.20 Message-Score: added 25 (etValencePB) for EarlyTalker, total
score for this message is now 25
154.21.28.20 info: PB-IP-Score for '154.21.28.0' is 50, added 25 in this
session
154.21.28.20 disconnected: session:23D15F00 154.21.28.20 - processing time
0 seconds
Connected: session:223748B0 154.21.28.20:60791 > (assp internal ip):25 >
(smtp internal ip):25
Error: Worker_3 accept_SSL to client 154.21.28.20 denied - the client
failed before on SSL/TLS
Error: Worker_3 accept_SSL to client 154.21.28.20 denied - the client
failed before on SSL/TLS
Connected: session:297ED5F8 154.21.28.20:60795 > (assp internal ip):25 >
(smtp internal ip):25
154.21.28.20 disconnected: session:223748B0 154.21.28.20 - processing time
1 seconds


This group of pesky servers (seems like a big range that keeps trying to
connect) keeps throwing SSL errors, but ASSP is now handling the errors
much more gracefully.  No more bad file descriptor errors!!!

It seems like a decent sized SMTP farm, I assume spamming. * Do you think I
should add the IP block to noTLSIP?   *Will that force them not to use SSL
or are they just sending a handshake way too early for that to work?










On Wed, Oct 6, 2021 at 10:08 AM Thomas Eckardt 
wrote:

> try https://sourceforge.net/p/assp/svn/HEAD/tree/assp2/trunk/test/
>
> tell me if and how it works for you
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:05.10.2021 16:30
> Betreff:Re: [Assp-test] Too many open files - Windows
> --
>
>
>
> I had originally started a new thread on the below, but I wonder if stuck
> open sockets might be the cause of the Too many open files.  Bad SSL
> connections seem to be failing to close "close failed on
> IO::Socket::SSL=GLOB(0x2b7a5cd0) : Bad file descriptor"  That error happens
> a lot when a SMTP server is persistent, even when we kick it for previous
> failed SSL attempts.
>
> I'm aware that 21277 offers "Improved error handling in case a client or
> server connects to the default SMTP-listener (25) using SSL."
>
> I saw:
> got an unexpected TLSv1_2 handshake Client-Helo-Frame of version (3.3)
> from IP '154.21.28.74' at local IP '(my ip here)' and Port '25' - this
> frame is ignored
> and in the 21277 release thread, you said that's the new code throwing the
> warning
>
> I want to make sure something isn't awry though.  It might just be
> coincidental, but with 21277, there's a couple smtp servers throwing
> warnings now and then close errors.  Maybe this was always a problem with
> bad SMTP servers, just not warned about before.  Most of the IP's I've seen
> have a poor Senderbase reputation, but aren't blacklisted.  I've seen a lot
> from one particular IP, but there are others.Poor Reputation for the
> whole block of servers:
> *https://talosintelligence.com/reputation_center/lookup?search=154.21.114.200*
> <https://talosintelligence.com/reputation_center/lookup?search=154.21.114.200>
>
>
> Here's an excerpt from the log, which complains about Bad file descriptor
> and sockets that can't close..  Could I have something misconfigured??
> Could the new SSL error handling be causing this, leaving things open, and
> then causing the Too many open files problem??  Is this a really bad
> behaving set of SMTP servers that ASSP is having trouble with and leaving
> things open?  Or maybe it's completely unrelated to the too many open files
> problem?  The below excerpt is all in the period of 2 seconds.  And I see
> the same kind of thing happening, especially from this IP, many times in an
> hour.
>
> Connected: session:4C6AD911 *154.21.114.200:60973*
> <http://154.21.114.200:60973/> > (assp internal ip):25 > (smtp internal
> ip):25
> 154.21.114.200 *warning: got an unexpected TLSv1_2 handshake
> Client-Helo-Frame of version (3.3) from IP '154.21.114.200' at local IP
> '(assp internal ip)' and Port '25' - this frame is ignored*
> Connected: session:23DBEBB0 *154.21.114.200:60975*
> <http://154.21.114.200:60975/> > (assp internal ip):25 > (smtp internal
> ip):25
> 154.21.114.200 disconnected: session:4C6AD911 154.21.114.200 - processing
> time 0 seconds
> Error: Worker_2 accept_SSL to client 154.21.114.200 denied - the client
> failed before on SSL/TLS  *<-- great, but then it doesn't seem to cl

Re: [Assp-test] Too many open files - Windows

[K Post]

I had originally started a new thread on the below, but I wonder if stuck
open sockets might be the cause of the Too many open files.  Bad SSL
connections seem to be failing to close "close failed on
IO::Socket::SSL=GLOB(0x2b7a5cd0) : Bad file descriptor"  That error happens
a lot when a SMTP server is persistent, even when we kick it for previous
failed SSL attempts.

I'm aware that 21277 offers "Improved error handling in case a client or
server connects to the default SMTP-listener (25) using SSL."

I saw:

got an unexpected TLSv1_2 handshake Client-Helo-Frame of version (3.3) from
IP '154.21.28.74' at local IP '(my ip here)' and Port '25' - this frame is
ignored

and in the 21277 release thread, you said that's the new code throwing the
warning

I want to make sure something isn't awry though.  It might just be
coincidental, but with 21277, there's a couple smtp servers throwing
warnings now and then close errors.  Maybe this was always a problem with
bad SMTP servers, just not warned about before.  Most of the IP's I've seen
have a poor Senderbase reputation, but aren't blacklisted.  I've seen a lot
from one particular IP, but there are others.Poor Reputation for the
whole block of servers:
https://talosintelligence.com/reputation_center/lookup?search=154.21.114.200


Here's an excerpt from the log, which complains about Bad file descriptor
and sockets that can't close..  Could I have something misconfigured??
Could the new SSL error handling be causing this, leaving things open, and
then causing the Too many open files problem??  Is this a really bad
behaving set of SMTP servers that ASSP is having trouble with and leaving
things open?  Or maybe it's completely unrelated to the too many open files
problem?  The below excerpt is all in the period of 2 seconds.  And I see
the same kind of thing happening, especially from this IP, many times in an
hour.

Connected: session:4C6AD911 154.21.114.200:60973 > (assp internal ip):25 >
(smtp internal ip):25
154.21.114.200 *warning: got an unexpected TLSv1_2 handshake
Client-Helo-Frame of version (3.3) from IP '154.21.114.200' at local IP
'(assp internal ip)' and Port '25' - this frame is ignored*
Connected: session:23DBEBB0 154.21.114.200:60975 > (assp internal ip):25 >
(smtp internal ip):25
154.21.114.200 disconnected: session:4C6AD911 154.21.114.200 - processing
time 0 seconds
Error: Worker_2 accept_SSL to client 154.21.114.200 denied - the client
failed before on SSL/TLS  *<-- great, but then it doesn't seem to close the
socket.*
Error: Worker_2 *close failed on IO::Socket::SSL=GLOB(0x5dfd18c8) : Bad
file descriptor*
Error: Worker_2 accept_SSL to client 154.21.114.200 denied - the client
failed before on SSL/TLS
Error: Worker_2 *close failed on IO::Socket::SSL=GLOB(0x5dfd18c8) : Bad
file descriptor*
Connected: session:70F3C0C8 154.21.114.200:60979 > (assp internal ip):25 >
(smtp internal ip):25
154.21.114.200 disconnected: session:23DBEBB0 154.21.114.200 - processing
time 1 seconds
Connected: session:2452D908 154.21.114.200:60982 > (assp internal ip):25 >
(smtp internal ip):25
154.21.114.200 disconnected: session:70F3C0C8 154.21.114.200 - processing
time 1 seconds
Error: Worker_2 accept_SSL to client 154.21.114.200 denied - the client
failed before on SSL/TLS
Error: Worker_2 *close failed on IO::Socket::SSL=GLOB(0x22463520) : Bad
file descriptor*
154.21.114.200 disconnected: session:2452D908 154.21.114.200 - processing
time 1 seconds
Error: Worker_2 accept_SSL to client 154.21.114.200 denied - the client
failed before on SSL/TLS
Error: Worker_2 *close failed on IO::Socket::SSL=GLOB(0x22463520) : Bad
file descriptor*
Info: successfully sent file messages/resendmail/n20232.txt to (smtp
internal ip):25 (smtpDestination)


There's repeated attempts from this single IP (and others that seem
unrelated with a similar problem).  Yes, I can block the IP, but I want to
ensure that 21277 hasn't introduced an issue that's leaving files open.

Here's another where 154.21.114.200 is connecting from the outside *to port
587*.  Odd that it would use that port instead of 25, but they're likely a
spammer trying to get around filtering.  Either way.  I get the SSL accept
attempt failed.  Is it timing out??

Connected: session:48196D90 154.21.114.200:65253 > (interal assp ip):587 >
(internal smtp ip):25
154.21.114.200 disconnected: session:48196D90 154.21.114.200 - processing
time 1 seconds
Error: Worker_4 accept_SSL to client 154.21.114.200 failed
IO::Socket::SSL=GLOB(0x2b79ffd0) (timeout: 5 s) : SSL accept attempt failed



On Tue, Oct 5, 2021 at 9:18 AM K Post  wrote:

> I've seen this a couple times now with 21277.  Windows.
>
> error: unable to close Socket IO::Socket::INET=GLOB(0x10fd4f70) -  - Bad
> file descriptor
>
>
> and
>
> Oct-05-21 08:51:09 *** (our internal smtp ip):25 didn't work, trying
> others... - Too many open files
> Oct-05-21 08:51:09 Error: couldn't create se

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21277

[K Post]

I have about 4million HMM records, startup was 2-3 minutes, but ASSP is on
Windows, in a VM, on a not so new (but new to the charity) host
Whatever SQL you simplified has it at a couple seconds now!  Terrific.

On the new SSL error/warnings, I'll open a thread to discuss in case it's
something more than just new warnings.  I'm seeing bad file descriptor log
lines.


On Tue, Oct 5, 2021 at 2:03 AM Thomas Eckardt 
wrote:

> >my very slow startup with the rebuild spamdb worker
>
> MySQL was blocking for a long time, because of a unnecessary complex
> SQL-statement for hmmdb and spamdb.
>
> In my case the delay was ~25 seconds for ~10 million database records.
>
> >Is that part of the improved error handling for SSL SMTP sessions over
> port 25?
>
> yes.
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:05.10.2021 01:27
> Betreff:Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
> build 21277
> --
>
>
>
> Thank you Thomas!!!
> This version solved my very slow startup with the rebuild spamdb worker.
>
> I am seeing some entries like:
> got an unexpected TLSv1_2 handshake Client-Helo-Frame of version (3.3)
> from IP '154.21.28.74' at local IP '(my ip here)' and Port '25' - this
> frame is ignored
>
> I don't believe I've seen those before this version.  Is that part of the
> improved error handling for SSL SMTP sessions over port 25?
>
>
>
> On Mon, Oct 4, 2021 at 4:21 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21277:
>
> - If a relational DB engine was used for hmmDB and/or SpamDB, the start of
> the worker 10001 (rebuildspamdb worker) has taken much more time (under
> certain conditions)
>   compared to the start time of all other workers
>
> - Improved error handling in case a client or server connects to the
> default SMTP-listener (25) using SSL.
>
> - The 'fakeAUTHsuccess' feature was not working like expected (caused by a
> fix for AUTH-error handling in assp 2.6.4 *SPAM-Evaporator* build 19284).
>
>
> added:
>
> - At the point in time the X-ASSP-..  headers were calculated, assp now
> calls '::modMyHeader($Con{fh})' (if this sub is defined).
>   This makes it possible to in place modify $Con{fh}->{myheader} to any
> special needs.
>   Keep in mind: Modifying SMTP-headers in a wrong manner may prevent mails
> from beeing transported!
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Too many open files - Windows

[K Post]

I've seen this a couple times now with 21277.  Windows.

error: unable to close Socket IO::Socket::INET=GLOB(0x10fd4f70) -  - Bad
file descriptor


and

Oct-05-21 08:51:09 *** (our internal smtp ip):25 didn't work, trying
others... - Too many open files
Oct-05-21 08:51:09 Error: couldn't create server socket to (our internal
smtp ip):25 -- aborting connection
Oct-05-21 08:51:09 [SMTP Status] 421  service
temporarily unavailable, closing transmission
Oct-05-21 08:51:09 xx.32.204.172 disconnected: session:98B71A10
xx.32.204.172 - processing time 0 seconds


I've restarted the machine, but if this happens again (and it likely will
being that I've seen it a couple times now), where can I look to help
determine what files are getting stuck open and the cause?

I should note that this is a relatively new Windows 2019 install.
Strawberry Perl 5.32.1.0.  MySQL 8.0.26.  All modules up to date.  Very
light load.  I didn't see this error until I started with 21277, but there
wasn't much traffic on previous builds in this particular installation.

Thanks
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator* build 21277

[K Post]

Thank you Thomas!!!
This version solved my very slow startup with the rebuild spamdb worker.

I am seeing some entries like:
got an unexpected TLSv1_2 handshake Client-Helo-Frame of version (3.3) from
IP '154.21.28.74' at local IP '(my ip here)' and Port '25' - this frame is
ignored

I don't believe I've seen those before this version.  Is that part of the
improved error handling for SSL SMTP sessions over port 25?



On Mon, Oct 4, 2021 at 4:21 AM Thomas Eckardt 
wrote:

> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21277:
>
> - If a relational DB engine was used for hmmDB and/or SpamDB, the start of
> the worker 10001 (rebuildspamdb worker) has taken much more time (under
> certain conditions)
>   compared to the start time of all other workers
>
> - Improved error handling in case a client or server connects to the
> default SMTP-listener (25) using SSL.
>
> - The 'fakeAUTHsuccess' feature was not working like expected (caused by a
> fix for AUTH-error handling in assp 2.6.4 *SPAM-Evaporator* build 19284).
>
>
> added:
>
> - At the point in time the X-ASSP-..  headers were calculated, assp now
> calls '::modMyHeader($Con{fh})' (if this sub is defined).
>   This makes it possible to in place modify $Con{fh}->{myheader} to any
> special needs.
>   Keep in mind: Modifying SMTP-headers in a wrong manner may prevent mails
> from beeing transported!
>
> Thomas
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs

[K Post]

Confirmed!  The new version starts up nice and quick

On Mon, Oct 4, 2021 at 4:56 AM Doug Lytle  wrote:

> On 10/3/21 7:53 PM, K Post wrote:
> > Thanks Doug.  How long does it take to start the rebuild SpamDB worker
> > though?  Mine also runs 11, and seemingly fine, it just takes a
> > long time to start.  I don't know why it starts before the GUI or
> > SMTP, but suspect that it needs to be running if it's going to track
> > mail for the rebuild as it arrives.
>
> Looks like Thomas may have fixed your startup problem,
>
> Doug
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs

[K Post]

Thanks Doug.  How long does it take to start the rebuild SpamDB worker
though?  Mine also runs 11, and seemingly fine, it just takes a long
time to start.  I don't know why it starts before the GUI or SMTP, but
suspect that it needs to be running if it's going to track mail for the
rebuild as it arrives.

On Fri, Oct 1, 2021 at 3:13 PM Doug Lytle  wrote:

> >>> I've been messing around (I don't think I broke anything), but at
> command line "starting rebuild SpamDB worker thread" sits for 3+ minutes.
>
> Kevin,
>
> Restarting my install shows
>
> Starting rebuild SpamDB worker thread [10001] - ThreadCycleTime is set to
> 30 microseconds
>
> I'm running under Devuan 3 Linux
>
> Doug
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs

[K Post]

I've been messing around (I don't think I broke anything), but at command
line "starting rebuild SpamDB worker thread" sits for 3+ minutes.  Is that
normal?  So I understand, why does the rebuild SpamDB worker thread start
at startup?  Does it do things that early where it doesn't make sense to
first have SMTP and the GUI operational first?

On Wed, Sep 29, 2021 at 2:33 PM K Post  wrote:

> Thank you Thomas!!  I will run through these steps this weekend and report
> back.
>
> On Wed, Sep 29, 2021 at 12:03 PM Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> - check all the setting for the lists and hashes to be set to 'DB:'
>> (using mysql)
>> - check that the mysql server is working correctly (if not assp will
>> failover to plain files)
>> - check all settings related to the rebuildspamdb
>> - shutdown assp
>> - clear the folder 'tmpDB'
>> - remove all 'griplist' related files from assp root folder
>> - check the file time of the assp.cfg (written at assp shutdown)
>> - check that the last line in assp.cfg is 'ConfigSavedOK:=1'
>>
>> - start assp at commandline
>> - wait 10 minutes , watch the log for normal maintenance
>> - stop assp
>> - start it as service
>> - wait 10 minutes
>> - start a rebuildspamdb task from the GUI or cmdqueue (in doubt enable
>> the debugging for the rebuild)
>> - wait until this task is finished
>> - stop assp
>>
>> - start assp from commandline - everythng should work like expected
>> - stop assp
>> - start the service
>>
>> Thomas
>>
>>
>> Von:"K Post" 
>> An:"ASSP development mailing list" <
>> assp-test@lists.sourceforge.net>
>> Datum:29.09.2021 16:41
>> Betreff:[Assp-test] Rebuild SpamDB worker thread at startup hangs
>> --
>>
>>
>>
>> If I restart my server or just the ASSP windows service, I've discovered
>> that ASSP generally will no longer fully start automatically as a service.
>>  I'm sure there's something misconfigured, corrupt, or broken on my
>> machine.  If the Windows server crashes or does an after hours update
>> restart, ASSP needs to be started from the command line  before it'll run
>> as a service, which presents a big problem.
>>
>> *The service does always run, but I cannot connect to the GUI nor to
>> SMTP.*  At start, it uses about 10% of processor as expected.  In task
>> manager, I see memory usage go ~300mb, ~600mb, then up to 984mb and stop.
>> CPU then goes down to 0.0% or 0.1% and it just hangs.  In testing, I've
>> left it for over an hour like this.  It just sits.  No SMTP, no GUI.  RAM
>> usage stays at 984mb.  Nothing in maillog, nothing in the Windows
>> system log that I can find.
>>
>> If I start from command line, ASSP will work, but there is a delay on the:
>> starting rebuild SpamDB worker thread
>> line, memory sits at 984mb, then after a two minutes or so, memory grows
>> a bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP
>> and the GUI *start to work*.  It'll run like this quite happily
>> forever.
>>
>> I want it running as a service though, so I'll ctrl+c, then start ASSP
>> from the services manager. Task manager will show it again pausing at
>> 984mb, but this time, it always continues and operates normally!  So the
>> temporary fix is to start on command line, ctrl+c, then start the service
>> normally.
>>
>> Everything is up to date, including all modules.  I'm using MySQL.
>> useDB4Rebuild is checked
>> RebuildEsesFileModel is unchecked, though I did try this when it first
>> became an option.  I turned it off because it used so much RAM and my
>> rebuild is already under 45 minutes.  Could there be something lingering
>> because of this?
>>
>> Can you suggest anything that I should be checking?
>> Is there something that starting ASSP from command line could be clearing
>> out or resolving that starting it as a service wouldn't?
>> Is there a debug switch that would capture what's happening at this very
>> early stage of startup?  There doesn't seem to be anything written to the
>> logs at all.
>>
>> I'm making do as is, but sure would like to fix this issue.
>>
>> Thank you.
>>
>>
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> 

Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs

[K Post]

Thank you Thomas!!  I will run through these steps this weekend and report
back.

On Wed, Sep 29, 2021 at 12:03 PM Thomas Eckardt 
wrote:

> - check all the setting for the lists and hashes to be set to 'DB:' (using
> mysql)
> - check that the mysql server is working correctly (if not assp will
> failover to plain files)
> - check all settings related to the rebuildspamdb
> - shutdown assp
> - clear the folder 'tmpDB'
> - remove all 'griplist' related files from assp root folder
> - check the file time of the assp.cfg (written at assp shutdown)
> - check that the last line in assp.cfg is 'ConfigSavedOK:=1'
>
> - start assp at commandline
> - wait 10 minutes , watch the log for normal maintenance
> - stop assp
> - start it as service
> - wait 10 minutes
> - start a rebuildspamdb task from the GUI or cmdqueue (in doubt enable the
> debugging for the rebuild)
> - wait until this task is finished
> - stop assp
>
> - start assp from commandline - everythng should work like expected
> - stop assp
> - start the service
>
> Thomas
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:29.09.2021 16:41
> Betreff:[Assp-test] Rebuild SpamDB worker thread at startup hangs
> --
>
>
>
> If I restart my server or just the ASSP windows service, I've discovered
> that ASSP generally will no longer fully start automatically as a service.
>  I'm sure there's something misconfigured, corrupt, or broken on my
> machine.  If the Windows server crashes or does an after hours update
> restart, ASSP needs to be started from the command line  before it'll run
> as a service, which presents a big problem.
>
> *The service does always run, but I cannot connect to the GUI nor to SMTP.*
> At start, it uses about 10% of processor as expected.  In task manager, I
> see memory usage go ~300mb, ~600mb, then up to 984mb and stop.  CPU then
> goes down to 0.0% or 0.1% and it just hangs.  In testing, I've left it for
> over an hour like this.  It just sits.  No SMTP, no GUI.  RAM usage stays
> at 984mb.  Nothing in maillog, nothing in the Windows system log that I can
> find.
>
> If I start from command line, ASSP will work, but there is a delay on the:
> starting rebuild SpamDB worker thread
> line, memory sits at 984mb, then after a two minutes or so, memory grows a
> bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP
> and the GUI *start to work*.  It'll run like this quite happily forever.
>
> I want it running as a service though, so I'll ctrl+c, then start ASSP
> from the services manager. Task manager will show it again pausing at
> 984mb, but this time, it always continues and operates normally!  So the
> temporary fix is to start on command line, ctrl+c, then start the service
> normally.
>
> Everything is up to date, including all modules.  I'm using MySQL.
> useDB4Rebuild is checked
> RebuildEsesFileModel is unchecked, though I did try this when it first
> became an option.  I turned it off because it used so much RAM and my
> rebuild is already under 45 minutes.  Could there be something lingering
> because of this?
>
> Can you suggest anything that I should be checking?
> Is there something that starting ASSP from command line could be clearing
> out or resolving that starting it as a service wouldn't?
> Is there a debug switch that would capture what's happening at this very
> early stage of startup?  There doesn't seem to be anything written to the
> logs at all.
>
> I'm making do as is, but sure would like to fix this issue.
>
> Thank you.
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Rebuild SpamDB worker thread at startup hangs

[K Post]

If I restart my server or just the ASSP windows service, I've discovered
that ASSP generally will no longer fully start automatically as a service.
 I'm sure there's something misconfigured, corrupt, or broken on my
machine.  If the Windows server crashes or does an after hours update
restart, ASSP needs to be started from the command line  before it'll run
as a service, which presents a big problem.

*The service does always run, but I cannot connect to the GUI nor to SMTP.*
At start, it uses about 10% of processor as expected.  In task manager, I
see memory usage go ~300mb, ~600mb, then up to 984mb and stop.  CPU then
goes down to 0.0% or 0.1% and it just hangs.  In testing, I've left it for
over an hour like this.  It just sits.  No SMTP, no GUI.  RAM usage stays
at 984mb.  Nothing in maillog, nothing in the Windows system log that I can
find.

If I start from command line, ASSP will work, but there is a delay on the:

starting rebuild SpamDB worker thread

line, memory sits at 984mb, then after a two minutes or so, memory grows a
bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP
and the GUI *start to work*.  It'll run like this quite happily forever.

I want it running as a service though, so I'll ctrl+c, then start ASSP from
the services manager. Task manager will show it again pausing at 984mb, but
this time, it always continues and operates normally!  So the temporary fix
is to start on command line, ctrl+c, then start the service normally.

Everything is up to date, including all modules.  I'm using MySQL.
useDB4Rebuild is checked
RebuildEsesFileModel is unchecked, though I did try this when it first
became an option.  I turned it off because it used so much RAM and my
rebuild is already under 45 minutes.  Could there be something lingering
because of this?

Can you suggest anything that I should be checking?
Is there something that starting ASSP from command line could be clearing
out or resolving that starting it as a service wouldn't?
Is there a debug switch that would capture what's happening at this very
early stage of startup?  There doesn't seem to be anything written to the
logs at all.

I'm making do as is, but sure would like to fix this issue.

Thank you.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Migration Tips - windows & mysql

[K Post]

Oh that's outstanding Leo.  I hadn't seen that option before.  Seems to be
exactly what I need!  Thank you.

On Thu, Sep 9, 2021 at 8:38 AM Leandro N. Castro - INSETEC Informática <
leandro.cas...@insetec.com.ar> wrote:

> Hi I solved this kind of situation reading in the GUI the option
> 'ExportMysqlDB' and before importing the configuration and databases in the
> new operating system.
>
> Good luck
>
> Bye.
>
>
>
> Leo.
>
>
>
> *De:* K Post [mailto:nntp.p...@gmail.com]
> *Enviado el:* miércoles, 08 de septiembre de 2021 17:41
> *Para:* ASSP development mailing list
> *Asunto:* [Assp-test] Migration Tips - windows & mysql
>
>
>
> Hi,
>
>
>
> I'm excited to have received a new (to us but still old) donated server
> for our charity.  I'd like to move the current ASSP installation away from
> Windows 2012 and install Windows 2019.  (linux isn't an option due to
> policy).
>
>
>
> I'm running MySQL on the current installation.  To migrate the ASSP
> installation, is there anything I need to be aware of?
>
>
>
> I was planning on creating a new Windows installation, getting mysql
> running, backing up the mysql database using phpmyadmin or something,
> importing that database into the new server, copying the full ASSP folder
> including logs and the message corpus.  I assume that once ASSP spins up,
> I'd need to check for any internal IP address references to ASSP that
> changed.
>
>
>
> Am I missing anything? Any suggestions?
>
>
>
> Thanks
>
> Ken.
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

I did restore ASSP.cfg after your previous email.  Your insight is always
appreciated.  I just found it strange that a full disk could cause the spam
folder to be moved to discarded.  Even if the assp.cfg file was corrupted,
how could the entire spam folder have been moved to be a subfolder of the
discared folder? If it only happened once, that could have just been a
careless drag and drop, but it's been at least twice.  It's not like a
destroyed config file would have spam go to a subfolder in discarded is
it?  I am really NOT worried, just curious.  I've only seen this after a
full disk, and that shouldn't happen again once I move the logs to a
separate disk.

I did do a compare of the re-restored ASSP.cfg file to the backup, and it
looked identical, but I restored anyway, just to be safe.

Thank you again.


On Thu, Sep 9, 2021 at 5:55 AM Thomas Eckardt 
wrote:

> >last ditch effort
> >free up space upon cleanup somehow
>
> assp does NOT care about disk space !
> (only the rebuild task shows some informational disk space values)
>
> >Strange that spam wound up in discarded though right?
>
> but if the disk is full, assp may be unable to write any new file and
> updated file (incl. the assp.cfg, stats, sync, tmp) to disk - which leads
> in to a destroyed configuration
>
> if configuration values are missing in the assp.cfg (possibly partly
> written), these values may be set to defaults at startup or recovered from
> a backup (*.bak)
> assp tries to do its best to recover a destroyed assp.cfg (in several
> ways), but a 'disk full' condition prevent this and the overall result in
> the assp folder is (may be) very unexpected
>
> release change backups of the assp.cfg are stored in the 'backup.config'
> folder - compare the last backup with your current config file
> *if you know*, *everything* in the assp folder is OK except the assp.cfg,
> configuration changes will fix every issue caused by the disk fault
>
> -
> On *Sun, Sep 5, 2021* at 4:02 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
>
> The state of the *assp folders and files* (also the assp.cfg !!!) may be
> *unexpected* after a 'disk full' condition happened.
>
> *I recommend to restore a known good state.*
>
> --
>
> So the problem would has been solved at Sun, Sep 5, 2021 - but at least at
> Mon, Sep 6, 2021 in the morning.
>
> Seems you think, you can manage this using your own strategy - good luck!
> But keep in mind: bad conditions in any file caused by the fault, can lead
> in to bad behavior or crashes of assp in future (possibly in months or in
> years, depends on the config, GUI actions, configuration changes, perl
> module changes ..)
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:08.09.2021 22:37
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> looking okay here now.  Strange that spam wound up in discarded
> though right?  Is that something that ASSP does as an emergency last ditch
> effort when a disk fills?  Move the old spam to discarded in an attempt to
> free up space upon cleanup somehow?
>
> On Sun, Sep 5, 2021 at 4:02 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> The state of the assp folders and files (also the assp.cfg !!!) may be
> unexpected after a 'disk full' condition happened.
>
> I recommend to restore a known good state.
>
> Thomas
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:04.09.2021 15:54
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> also, while I've got you  This isn't the first time a filling log has
> filled the data drive.
>
> A couple times, after moving the huge log, I've noticed that *my
> messages/spam corpus suddenly only has the files that hit after the log
> cleanup*.  I always later find that the spam folder has been *moved to
> the discarded folder* (as a spam subfolder).
>
> There is a chance that I carelessly dragged it there once, but multiple
> times seems too coincidental.   Does ASSP do this intentionally when a full
> drive is detected?  not-spam and the others are always intact, it's been
> the spam folder. It's easy to restore, but I need to remember that it
> happe

[Assp-test] Migration Tips - windows & mysql

[K Post]

Hi,

I'm excited to have received a new (to us but still old) donated server for
our charity.  I'd like to move the current ASSP installation away from
Windows 2012 and install Windows 2019.  (linux isn't an option due to
policy).

I'm running MySQL on the current installation.  To migrate the ASSP
installation, is there anything I need to be aware of?

I was planning on creating a new Windows installation, getting mysql
running, backing up the mysql database using phpmyadmin or something,
importing that database into the new server, copying the full ASSP folder
including logs and the message corpus.  I assume that once ASSP spins up,
I'd need to check for any internal IP address references to ASSP that
changed.

Am I missing anything? Any suggestions?

Thanks
Ken.
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

looking okay here now.  Strange that spam wound up in discarded
though right?  Is that something that ASSP does as an emergency last ditch
effort when a disk fills?  Move the old spam to discarded in an attempt to
free up space upon cleanup somehow?

On Sun, Sep 5, 2021 at 4:02 AM Thomas Eckardt 
wrote:

> The state of the assp folders and files (also the assp.cfg !!!) may be
> unexpected after a 'disk full' condition happened.
>
> I recommend to restore a known good state.
>
> Thomas
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:04.09.2021 15:54
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> also, while I've got you  This isn't the first time a filling log has
> filled the data drive.
>
> A couple times, after moving the huge log, I've noticed that *my
> messages/spam corpus suddenly only has the files that hit after the log
> cleanup*.  I always later find that the spam folder has been *moved to
> the discarded folder* (as a spam subfolder).
>
> There is a chance that I carelessly dragged it there once, but multiple
> times seems too coincidental.   Does ASSP do this intentionally when a full
> drive is detected?  not-spam and the others are always intact, it's been
> the spam folder. It's easy to restore, but I need to remember that it
> happened.  The ASSP service starts up quite happily with no files in spam,
> and it's generally only after seeing that the rebuild ran quicker than
> usual overnight that I realize there's virtually nothing in spam.
>
> Thanks again.
>
>
>
> On Sat, Sep 4, 2021 at 2:43 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21198:
>
> 
>
> - invalid UTF8 characters in a mail subject may caused SEGV errors in
> 'Win32::Unicode::Dir' on windows systems
>
>
> I hoped I had fixed it. New generated files should not cause the SEGV, but
> old filenames may cause it.
>
> set
> Regular Expression to Identify skipped Log Lines* (noLogLineRe)
> to
> signal SEGV in
>
> >I just don't know if putting a drive letter in the config is another
> option.
> No, this will not work. But links will work on windows.
>
> >I have everything on my C drive in c:\ASSP\logs.
> Is there any good reason to this at a production system???
>
> Thomas
>
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:03.09.2021 22:11
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> Thanks James.
> The same idea on Windows with a symbolic link should work fine.  I just
> don't know if putting a drive letter in the config is another option.
> My real concern is the SEGV fauls going nuts.
>
>
> On Fri, Sep 3, 2021 at 2:14 PM James Moe via Assp-test <
> *assp-test@lists.sourceforge.net* >
> wrote:
> On 2021-09-03 09:11, K Post wrote:
>
> > 1) Can the logfile configuration be set to use a different disk?
> >
>   This was relatively easy on linux. I do not know if Windows has a similar
> functionality. Basically the ASSP log directories are re-directed to
> another
> area with more space.
>
> ASSPDIR="/usr/local/bin/assp2";
>
> ASSPDATA="/data01/var/assp";
>
> assp_mount () {
>
> echo "Mounting assp volumes..."
>
> mount --bind ${ASSPDATA}/okmail ${ASSPDIR}/okmail
>
> mount --bind ${ASSPDATA}/discarded  ${ASSPDIR}/discarded
>
> mount --bind ${ASSPDATA}/quarantine ${ASSPDIR}/quarantine
>
> mount --bind ${ASSPDATA}/spam-yes   ${ASSPDIR}/spam-yes
>
> mount --bind ${ASSPDATA}/spam-not   ${ASSPDIR}/spam-not
>
> mount --bind ${ASSPDATA}/logs   ${ASSPDIR}/logs
>
> }
>
>
> --
> James Moe
> moe dot james at sohnen-moe dot com
> 520.743.3936
> Think.
>
>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
>
> DISCLAIMER:
> ***

Re: [Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

also, while I've got you  This isn't the first time a filling log has
filled the data drive.

A couple times, after moving the huge log, I've noticed that *my
messages/spam corpus suddenly only has the files that hit after the log
cleanup*.  I always later find that the spam folder has been *moved to the
discarded folder* (as a spam subfolder).

There is a chance that I carelessly dragged it there once, but multiple
times seems too coincidental.   Does ASSP do this intentionally when a full
drive is detected?  not-spam and the others are always intact, it's been
the spam folder. It's easy to restore, but I need to remember that it
happened.  The ASSP service starts up quite happily with no files in spam,
and it's generally only after seeing that the rebuild ran quicker than
usual overnight that I realize there's virtually nothing in spam.

Thanks again.



On Sat, Sep 4, 2021 at 2:43 AM Thomas Eckardt 
wrote:

> fixed in assp 2.6.6 *SPAM-Evaporator* build 21198:
>
> 
>
> - invalid UTF8 characters in a mail subject may caused SEGV errors in
> 'Win32::Unicode::Dir' on windows systems
>
>
> I hoped I had fixed it. New generated files should not cause the SEGV, but
> old filenames may cause it.
>
> set
> Regular Expression to Identify skipped Log Lines* (noLogLineRe)
> to
> signal SEGV in
>
> >I just don't know if putting a drive letter in the config is another
> option.
> No, this will not work. But links will work on windows.
>
> >I have everything on my C drive in c:\ASSP\logs.
> Is there any good reason to this at a production system???
>
> Thomas
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:03.09.2021 22:11
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> Thanks James.
> The same idea on Windows with a symbolic link should work fine.  I just
> don't know if putting a drive letter in the config is another option.
> My real concern is the SEGV fauls going nuts.
>
>
> On Fri, Sep 3, 2021 at 2:14 PM James Moe via Assp-test <
> *assp-test@lists.sourceforge.net* >
> wrote:
> On 2021-09-03 09:11, K Post wrote:
>
> > 1) Can the logfile configuration be set to use a different disk?
> >
>   This was relatively easy on linux. I do not know if Windows has a similar
> functionality. Basically the ASSP log directories are re-directed to
> another
> area with more space.
>
> ASSPDIR="/usr/local/bin/assp2";
>
> ASSPDATA="/data01/var/assp";
>
> assp_mount () {
>
> echo "Mounting assp volumes..."
>
> mount --bind ${ASSPDATA}/okmail ${ASSPDIR}/okmail
>
> mount --bind ${ASSPDATA}/discarded  ${ASSPDIR}/discarded
>
> mount --bind ${ASSPDATA}/quarantine ${ASSPDIR}/quarantine
>
> mount --bind ${ASSPDATA}/spam-yes   ${ASSPDIR}/spam-yes
>
> mount --bind ${ASSPDATA}/spam-not   ${ASSPDIR}/spam-not
>
> mount --bind ${ASSPDATA}/logs   ${ASSPDIR}/logs
>
> }
>
>
> --
> James Moe
> moe dot james at sohnen-moe dot com
> 520.743.3936
> Think.
>
>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

Thanks Thomas.  I guess an old file still caused the issue with the SEGV.
I did do some corpus clean up the other day including manual copying to the
errors folders,  I'll see if I have notes on what I was doing to help
identify any file that day that might be the culprit.

Other notes:
Definitely going to segregate logs from the data drive for assp using a
symlink.

And I only said c:\ASSP so not to spark question from anyone (not you, but
others) as to the problem about it being a non-default location.  I have it
on another partition, dedicated to ASSP, not even using ASSP as the folder
name.  It's just that this partition,didn't have enough free space to to
handle the massive logs caused by the SEGV errors filling them.

On Sat, Sep 4, 2021 at 2:43 AM Thomas Eckardt 
wrote:

> fixed in assp 2.6.6 *SPAM-Evaporator* build 21198:
>
> 
>
> - invalid UTF8 characters in a mail subject may caused SEGV errors in
> 'Win32::Unicode::Dir' on windows systems
>
>
> I hoped I had fixed it. New generated files should not cause the SEGV, but
> old filenames may cause it.
>
> set
> Regular Expression to Identify skipped Log Lines* (noLogLineRe)
> to
> signal SEGV in
>
> >I just don't know if putting a drive letter in the config is another
> option.
> No, this will not work. But links will work on windows.
>
> >I have everything on my C drive in c:\ASSP\logs.
> Is there any good reason to this at a production system???
>
> Thomas
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:03.09.2021 22:11
> Betreff:Re: [Assp-test] Logging Options: Log filled with SEGV
> warnings
> --
>
>
>
> Thanks James.
> The same idea on Windows with a symbolic link should work fine.  I just
> don't know if putting a drive letter in the config is another option.
> My real concern is the SEGV fauls going nuts.
>
>
> On Fri, Sep 3, 2021 at 2:14 PM James Moe via Assp-test <
> *assp-test@lists.sourceforge.net* >
> wrote:
> On 2021-09-03 09:11, K Post wrote:
>
> > 1) Can the logfile configuration be set to use a different disk?
> >
>   This was relatively easy on linux. I do not know if Windows has a similar
> functionality. Basically the ASSP log directories are re-directed to
> another
> area with more space.
>
> ASSPDIR="/usr/local/bin/assp2";
>
> ASSPDATA="/data01/var/assp";
>
> assp_mount () {
>
> echo "Mounting assp volumes..."
>
> mount --bind ${ASSPDATA}/okmail ${ASSPDIR}/okmail
>
> mount --bind ${ASSPDATA}/discarded  ${ASSPDIR}/discarded
>
> mount --bind ${ASSPDATA}/quarantine ${ASSPDIR}/quarantine
>
> mount --bind ${ASSPDATA}/spam-yes   ${ASSPDIR}/spam-yes
>
> mount --bind ${ASSPDATA}/spam-not   ${ASSPDIR}/spam-not
>
> mount --bind ${ASSPDATA}/logs   ${ASSPDIR}/logs
>
> }
>
>
> --
> James Moe
> moe dot james at sohnen-moe dot com
> 520.743.3936
> Think.
>
>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

Thanks James.
The same idea on Windows with a symbolic link should work fine.  I just
don't know if putting a drive letter in the config is another option.
My real concern is the SEGV fauls going nuts.


On Fri, Sep 3, 2021 at 2:14 PM James Moe via Assp-test <
assp-test@lists.sourceforge.net> wrote:

> On 2021-09-03 09:11, K Post wrote:
>
> > 1) Can the logfile configuration be set to use a different disk?
> >
>   This was relatively easy on linux. I do not know if Windows has a similar
> functionality. Basically the ASSP log directories are re-directed to
> another
> area with more space.
>
> ASSPDIR="/usr/local/bin/assp2";
>
> ASSPDATA="/data01/var/assp";
>
> assp_mount () {
>
> echo "Mounting assp volumes..."
>
> mount --bind ${ASSPDATA}/okmail ${ASSPDIR}/okmail
>
> mount --bind ${ASSPDATA}/discarded  ${ASSPDIR}/discarded
>
> mount --bind ${ASSPDATA}/quarantine ${ASSPDIR}/quarantine
>
> mount --bind ${ASSPDATA}/spam-yes   ${ASSPDIR}/spam-yes
>
> mount --bind ${ASSPDATA}/spam-not   ${ASSPDIR}/spam-not
>
> mount --bind ${ASSPDATA}/logs   ${ASSPDIR}/logs
>
> }
>
>
> --
> James Moe
> moe dot james at sohnen-moe dot com
> 520.743.3936
> Think.
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Logging Options: Log filled with SEGV warnings

[K Post]

Summary:
1) Get log files to separate disk on Windows system
2) Stop repeating warnings from filling log?


Yesterday and today, my log filled with

Sep-03-21 02:50:05 Warning: got unexpected signal SEGV in Worker_10001:
package - Win32::Unicode::Dir, file -
c:/strawberry/perl/site/lib/Win32/Unicode/Dir.pm, line - 80!


yesterday's log is 20gb+ of this error
Today's grew until it filled my disk.  (it's a small operation without a
ton of disk space)

I cleared those big logs out and restarted ASSP without issue.   I've had
this before where something goes haywire, the logs fill, and I eventually
run out of space.

SO:

1) Can the logfile configuration be set to use a different disk?  I have
everything on my C drive in c:\ASSP\logs.  I understand that the path is
relative to the ASSP folder, but could I put in an inexpensive 1TB drive
just for logs, have that be the D drive, and do something like d:\ASSPLogs
for maillog??   I suppose I could get a symbolic link to the other drive,
but if I can just enter an alternate drive natively in the GUI, that'll be
better IMO.

2) Any chance of changing ASSP so that when something odd like this happens
warnings are only written every X minutes or something?  Some sort of cache
of the last error written? If the next one is the same, skip it unless X
minutes have passed?  Or trigger an ERROR line after X of the same error in
a row?  (I have email notification if error shows in the log)
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] ASSP with Office365 - discussion

[K Post]

Hello everyone,

One department with their own subdomain at our charity is going to be
moving to Office365 for email.  I want to be able to *keep ASSP in use for
this handful of users*, but could use some guidance from the community here.

My plan is:

1) Inbound mail to dept1.ourcharity.org will still have their MX record as
our internal server.

1a) The internal SMTP server will then forward mail to users @
dept1.ourcharity.org  to @forward.dept1.ourcharity.org which
will be set up as an alternate address for each user at Office365.   that
forward subdomain will have MX records pointing to office365.

I think we're okay there.

2) Outbound mail from Office365 will use a smarthost to connect to ASSP so
that outbound mail can be added to the corpus, whitelist and other lists
updated, etc.

My issue is with relaying. I do not want to allow all of Microsoft's
Office365 IP space (into allowRelayCon) as allowed relay hosts because the
huge IP space that they use to send messages out is not exclusively used by
me.  That means that any other Office365 user could set up an Exchange
connector to send messages through our ASSP.

Authentication would solve that issue, *but apparently, Office365 does NOT
allow SMTP AUTH for outgoing smarthost. *

I've got to believe that I'm not the only one out there who has run into
this problem before.  Any ideas would be incredibly appreciated!!

Thanks,
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Main_Thread is unable to transfer connection to any worker - try again

[K Post]

I'm desperate for help.  It seems that ASSP has thread problems frequently
when it reloads the config.  I see warnings that the Main_Thread is unable
to transfer connection to any worker, often every 5 minutes for hours.

May-17-21 20:40:35 Saving config
May-17-21 20:40:35 Info: no configuration changes detected - nothing to
save - file c:/assp/assp.cfg is unchanged
May-17-21 20:41:08 Info: notification message queued to sent to
assp-not...@ourcharity.org
May-17-21 20:41:08 Warning: Main_Thread is unable to transfer connection to
any worker - try again!

Even when there's no settings detected (line 2 above), it can hang.

I can't seem to figure out why.

Windows 2012 R2
Strawberry Perl
Latest ASSP

Any guidance on where to start?

Thank you
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Feature Reqiest: Customizing more of the 554 5.7.1 error messages in rejections

[K Post]

Thanks Thomas.  This would definitely help us give more information to
senders who are erroneously rejected.  Your recommendation will help to
solve the problem at hand!  Thank you!

I don't follow why an optional modification to the hard coded 554's would
potentially break ASSP, but obviously you do.  I'm assuming it's
significant enough of a risk that it's not worth it for me to forge on
testing code changes?  Can you shed some light on that?  The 554.x.x reason
would stay in place, only the description of it would change like we do for
customizing the delay reply.

On Thu, May 6, 2021 at 5:24 AM Thomas Eckardt 
wrote:

> I don't think this is a good idea. Dirk explained one reason (IMHO a minor
> one), but there are much more reasons why such things should go another way.
> sub seterror is only the surface - changing any currently not configurable
> error-reply (eg. 5xx to 4xx or 2xx) may lead in to an unexpected behavior
> of assp or the peer.
>
> My suggestion:
>
> Anyone who wants to explain blocking reasons should build a public web
> page with the explanations (detailed or not - how ever it is wanted). This
> web page can also explain how to fix problems, e.g. how to use the
> NOTSPAMTAG, encrypt zip files 
>
> ASSP will get one new configuration parameter - e.g.
> 'addErrorReplyExplanation'
>
> -
> 'addErrorReplyExplanation'
>
> The text defined here will be added to every permanent SMTP-error-reply
> (starting with 5xx) send by assp. For example to add a web link, where
> blocking reasons are explained.
> e.g.:
> - error explanations at https://your.web.domain/block-reasons
> <https://your.web.domain/blockedmailreasons>
> or
> - error explanations at https://your.web.domain/block-reasons
> <https://your.web.domain/blockedmailreasons>
> ?session=SESSIONID=IPCONNECTED
>
> The text (and possibly a clickable link) will become visible to blocked
> senders in the NDR (No Delivery Report) of the blocked mail.
> In the second example the assp session-id and the connected IP-address are
> part of the link. The web server can extract the log entries for the mail
> from the maillog.txt and can explain much better and/or check the database
> for the IP reputation and ... and ...  .
> If you want to skip this addition for any configurable SMTP-reply, write
> the literal NOEXPLAIN at the end of the configured SMTP-reply definition.
> The literal will be removed from the reply before it is sent.
> -
>
> These are the advantages of my suggestion:
>
> - minor code changes
> - flexible and more detailed error explanations - possibly in local
> language
> - much less (than in your suggestion) confusing reply configurations
> - keeps hardcoded (and required) 5xx reply codes
> - the assp code can force skipping the explanation addition where it is
> really not wanted - e.g. AUTH errors
> - if someone builds such an explanation web page, it can be shared
>
>
> Thomas
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:05.05.2021 17:14
> Betreff:[Assp-test] Feature Reqiest: Customizing more of the 554
> 5.7.1 error messages in rejections
> --
>
>
>
>
> Thomas,
> First,  as requested, I've tried to change the way I generally ask for
> features.  I hope this is better, helpful, and fully explains my reasoning.
>
> Just like we can customize the spamError, DelayError, NoValidRecipient,
> and other reasons that appear in rejection/delay messages to the sender, I
> believe it would be valuable to have additional optional settings to
> customize rejection messages in ASSP.  This would help legitimate senders
> who are erroneously rejected reach out to their IT for the following
> reasons:
>
> 554 5.7.1 Extreme Bad IP Profile
> 554 5.7.1 too many different IP's for domain (domain)
> 554 5.7.1 too frequent connections for (ip)
> 554 5.7.1 too frequent connections for originated IP-address (ip)
> 554 5.7.1 too many mails with same subject
>
> there's also 521 transmission terminated, but I've never encountered that.
>
> My top priority is the Extreme Bad IP Profile message.  Here's why:
> We've seen several schools that our charity work with get IP blocked by
> ASSP recently, and rightfully so.  But there's legitimate senders too using
> the same IP space. That then gets our charity calls from the students
> asking what Extreme Bad IP Profile is (to which our well intentioned but
> non-technical volunteers obviously have no idea).
>
> Having a message like:
> 554 5.7.1 Your message was rejected due to your server's reputation.
> Please work with your 

[Assp-test] Feature Reqiest: Customizing more of the 554 5.7.1 error messages in rejections

[K Post]

Thomas,
First,  as requested, I've tried to change the way I generally ask for
features.  I hope this is better, helpful, and fully explains my reasoning.

Just like we can customize the spamError, DelayError, NoValidRecipient, and
other reasons that appear in rejection/delay messages to the sender, I
believe it would be valuable to have additional optional settings to
customize rejection messages in ASSP.  This would help legitimate senders
who are erroneously rejected reach out to their IT for the following
reasons:

554 5.7.1 Extreme Bad IP Profile
554 5.7.1 too many different IP's for domain (domain)
554 5.7.1 too frequent connections for (ip)
554 5.7.1 too frequent connections for originated IP-address (ip)
554 5.7.1 too many mails with same subject

there's also 521 transmission terminated, but I've never encountered that.

My top priority is the Extreme Bad IP Profile message.  Here's why:
We've seen several schools that our charity work with get IP blocked by
ASSP recently, and rightfully so.  But there's legitimate senders too using
the same IP space. That then gets our charity calls from the students
asking what Extreme Bad IP Profile is (to which our well intentioned but
non-technical volunteers obviously have no idea).

Having a message like:

554 5.7.1 Your message was rejected due to your server's reputation.
Please work with your local email administrator to resolve this issue.
[Mail administrator: Your sender IP is on our blocklist due to previously
observed bad activity.]

would be more clear to the sender, and they'd know to get their IT
involved.  If their IT calls us, so be it.

Even better would be to put the IP address and sender domain into the
already variable message, but that's probably more coding work than it's
worth.

I tried my hand at editing ASSP code (breaking the code signature) on a
test server to make it so that we can optionally customize the error
messages returned for some of the rejected mails.  Something's not right
with the way the GUI prompts for the info, but I think my concept is solid,
there should be no sweat for a perl pro to modify the code, and I believe
it would be a widely valuable change.  If you'll only consider this request
if I first get it working, I will press along, but I feel like you'll hate
my sloppy code and will need to rework it anyway.


To accomplish what I'm asking for, I believe ASSP would need to be modified
to have optional configuration entries in the GUI for each of the above 554
error scenarios.

Then everywhere that there are lines like

seterror( $fh, "554 5.7.1 Extreme Bad IP Profile", 1 );

(which is only 7 554 locations that aren't customizable already)

We'd need something like the logic that is already used for delayed
messages:

if ($DelayError) {

$reply = $DelayError."\r\n";
} else {
$reply = "451 4.7.1 Please try again later\r\n";
}

so something like

if ($ExtremeBadIPProfileErrorMessage) {
$reply = $ExtremeBadIPProfileErrorMessage ."\r\n";
} else {
$reply = "451 4.7.1 Extreme Bad IP Profile \r\n";
}

Do you think that's a good idea, would it be reasonable to enhance the code
to accomplish this?  This wouldn't impact the globalPB right?

and as importantly, are you happier with the way that I asked this question?

Thanks
Ken
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Senderbase Matching Substring

[K Post]

Loud and clear Thomas.  No need to reply unless I'm wrong or offbase here:
I prefixed *everything* in whitesenderBase to start with ^ instead of just
\b since \b seems to match hypens.  I'm assuming that the ^ prefix instead
of \b will not cause any sort of performance or other problem.  I haven't
seen one, but if it's not advisable, please let me know?

So my original \bwalmart\.com to ^walmart\.com$ so that fake-walmart.com
doesn't match.

I feel like the ^name$ syntax ensures that I'm matching complete domain or
network name as listed in senderbase vs "just" a word boundary to the end
of one with \bdomain$Bad idea?

Your explanation of whiteSenderBase being a RE vs dkimWLAddresses being
just a list was very helpful and an important point for me to remember.

While I can hack the very basics of perl, there's no way I could implement
new functionality.  That's why I ask the "wouldn't it be nice" questions.
That sometimes gets good ideas out there, like my suggestion for the dkimWL
and dkimNP which has been a game change here.  Often it gets a, "no, that's
a bad idea" or "that's not necessary, you could instead..." reply which is
fine.  I feel like that's why we have this discussion list, the free
exchange of ideas, with one goal, blocking spammers and scammers.  We're on
the same team, even if I might make you feel like I'm an adversary
sometimes with my questions.




On Sun, May 2, 2021 at 7:14 AM Thomas Eckardt 
wrote:

> My final comment to this thread.
>
> -  whiteSenderBase is a reglar expression, while dkimAddresses and
> dkimNPAddresses are address- and domain lists
> - if you've added '\bwalmart\.com' to whiteSenderBase, change it to
> '\bwalmart\.com$'
> - do not change anything else in whiteSenderBase as long as you don't get
> a wrong (or not the expected) detection for this feature
> - trust ARIN, RIPE and all the other registars - they will not register
> suspicious company names
>
> In terms of assp - 'assumes' and 'believes' are most times bad. The
> feature descriptions in the GUI and the manual should be informativ enough
> for IT professionals - if you think, there is anything missing, post your
> suggestions for changes here or in the forum - BUT RTMF!
> Features should always work like described. If this is not the case, post
> this here or in the sourceforge ticket system (
> https://sourceforge.net/p/assp/tickets/)
> If assp will remove, add or change anything in your configuration values
> 'automatically', this should be stated in the GUI (if not, please report
> this)
> As long as it is not exeptional stated in the GUI, assp features should
> work RFC conform.
> Any RFC related information will not be added to the GUI - RFC's, drafts
> and there substitutes can be read in the IANA web. Things like 'what is
> SPF, SRS, DKIM ?' will not become part of the assp manual - there are
> very good explanations and examples in the web.
>
> Suggestions for new features or changes are wellcome. How ever: I prever
> to see something like 'I saw, changed, tested and got very good results
> ...' - than something like 'would'nt it be nice to have'
>
>
> Thomas
>
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:29.04.2021 22:29
> Betreff:Re: [Assp-test] Senderbase Matching Substring
> --
>
>
>
> and I'll add that I believe it's a good idea to start whiteSenderBase
> lines with ^ instead of \b
>
> For example:
> \bapple\,com$
> would allow *bad-apple.com* <http://bad-apple.com/>, whereas
> ^apple\.com$
> won't.
>
> I suppose it should be obvious, it's just a regex, but the example file
> for all those years ago, threw me off.
>
>
> On Thu, Apr 29, 2021 at 10:20 AM K Post <*nntp.p...@gmail.com*
> > wrote:
> also, fyi, the sample file files/whiteorg.txt doesn't have $ to force it
> only to match the line ending, which I believe is what I used to come up
> with my incorrect assumption.  Putting in at least 1 example in that file
> with a $ might help others not make the same mistake that I have.  My
> money's on well more than half of the admins using ASSP haven't made the
> same mistake.
>
> In the sample file, everything starts with \b, clearly telling us that it
> needs to be the start of a word, but no lines end with $ or even \b   For
> example:
> \bbank of america
> When I see that, I think "we have the \b so that some scammer can't get
> senderbase to have their network as 'BADbank of america' and get through
> our filters."  That's logical, but I also assumed that because there isn't
> a trailing \b (or actually $) that it's going to the end o

Re: [Assp-test] HeloBlacklistIgnore still matching helo?

[K Post]

got it.  thanks.  enjoy your weekend and break from me.

On Fri, Apr 30, 2021 at 1:49 PM Thomas Eckardt 
wrote:

> from the analyzer GUI:
>
> Note: Analysis is performed using the current spam database, hashes and
> lists -- if yours was rebuilt since the time the mail was received, you'll
> receive a different result. This also applies to the feature matching
> results, they may be diffent from the results when the mail was received. *All
> feature matching results are shown **stateless* - which means for
> *example*: if '*noprocessing*' and a '*RBL/DNSBL* hit' is shown here, in
> the real mail processing, *the RBL check may be skipped* because of the 
> *noprocessing
> state*.
>
>
> Yes, there is no example for the possible combinations related to '
> heloBlacklistIgnore' and all the other 'helo' settings, lists, hashes ...
> but who would need them?
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:30.04.2021 17:39
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> and there's my answer to " What could I be doing wrong / misunderstanding?"
> *I didn't realize that analyze didn't consider heloBlacklistIgnore.*
> I don't see that indicated in the gui.  Analyze says that it uses the
> current db, hashes, and lists.  I, apparently incorrectly, thought that
> "lists" would include the heloBlaklistIgnore list.
>
> I'm sorry to once again have frustrated you with my questions.  if there's
> a better way I could have asked what I did, or somewhere else I should be
> asking, I'm all ears.
>
>
> On Fri, Apr 30, 2021 at 10:31 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> >in my heloBlacklistIgnore and then a mail comes in with helo
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/> *it
> is still getting scored** due to a known bad helo.*
>
> show me a single logline from a processed mail where this happens !
>
>
> >I see:  HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
>
> this is the result line from the analyzer - the analyzer shows every
> match, it's on YOU to know that a heloBlacklistIgnore match will skip the
> helo checks for real mail processing
>
> Thomas
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:30.04.2021 15:38
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> I guess I was unclear in my question.  I'm not saying that a match in
> heloBlacklistIgnore is *causing *scoring, I'm saying that I have
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> in my heloBlacklistIgnore and then a mail comes in with helo
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/>* it
> is still getting scored due to a known bad helo.*
>
> I see:  HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
> but why would it be on the blacklist (and get the score assigned for that)
> when I have *.*sparkpostmail.com* <http://sparkpostmail.com/> in my
> heloBlacklistIgnore?
>
>
>
>
> On Fri, Apr 30, 2021 at 2:41 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> If I ask you about something that does'nt exists in assp . what it is
>  LOL
>
> Thomas
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:29.04.2021 16:04
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> The line in heloBlacklistIgnore is
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> There's no score associated to that line, but I must not be understanding
> what you're asking or what heloBlacklistIgnore does - how could there be a
> score assosiated with this?.  Doesn't heloBlacklistIgnore just tell ASSP
> not to pay any attention to helo's that match that list when scoring a
> message?  if so, shouldn't the line above make it so that a helo from
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/>
> isn't considered when scoring the message?
>
> If you're asking for my hlValencePB, that's set to 15, but I don't know
> how that comes i

Re: [Assp-test] HeloBlacklistIgnore still matching helo?

[K Post]

and there's my answer to " What could I be doing wrong / misunderstanding?"
*I didn't realize that analyze didn't consider heloBlacklistIgnore.*
I don't see that indicated in the gui.  Analyze says that it uses the
current db, hashes, and lists.  I, apparently incorrectly, thought that
"lists" would include the heloBlaklistIgnore list.

I'm sorry to once again have frustrated you with my questions.  if there's
a better way I could have asked what I did, or somewhere else I should be
asking, I'm all ears.


On Fri, Apr 30, 2021 at 10:31 AM Thomas Eckardt 
wrote:

> >in my heloBlacklistIgnore and then a mail comes in with helo
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/> *it
> is still getting scored** due to a known bad helo.*
>
> show me a single logline from a processed mail where this happens !
>
>
> >I see:  HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
>
> this is the result line from the analyzer - the analyzer shows every
> match, it's on YOU to know that a heloBlacklistIgnore match will skip the
> helo checks for real mail processing
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:30.04.2021 15:38
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> I guess I was unclear in my question.  I'm not saying that a match in
> heloBlacklistIgnore is *causing *scoring, I'm saying that I have
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> in my heloBlacklistIgnore and then a mail comes in with helo
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/>* it
> is still getting scored due to a known bad helo.*
>
> I see:  HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
> but why would it be on the blacklist (and get the score assigned for that)
> when I have *.*sparkpostmail.com* <http://sparkpostmail.com/> in my
> heloBlacklistIgnore?
>
>
>
>
> On Fri, Apr 30, 2021 at 2:41 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> If I ask you about something that does'nt exists in assp . what it is
>  LOL
>
> Thomas
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:29.04.2021 16:04
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> The line in heloBlacklistIgnore is
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> There's no score associated to that line, but I must not be understanding
> what you're asking or what heloBlacklistIgnore does - how could there be a
> score assosiated with this?.  Doesn't heloBlacklistIgnore just tell ASSP
> not to pay any attention to helo's that match that list when scoring a
> message?  if so, shouldn't the line above make it so that a helo from
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/>
> isn't considered when scoring the message?
>
> If you're asking for my hlValencePB, that's set to 15, but I don't know
> how that comes into play here.
>
> As always, much appreciated.
> ken
>
>
>
> On Mon, Apr 26, 2021 at 2:17 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> How high is the scrore used by assp for 'heloBlacklistIgnore' ?
>
> Thomas
>
> RTM,RTM,RTM,RTM
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:25.04.2021 10:14
> Betreff:[Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> I have
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> in my heloBlacklistIgnore, yet messages from
> *mta-85-129.sparkpostmail.com* <http://mta-85-129.sparkpostmail.com/>
> seem to still be scored based on that. From analyze:
>
> • HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
>
> What could I be doing wrong / misunderstanding?
>
> Thanks
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
>
> DISCLAIMER:
> **

Re: [Assp-test] HeloBlacklistIgnore still matching helo?

[K Post]

I guess I was unclear in my question.  I'm not saying that a match in
heloBlacklistIgnore is *causing *scoring, I'm saying that I have
*.sparkpostmail.com
in my heloBlacklistIgnore and then a mail comes in with helo
mta-xx-xxx.sparkpostmail.com* it is still getting scored due to a known bad
helo.*

I see:  HELO Blacklist: 'mta-85-129.sparkpostmail.com'
but why would it be on the blacklist (and get the score assigned for that)
when I have *.sparkpostmail.com in my heloBlacklistIgnore?




On Fri, Apr 30, 2021 at 2:41 AM Thomas Eckardt 
wrote:

> If I ask you about something that does'nt exists in assp . what it is
>  LOL
>
> Thomas
>
>
>
>
>
> Von:"K Post" 
> An:"ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum:29.04.2021 16:04
> Betreff:Re: [Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> The line in heloBlacklistIgnore is
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> There's no score associated to that line, but I must not be understanding
> what you're asking or what heloBlacklistIgnore does - how could there be a
> score assosiated with this?.  Doesn't heloBlacklistIgnore just tell ASSP
> not to pay any attention to helo's that match that list when scoring a
> message?  if so, shouldn't the line above make it so that a helo from
> *mta-xx-xxx.sparkpostmail.com* <http://mta-xx-xxx.sparkpostmail.com/>
> isn't considered when scoring the message?
>
> If you're asking for my hlValencePB, that's set to 15, but I don't know
> how that comes into play here.
>
> As always, much appreciated.
> ken
>
>
>
> On Mon, Apr 26, 2021 at 2:17 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* > wrote:
> How high is the scrore used by assp for 'heloBlacklistIgnore' ?
>
> Thomas
>
> RTM,RTM,RTM,RTM
>
>
>
>
>
> Von:"K Post" <*nntp.p...@gmail.com* >
> An:"ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* >
> Datum:25.04.2021 10:14
> Betreff:[Assp-test] HeloBlacklistIgnore still matching helo?
> --
>
>
>
> I have
> *.*sparkpostmail.com* <http://sparkpostmail.com/>
> in my heloBlacklistIgnore, yet messages from
> *mta-85-129.sparkpostmail.com* <http://mta-85-129.sparkpostmail.com/>
> seem to still be scored based on that. From analyze:
>
> • HELO Blacklist: '*mta-85-129.sparkpostmail.com*
> <http://mta-85-129.sparkpostmail.com/>'
>
> What could I be doing wrong / misunderstanding?
>
> Thanks
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* 
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> ***
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> ***
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test