I've spent the better part of 2 hours trying to figure out what's going on.
First, please note that for me, both 21293 and 21302 appear to record the
.msg file in errors-spam/errors-notspam correctly. The .msg is getting
extracted from the report properly. If I do an analyze from the GUI on
these files, it works perfectly.
ConfigAnalyze seems to be analyzing the wrong content, differently in 21302
than 21293. I believe that with 21302, when ConfigAnalyze is called when
processing a report that's sent with a .msg attachment, it might be looking
at the entirety of the email report instead of just what's in the .msg
file. I don't know if that's as intended.
Using the (very helpful) ability of 21302 to write reports to debug, here's
what I've discovered:
I see in ConfigAnalyze where it's spitting out the mime headers starting in
the middle error. I don't really know why it's adding the CR LF <br />,
but suspect it has to do with what I actually did find, *TABS saved in the
file in the debug folder. *I don't know if Outlook is doing the tab
wrapping or if it's ASSP, but either way, ASSP isn't happy about them.
My reporting address is pretty long, something like
report-not-s...@assp.detroit.ourcharity.org
When I forward a report as a .msg attachment, I've just been using that
address from remembered addresses in Outlook. There's no address book
entry in Outlook, so Outlook behaves like Outlook behaves, adding the email
address in quotes to the to line. The message that outlook sends is saved
to the debug folder starting like this:
From: Ken Post <m...@ourchairty.org>
To: "'report-not-s...@assp.detroit.ourcharity.org'" <-- that's a double
quote, then single quote, then the address, then a single quote, then a
double.
<report-not-s...@assp.detroit.ourcharity.org> <-- *this line starts
with TAB, not spaces*
Subject: FW: test of report message
When it's received like this, ASSP gives me
Found a possible MIME header start in the middle of the mail - the analyze
may be wrong
[CR][LF]<br /> •
Subject: FW: test of report message[CR][LF]<br />
To test this further, I sent the same message from outlook, but this used a
very short name in the To field. The message shows in the debug folder like:
From: Ken Post <m...@ourchairty.org>
To: RNS <report-not-s...@assp.detroit.ourcharity.org> <-- no quotes
around the name, to line on single line now
Subject: FW: test of report message
The headers continue with stuff about message id, content language, etc,
until it gets to a line like:
Content-Type: multipart/mixed;
boundary="_004_06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a_"
MIME-Version: 1.0
That second line starts with a tab again.
ASSP this time *doesn't complain about the subject*, instead, I get
found a possible MIME header start in the middle of the mail - the analyze
may be wrong
boundary="_
004_06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a06dbb2065c154d4a_"[CR][LF]<br
/> •
MIME-Version: 1.0[CR][LF]<br />
With 21302, I tried saving the message in Outlook, zipping it, and
attaching that. The report in the debug still has the tab indented To
line, but ASSP does not complain, analyze accurately reports on the
extracted message. I still don't know if your note in the changelog "
Notice: always compress (e.g. zip) reported emails before they are sent to
assp!" applies to .msg attachments too. Doing do, makes this work, but
requiring users to zip would spell the end of my already reluctant users
from ever reporting again :(
I hope information is helpful and this gives you enough information to
either change ASSP further or point me in the direction of where I'm going
wrong.
On Sat, Oct 30, 2021 at 11:28 AM Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> >found a possible MIME header start in the middle of the mail - the
> analyze may be wrong
> [CR][LF]<br /> •
> Subject: FW: test of report message[CR][LF]<br />
>
>
>
> Is there any one else on this mailing list, who expects an email received
> by assp to start with an empty line followed by <br /> (html code) or that
> the first header line of such a mail is the subject header line ?
>
> Thomas
>
>
>
>
>
> Von: "K Post" <nntp.p...@gmail.com>
> An: "ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum: 30.10.2021 09:15
> Betreff: Re: [Assp-test] fixes in assp 2.6.6 *SPAM-Evaporator*
> build 21302
> ------------------------------
>
>
>
> sorry, I sent the last message before proofing or finishing. Grr, gmail.
> I'll wait to hear from you. I have more thoughts on NWLI and other
> sections.
>
> On Fri, Oct 29, 2021 at 6:00 PM K Post <*nntp.p...@gmail.com*
> <nntp.p...@gmail.com>> wrote:
> This is simply terrific. You keep making ASAP better! The rebuild config
> efficiency improvements are especially appreciated. Thanks so much as
> usual for spending what must have been a long time thinking about and
> making all of these changes.
>
> SURPRISE, I have questions and comments:
>
> *Fix to emailing report with attached .msg report not working?*
> *(if email reports need to be zipped, ignore this)*
>
> I just tested, sending a zip of a .msg has the analyze report works
> correctly. Sending just the .msg attachment with 21293 give the previous
> errors. *Sending just the .msg with this new 21302 unfortunately is
> worse and isn't working*. I'm now getting:
> found a possible MIME header start in the middle of the mail - the analyze
> may be wrong
> [CR][LF]<br /> •
> Subject: FW: test of report message[CR][LF]<br />
> and even less info is shown in the emailed analyze report than before
>
> I had ReportLog set to debug. The .eml file that goes to debug is
> attached to this reply.
>
>
>
> *Clarification on the need now(?) to compress Emailed reports from Outlook*
> You wrote: "Notice: always compress (e.g. zip) reported emails before they
> are sent to assp!" The GUI says " It is also possible to send MS-outlook
> '.msg' files (possibly zipped)." Is it now required?
>
> I've always done "Forward as Attachment" in Outlook to report which
> attaches the current message as a .msg formatted file (I previously
> incorrectly wrote .eml) and seems to work (except for the analyze big that
> you're attempting to squash). The vast majority of our staff do the same,
> but requiring them to first save the .msg, then zip, then attach to
> report might be asking too much.
>
> While attached Outlook .msg files are binary, I don't >>think<< they're
> compressed. The .msg files always have made it to the corpus saved as
> plain text files which seems right. It was only the analyze report that
> was failing.
>
>
>
>
>
>
> *MaxBytesReports*
>
> The gui talks about MaxBytesReports.
>
> Any mail sent or forwarded by local/authenticated users to this username
> will be interpreted as a spam report. Multiple attachments get truncated to
> MaxBytesReports. Do not put the full address here, just the user part.
>
> For example: asspspam . Use a fake domain like @assp.local or @
> *assp-nospam.org* <http://assp-nospam.org/> when you send the email- so
> the full address would be then asspspam@assp.local.
>
> You can sent multiple mails as attachments and/or zipped file(s). Each
> attached email-file must have the extension defined in "maillogExt". In
> this case only the attachments will be processed. To use this
> multi-attachment-feature an installed Email::MIME module in PERL is needed.
> It is also possible to send MS-outlook '.msg' files (possibly zipped). To
> use this MS-outlook-feature in addition an installed
> Email::Outlook::Message module in PERL is needed.
>
>
> I don't see MaxBytesReports any where else in the code. Is this supposed
> to be MaxBytes?
>
> Also, GUI correction " You can sent multiple mails as attachments and/or
> zipped file(s)" should be "send"
>
>
>
> *Finding DKIM matches during rebuild:*
>
>
> AddDKIMHeader needs to be on right?
>
> Since it looks like the code is looking for the X-ASSP-DKIMIdentity line,
> I think you should add a comment for the hidden DoRBWhite parameter and
> others* that AddDKIMHeader in the GUI needs to be on for this to work. *
>
>
> Speed -> more configuration choices necessary?
> Great point about slowness in rebuild if this is all on and the
> recommendation of potentially only turning it on periodically.
>
> Since we're now doing more checks, does it make sense to have additional
> hidden parameters to give more granular control? I feel like I'll always
> want to check for whitelisted in spam no matter how it matches, but other
> might only want to consider DKIMWLAddress, while others don't want
> DKIMWLAddresses and only matches to the actual whitelist. How about
> letting DoRBWhite be configured with different values like we have for
> DoNoFrom?
> Match:
> 0: nothing
> 1: whiteRe
> 2: npRe
> 4: whiteListedDomains
> 8: noProcessingDomains
> 16: whiteListedIPs
> 32: noProcessingIPs
> 64: DKIMWLAddresses
> 128: DKIMNPAddresses
> and have those summed up? Too complicated for not enough value? Dunno,
> thinking out loud here. I'm cool with everything on, but maybe there are
> others who would prefer to more granularly configure?
>
>
> related: GUI mistake. the AddDKIMHeader description still says that it
> adds X-ASSP-DKIM: instead of "X-ASSP-DKIMidentity
>
>
> *DoRBBlack removal of deny matches -- curiosity:*
> For the new DoRBBlack, why is it checking denySMTPConnectionsFromAlways
> and denySMTPConnectionsFrom? Aren't additions made to that list after
> we've collected what we've wanted (good or bad) from those IP's / emails
> which would be good to have in the corpus?
>
> NWLI
> I'd like to rewrite the NWLI description at the bottom of the GUI, but I
> need clarification first. I'm sure NWLI functionality works in the code,
> it's just not explained well in the GUI.
>
> I see the revised language, but I'm still not sure that I follow. When
> you say "optional to use '+'" do you mean something N and N+ are
> functionally identical? If that's true, why bother having the + as an
> option at all? If there's a difference between having a plus and not,
> please explain.
>
> For starters, you have
>
> "So the line ~Heuristics|Email~=>50:>N-W-LI could be read as: take the
> regex with a weight of 50, never score noprocessing mails, never score
> whitelisted mails, score local mails and mails from ISP's."
>
> But if it's ANDed together, it would really mean
> score 50 to a mail that matches Heuristics|Email when that mail is NOT
> noprocessing, is NOT whitelisted, IS local AND IS ALSO is an ISP mail.
> To me, the way you have it written implies that it would score 50 if it's
> either local OR ISP as long as it's neither whitelisted nor noprocessing.
>
>
>
> Also, parameters are separated everywhere else that I see by => but the
> third parameter here needs to be :> My lousy eyes missed that until just
> now. Why the the inconsistency?
>
>
>
>
>
> Other thanks, notes, and reqests
>
> - Thanks for adding that explanation bit into the GUI and the icon!
> Also, this:
> Info: file D:/assp/IP-Lists/IPS-facebookmail.com.cfg is now stored
> encrypted, because it is used in secured config Groups Excellent
> - The ReportLog addition will be really helpful, even if just for
> periodic reviews of user submitted reclassifications vs ones I've done
> through the GUI. Could we have it use the same file extension as we use
> for the corpus? (maillogExt)
> - In the code, in some places X-Assp- headers are referenced, others
> have X-ASSP- All of the compares I've seen appear to be case insensitive,
> but you might consider standardizing so that type-a personalities can be
> calmed :)
>
>
>
>
> On Fri, Oct 29, 2021 at 10:31 AM Thomas Eckardt <
> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote:
> Hi all,
>
> fixed in assp 2.6.6 *SPAM-Evaporator* build 21302:
>
> - Improved email address detection in the emailinterface list reports
> (whitelistadd , whitelistremove, ....).
>
> - The change time for include files used in the 'Groups' feature were not
> recorded in workers. This caused unexpected configuration reloads in the
> workers, until
> assp was restarted.
>
> - Any change made for 'Groups' caused a reload for all configuration
> parameters where a group was used, even the related group was not changed.
> A configuration reload is now
> only done for changed groups and there related configuration parameters.
>
> - Unexpected results were produced by the analyzer, if emails were sent as
> (not zipped) attachment to the emailinterface for analyzing - using outlook
> as mail client (+exchange).
> Notice: always compress (e.g. zip) reported emails before they are sent
> to assp!
>
>
> changed:
>
> - If the hidden parameter 'DoRBWhite' is set, the rebuildspamdb process
> searches for matches in
> whiteRe, npRe, whiteListedDomains, noProcessingDomains, whiteListedIPs,
> noProcessingIPs, DKIMWLAddresses and DKIMNPAddresses -
> and removes those mails from the assp/spam folder.
>
>
> - 'ReportLog','Enable Report logging'
> 'If set to diagnostic, each received report mail will be stored in the
> assp/debug folder.'
> This makes it more easy to track down report problems, based on the
> data sent by the mail client to assp.
>
> - The GUI description for the NWLI enhancement (for regular expressions)
> was updated. The code was changed to get the NWLI results exactly like
> descriped in the GUI.
>
> - A hint (and context help) about encryped configuration parameters and
> files was added to GUI.
>
>
> added:
>
> The set hidden parameter
> DoRBBlack = 0; # (0/1) check blacklisted mails on
> rebuildspamdb (default 0 - 1 = skip rebuild for notspam if black)
> removes all mails in the assp/notspam folder, which matches :
> noBlockingIPs, denySMTPConnectionsFromAlways, denySMTPConnectionsFrom and
> blackListedDomains
>
> Notice: if all of DoRBWhite, DoRBBlack and DoRBRed are enabled, the
> rebuild process will take ~12 times (or very much) longer than without
> setting these switches.
> Don't be confused. If .eml files were corrected by spam/ham
> reports, assp will process them correctly. But it may help to maintain the
> corpus from time to time.
>
>
>
> Thomas
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
> _______________________________________________
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
