[Bug 8062] [review] no URL makes uridnsbl rules "unrun"
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8062 Bug 8062 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8061] [review] Fix meta handling for $suppl_attrib->{rule_hits}
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8061 Bug 8061 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7987] DNSEval.pm,HashBL.pm,URILocalBL.pm: unnecessary use of rule_pending and rule_ready
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7987 Bug 7987 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8059] [review] Fix meta handling for URIDNSBL NS/A lookups
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8059 Bug 8059 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8060] [review] Fix meta handling for metas without dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8060 Bug 8060 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8065] disabled plugins cause undefined dependencies for meta rules
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8065 Bug 8065 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 Henrik Krohns changed: What|Removed |Added Resolution|--- |FIXED Status|REOPENED|RESOLVED --- Comment #94 from Henrik Krohns --- Third time's the charm? Sendingtrunk/lib/Mail/SpamAssassin/Plugin/Check.pm Sendingtrunk/t/basic_meta2.t Transmitting file data ..done Committing transaction... Committed revision 1905214. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8071] get_uri_detail_list & missing links
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8071 --- Comment #3 from Pascal --- I mean that when there is an html code (containing links) inside comment tags, the function does not identify the links. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8071] get_uri_detail_list & missing links
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8071 John Hardin changed: What|Removed |Added CC||jhar...@impsec.org --- Comment #2 from John Hardin --- Those are HTML comments, not links or tags, and they have no place in the uri list. If you want to see that content, use a rawbody rule. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8072] AWL incorrectly resets message count
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072 Paul Stead changed: What|Removed |Added CC||paul.st...@gmail.com --- Comment #4 from Paul Stead --- It's pretty late here, but as I recall, the reason for the "set msgcount = ?" is because if a message is re-learned by AWL, the totscore is adjusted positively/negatively, however the msgcount needs to stay the same to make the calculation work. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #93 from Kevin A. McGrail --- +1 from me too because Wolfgang's initial discovery, the subsequent testing, and the patch work from Henrik look good. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7943] TxRep gives nonsensical scores?
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7943 --- Comment #3 from Matija Nalis --- Thanks Paul for your efforts! Unfortunately, I hadn't had a change to try your fix yet, as I've had to drop TxRep in favor of AWL in early 2022 in order to make production functional again, and not having time to try to test and bring it back... However, since AWL with SQL backend also seems buggy, and I'll have to invest time to rebuild database anyway, I think I might give TxRep another try. Might be worth doing it before 4.0 gets out, in order to iron out bugs there and save other people some headaches. However, I've found another bug in SQLBasedAddrList.pm which seems it might be affecting not only AWL but TxRep as well: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072 Could you take a look there if that would affect TxRep as well? Also, can I just grab SQLBasedAddrList.pm and TxRep.pm from trunk; or do I have to go full-trunk (which would be much harder to swallow as I can basically test it only by deploying it in production) ? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8072] AWL incorrectly resets message count
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072
--- Comment #3 from Matija Nalis ---
Interestingly, code has following comment:
"NOTE: This code uses a self referential SQL call (ie set foo = foo + 1) which
is supported by most modern database backends, but not everything calling
itself a SQL database."
That might explain historic reasons why it uses "set msgcount = ?" instead of
"set msgcount = msgcount + 1", WHERE IT NOT FOR it also using "totscore =
totscore + ?" at the same time (in which case, such non-compliant SQL database
would fail anyway).
There are 3 places in that file using "msgcount = ?" which could easily be
fixed by converting them to "msgcount = msgcount + 1" (and removing that
parameter from SQL argument list).
While that would be welcome and make this specific problem go away (or at least
reduce its severity by several orders of magnitude), what makes me uneasy is
the fact that $entry->{msgcount} (and thus probably whole $entry) gets
corrupted (e.g. probably replaced by some other entry. Brings me back
nightmares from my previous life about non-reentrant C library functions used
in threaded code).
Or has such $entry corruption bug have been found and fixed since 3.4.6
release?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8072] AWL incorrectly resets message count
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072
--- Comment #2 from Matija Nalis ---
the SQL statement seems to be in the Mail/SpamAssassin/SQLBasedAddrList.pm
which seems to set msgcount to value specified by script:
my $sql = "UPDATE $self->{tablename} ".
"SET msgcount = ?, totscore = totscore + ? ".
"WHERE username = ? AND email = ?";
I haven't tried 4.0/trunk for that, but looking in SVN
https://svn.apache.org/viewvc/spamassassin/trunk/lib/Mail/SpamAssassin/SQLBasedAddrList.pm?revision=1903510=markup#l361
it seems to use similar logic.
Is there a reason why the SA code specifies the final msgcount score manually?
Sure, original bug leading to that should be found and fixed, but wouldn't
letting SQL update the counter avoid such horrific pitfalls, as we presumably
always work on message-by-message basis?
E.g. something like:
my $sql = "UPDATE $self->{tablename} ".
"SET msgcount = msgcount + 1, totscore = totscore + ? ".
"WHERE username = ? AND email = ?";
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8072] AWL incorrectly resets message count
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072 Matija Nalis changed: What|Removed |Added CC||mnalis-sa...@voyager.hr --- Comment #1 from Matija Nalis --- Some 20 minutes earlier, I find possibly related AWL update - for same email address, but different IP range, the msgcount was increased from 3 to 4 (and the above obviously buggy change sets msgcount to 5: /*!*/; # at 579668809 # at 579669029 #221104 9:34:03 server id 1 end_log_pos 579669029 CRC32 0xff91a46d Annotate_rows: #Q> UPDATE awl SET msgcount = '4', totscore = totscore + '1.39' WHERE username = 'amavis' AND email = 'redac...@example.com' AND signedby = 'example-com.20210112.gappssmtp.com' AND ip > #221104 9:34:03 server id 1 end_log_pos 579669099 CRC32 0xa7d7c1f0 Table_map: `spamassassin`.`awl` mapped to number 22 # at 579669099 #221104 9:34:03 server id 1 end_log_pos 579669313 CRC32 0x460a91e9 Update_rows: table id 22 flags: STMT_END_F ### UPDATE `spamassassin`.`awl` ### WHERE ### @1='amavis' ### @2='redac...@example.com' ### @3='195.190' ### @4=3 ### @5=9.794 ### @6='example-com.20210112.gappssmtp.com' ### @7=1667550842 ### SET ### @1='amavis' ### @2='redac...@example.com' ### @3='195.190' ### @4=4 ### @5=11.184 ### @6='example-com.20210112.gappssmtp.com' ### @7=1667550843 # Number of rows: 1 # at 579669313 #221104 9:34:03 server id 1 end_log_pos 579669344 CRC32 0xb73804b3Xid = 8550665 COMMIT/*!*/; -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8072] New: AWL incorrectly resets message count
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8072 Bug ID: 8072 Summary: AWL incorrectly resets message count Product: Spamassassin Version: 3.4.6 Hardware: PC OS: Linux Status: NEW Severity: normal Priority: P2 Component: Plugins Assignee: dev@spamassassin.apache.org Reporter: mnalis-sa...@voyager.hr Target Milestone: Undefined I've had situations when some legit messages suddenly have spam score increase of several hundreds after one email. Trying to debug, I've tracked it down to AWL module (using mysql backend). First looking at the mail log, it seems AWL is changing minimally as expected, and then suddenly it jumps HUGELY (and than trail off slowly again): AWL=0.016 AWL=0.016 AWL=0.016 AWL=-0.085 AWL=0.016 AWL=0.937 AWL=1.406 AWL=1.627 AWL=318.182 AWL=265.033 AWL=176.606 [...] sifting through mysqlbinlog to find what happened I find this piece of evidence: /*!*/; # at 585992025 # at 585992247 #221104 9:54:16 server id 1 end_log_pos 585992247 CRC32 0x3d2066fc Annotate_rows: #Q> UPDATE awl SET msgcount = '5', totscore = totscore + '-0.459' WHERE username = 'amavis' AND email = 'redac...@example.com' AND signedby = 'example-com.20210112.gappssmtp.com' AND i> #221104 9:54:16 server id 1 end_log_pos 585992317 CRC32 0x3463013b Table_map: `spamassassin`.`awl` mapped to number 22 # at 585992317 #221104 9:54:16 server id 1 end_log_pos 585992531 CRC32 0xd7fcc32a Update_rows: table id 22 flags: STMT_END_F ### UPDATE `spamassassin`.`awl` ### WHERE ### @1='amavis' ### @2='redac...@example.com' ### @3='213.191' ### @4=2392 ### @5=3189.25 ### @6='example-com.20210112.gappssmtp.com' ### @7=1667550625 ### SET ### @1='amavis' ### @2='redac...@example.com' ### @3='213.191' ### @4=5 ### @5=3188.79 ### @6='example-com.20210112.gappssmtp.com' ### @7=1667552056 # Number of rows: 1 # at 585992531 #221104 9:54:16 server id 1 end_log_pos 585992562 CRC32 0xff0bfeb5Xid = 8567075 COMMIT/*!*/; So it seems that while totscore is correctly increased, message count gets reset -- instead of increasing from 2392 to 2393, it gets reset to 5 -- which obviously leads to all further uses to get hugely increased AWL! That is spamassassin 3.4.6-1 from Debian Bullseye. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #92 from Bill Cole --- +1 to commit. I'm slightly uneasy about my (shallow) depth of understanding of this issue, but I trust those with a deeper understanding and really want to get 4.0 released. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735
--- Comment #91 from Henrik Krohns ---
I guess let's vote on the patch then. +1
Note that I did one small if tweak, $mp to $mt
if (!exists $mp->{$deprule}) {
-->
if (!exists $mt->{$deprule}) {
Basically it's the exact same result and doesn't affect anything, but more
clear on what we do. Too lazy to upload new patch.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735
--- Comment #90 from Wolfgang Breyha ---
Looking at the plain results that looks good. Same score and same
(Total Subtest Hits: 208 / Deduplicated Total Hits: 133)
I also added a dbg output to the loop so it looks like
foreach my $rulename (keys %$mp) {
foreach my $deprule (@{$md->{$rulename}||[]}) {
if (!exists $mp->{$deprule}) {
dbg("rules: $rulename: marking $deprule ready");
$h->{$deprule} ||= 0;
The output looks reasonable as far as I checked the rules mentioned in my
samples.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #89 from Henrik Krohns --- Created attachment 5855 --> https://bz.apache.org/SpamAssassin/attachment.cgi?id=5855=edit Meta patch for r1905203, fix finishing metas ^ You are very right.. I already finished a quick patch that marks all non-meta rules as finished for the loop. Everything is now done in do_meta_tests. Let me know how this patch works.. atleast tests pass here. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #88 from Wolfgang Breyha --- Just brainstorming... At the point the do_meta_rules()/finish_meta_rules() block is reached: If all unavailable non-meta rules/tags are set to a "0" result before that point shouldn't that clear all the dependencies except meta<->meta? And meta<->meta should be resolved in do_meta_rules() correctly then. And for DKIMDOMAIN and SENDERDOMAIN... it is highly unlikely IMO that those get a result at a later point if header scanning didn't bring them up. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8059] [review] Fix meta handling for URIDNSBL NS/A lookups
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8059 Bug 8059 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 Henrik Krohns changed: What|Removed |Added Resolution|FIXED |--- Status|RESOLVED|REOPENED --- Comment #87 from Henrik Krohns --- (In reply to Wolfgang Breyha from comment #86) > > In this Sample no DKIM-Sig is available... so DKIMDOMAIN and all __DKIMWL_* > and DKIMWL_* stay "unrun" and a missing dependency. > > Every meta rule which depends on them somehow is run only once in > finish_meta_tests(). And if a meta depends on a meta which depends on > DKIMDOMAIN/ff somehow I get random results (race condition) Ok thanks I think I get it now.. Here's a sample test case: meta __TEST_META_I (1 || TEST_DISABLED3) meta TEST_META_I __TEST_META_I Both should always hit in this case. If unrun TEST_DISABLED3 postpones things to finish_meta_tests, and the foreach with it's random hash order processes TEST_META_I first, obviously things will get wrong. I'll see if can figure out a better logic, maybe just migrate finish_meta_tests final logic into do_meta_tests somehow. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8062] [review] no URL makes uridnsbl rules "unrun"
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8062 Bug 8062 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7987] DNSEval.pm,HashBL.pm,URILocalBL.pm: unnecessary use of rule_pending and rule_ready
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7987 Bug 7987 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8061] [review] Fix meta handling for $suppl_attrib->{rule_hits}
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8061 Bug 8061 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8060] [review] Fix meta handling for metas without dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8060 Bug 8060 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8065] disabled plugins cause undefined dependencies for meta rules
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8065 Bug 8065 depends on bug 7735, which changed state. Bug 7735 Summary: Meta rules need to handle missing/unrun dependencies https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #86 from Wolfgang Breyha --- regarding the revert... sorry, my fault. The part in Check.pm was moved up (out of my sight). regarding the new troubles: With my new sample I again see a lot of meta rules delayed to the point that they get only handled once in finish_meta_tests() leading to random results dependent from perl hashing. In this Sample no DKIM-Sig is available... so DKIMDOMAIN and all __DKIMWL_* and DKIMWL_* stay "unrun" and a missing dependency. Every meta rule which depends on them somehow is run only once in finish_meta_tests(). And if a meta depends on a meta which depends on DKIMDOMAIN/ff somehow I get random results (race condition) I'm wondering why this worked on the patched rc3. An undetermined SENDERDOMAIN seems to block a lot as well. At least I see the same dbg: check: tagrun - tag SENDERDOMAIN is still blocking action 2, 3, 4, 5, 11, 13 as for DKIMDOMAIN dbg: check: tagrun - tag DKIMDOMAIN is still blocking action 0, 1, 2, 3, 5, 6, 12, 13 But since non of my meta rules have dependencies on SENDERDOMAIN yet I can't tell for sure. But I see some PDS_SPF_* and JMQ_SPF_* rules blocked in the standard ruleset. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #85 from Henrik Krohns --- (In reply to Wolfgang Breyha from comment #84) > I noticed that the "revert" patch in comment 71 > also reverted parts (but not all) of the fix for #8060. Was this intentional? Not sure what you are referring to, all of 8060 is still committed from what I see. A detailed hint would help. :-) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 7735] Meta rules need to handle missing/unrun dependencies
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7735 --- Comment #84 from Wolfgang Breyha --- I noticed that the "revert" patch in comment 71 also reverted parts (but not all) of the fix for #8060. Was this intentional? And I'm sorry to say that I'm currently struggling with another sample where I get bad results with the current trunk compared to rc3 with all the fixes we had for it. I'm still looking into it. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8071] get_uri_detail_list & missing links
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8071 Pascal changed: What|Removed |Added CC||pascal@halogen-consulting.c ||om --- Comment #1 from Pascal --- my bugfix: before using get_uri_detail_list, I remove these tags: s/(?:<\!--)?<\!\[endif\]-->// for @lines; s/<\!--\[if [\(\)\|\!\w\s]+\]>(?:<\!-->)?// for @lines; -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8071] New: get_uri_detail_list & missing links
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8071 Bug ID: 8071 Summary: get_uri_detail_list & missing links Product: Spamassassin Version: 3.4.6 Hardware: PC OS: Linux Status: NEW Severity: normal Priority: P2 Component: Libraries Assignee: dev@spamassassin.apache.org Reporter: pas...@halogen-consulting.com Target Milestone: Undefined there are links which are not identified when the html code contains tags like that:
[Bug 8024] get_uri_detail_list &
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8024 Pascal changed: What|Removed |Added Priority|P2 |P3 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 8024] get_uri_detail_list &
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8024 Pascal changed: What|Removed |Added Component|Tools |Libraries CC||pascal@halogen-consulting.c ||om -- You are receiving this mail because: You are the assignee for the bug.
