updateDNS.sh on sa-vm1.apache.org - DNS updates disabled

2017-11-03 Thread noreply 

3.3.3.updates (TXT) -> \"1814142\"

File /usr/local/bin/updateDNS.disabled exists, not updating DNS.

Re: mailspike in 72_scores

2017-11-03 Thread Dave Jones 

On 11/02/2017 08:11 AM, Kevin A. McGrail wrote:

On 11/2/2017 6:44 AM, Merijn van den Kroonenberg wrote:
While debugging the score generation problem, I encountered another 
weird case in 72_scores.cf.
It is about the mailspike rules which are in 50_scores so should not 
be in 72_scores.

There was already an existing issue about this:
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6400
I found the cause of this and attached a patch to bugzilla. but since 
its already a very old issue I wonder if someone will ever look it it 
anymore ;)
Thanks!  I did have a look at it but I don't want to interfere with 
David's work.  David, could you make the commit for him on this bug?


Regards,
KAM



Merijn,

I am confused as to what is trying to be accomplished with the patch on 
but 6400.  Maybe I am misunderstanding something.  It looks like these 
MSPIKE scores need to remain in 50_scores.cf since they are tied to a 
specific SA version check > 3.4.0.


Do we need to remove them from 72_scores.cf, note that in the bug then 
close it?


Dave

Re: Good 72_scores.cf

2017-11-03 Thread Dave Jones 

On 11/03/2017 08:13 AM, Merijn van den Kroonenberg wrote:



cd /tmp
wget http://sa-update.ena.com/1814048.tar.gz
wget http://sa-update.ena.com/1814048.tar.gz.sha1
wget http://sa-update.ena.com/1814048.tar.gz.asc
sa-update -v --install 1814048.tar.gz

I installed this ruleset on a couple of my test instances and it SA
linted OK.

We are back to a good sized 72_scores.cf:

[root@djones5 updates_spamassassin_org]# ll 72_scores.cf
-rw-r--r-- 1 root root 9810 Nov  3 07:47 72_scores.cf


But this is still the copied-over version of 72_scores.cf from mid March,
right?



Shouldn't be.  There are two places where rulesets are generated:

1. Daily rule promotion
2. Daily masscheck validation

The masscheck validation was where I had the temp fix to force the 
72_scores.cf from March.  However, I just removed that temp fix so we 
are now back to the latest 72_scores.cf being in the ruleset .tar.gz 
file.  DNS updates are still disabled so sa-update won't do anything 
until we are sure the 72_scores.cf is good.


I have to commit fixes to rules and rule processing script late in the 
evening US time so as to not mess up the masscheck processing.


I will look at the merge-scoresets tomorrow.




Yeah it would be good if the update system was resistant to this kind of
seemingly innocent changes.



Need to test this on the same OS version as the server:

root@sa-vm1:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.3 LTS
Release:16.04
Codename:   xenial

I will be doing some work on this over the weekend and will do some test
greps.  Just don't want to commit things at the wrong time to mess up
the masscheck processing.


Personally I think the grep locale issue is of lower priority than getting
things running again. Its now in bug 7497 and no need to hurry as it has
the potential to break other, unforseen, stuff.

Re: Good 72_scores.cf

2017-11-03 Thread Merijn van den Kroonenberg 

> cd /tmp
> wget http://sa-update.ena.com/1814048.tar.gz
> wget http://sa-update.ena.com/1814048.tar.gz.sha1
> wget http://sa-update.ena.com/1814048.tar.gz.asc
> sa-update -v --install 1814048.tar.gz
>
> I installed this ruleset on a couple of my test instances and it SA
> linted OK.
>
> We are back to a good sized 72_scores.cf:
>
> [root@djones5 updates_spamassassin_org]# ll 72_scores.cf
> -rw-r--r-- 1 root root 9810 Nov  3 07:47 72_scores.cf

But this is still the copied-over version of 72_scores.cf from mid March,
right?


>> Yeah it would be good if the update system was resistant to this kind of
>> seemingly innocent changes.
>>
>
> Need to test this on the same OS version as the server:
>
> root@sa-vm1:/# lsb_release -a
> No LSB modules are available.
> Distributor ID:   Ubuntu
> Description:  Ubuntu 16.04.3 LTS
> Release:  16.04
> Codename: xenial
>
> I will be doing some work on this over the weekend and will do some test
> greps.  Just don't want to commit things at the wrong time to mess up
> the masscheck processing.

Personally I think the grep locale issue is of lower priority than getting
things running again. Its now in bug 7497 and no need to hurry as it has
the potential to break other, unforseen, stuff.

>

Re: Good 72_scores.cf

2017-11-03 Thread Dave Jones 

On 11/03/2017 07:33 AM, Merijn van den Kroonenberg wrote:

On 11/3/2017 8:21 AM, Merijn van den Kroonenberg wrote:

we only
have the 'random removal' issue left. But I am confident the
merge-scoresets fix will take care of that.

I don't know the random removal issue you are referring to. Can you
detail it more?  Perhaps in bugzilla?


David Jones has the details and I believe its one of the next things on
his list. I just mentioned it here so people do not get the idea all is
good now.



Sure.  But this was a big step and we aren't completely done.  This is a 
very slow process since if anyone commits any fixes, then it throws off 
the masscheck processing.  I was wanting to get some good rulesets 
generating again but this is drawing out forever.


We did get ruleset 1814048.tar.gz promoted last night but it's going to 
take 2 more days of good masschecks before everything is rolling along 
properly -- I understand we still have to look at/fix the 
merge-scoresets before enabling DNS updates.  DNS updates are still 
disabled so sa-update won't do anything.


cd /tmp
wget http://sa-update.ena.com/1814048.tar.gz
wget http://sa-update.ena.com/1814048.tar.gz.sha1
wget http://sa-update.ena.com/1814048.tar.gz.asc
sa-update -v --install 1814048.tar.gz

I installed this ruleset on a couple of my test instances and it SA 
linted OK.


We are back to a good sized 72_scores.cf:

[root@djones5 updates_spamassassin_org]# ll 72_scores.cf
-rw-r--r-- 1 root root 9810 Nov  3 07:47 72_scores.cf




NOTE: we can do things with bugzilla if we want to have sysadmins or
something like that listed as a component or whatever.


That would probably be a good idea. Actually I was suprised the whole
update issue was not in bugzilla yet. It makes it much easier to work with
multiple people on the same issue.



Never had anyone else help until now.  :)






Now, the interesting thing is does changing the system to en_US fix the
issue if we revert my change to that rule?

No need to revert the change. You can just test this by hand by running
the grep command on the old 72_active.cf from one of the tgz files (from
shell).


Can you document the replication steps at
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7497


Sure, I just saw the ticket being created, good idea.



I was speaking more to the fact that I'd like to add the multilanguage
descriptions back.  I documented it in bugzilla especially if someone
else commits a rule with similar.


Yeah it would be good if the update system was resistant to this kind of
seemingly innocent changes.



Need to test this on the same OS version as the server:

root@sa-vm1:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.3 LTS
Release:16.04
Codename:   xenial

I will be doing some work on this over the weekend and will do some test 
greps.  Just don't want to commit things at the wrong time to mess up 
the masscheck processing.




Regards,

KAM

Re: Good 72_scores.cf

2017-11-03 Thread Kevin A. McGrail 

On 11/3/2017 8:33 AM, Merijn van den Kroonenberg wrote:

I was speaking more to the fact that I'd like to add the multilanguage
descriptions back.  I documented it in bugzilla especially if someone
else commits a rule with similar.

Yeah it would be good if the update system was resistant to this kind of
seemingly innocent changes.

It's not a recent change.  It worked with the old system.  Same rule, 
breaks on the new system, hence my thought around the LANG=en_US

Re: Good 72_scores.cf

2017-11-03 Thread Merijn van den Kroonenberg 
> On 11/3/2017 8:21 AM, Merijn van den Kroonenberg wrote:
>> we only
>> have the 'random removal' issue left. But I am confident the
>> merge-scoresets fix will take care of that.
> I don't know the random removal issue you are referring to. Can you
> detail it more?  Perhaps in bugzilla?

David Jones has the details and I believe its one of the next things on
his list. I just mentioned it here so people do not get the idea all is
good now.

>
> NOTE: we can do things with bugzilla if we want to have sysadmins or
> something like that listed as a component or whatever.

That would probably be a good idea. Actually I was suprised the whole
update issue was not in bugzilla yet. It makes it much easier to work with
multiple people on the same issue.

>
>>
>>> Now, the interesting thing is does changing the system to en_US fix the
>>> issue if we revert my change to that rule?
>> No need to revert the change. You can just test this by hand by running
>> the grep command on the old 72_active.cf from one of the tgz files (from
>> shell).
>
> Can you document the replication steps at
> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7497

Sure, I just saw the ticket being created, good idea.

>
> I was speaking more to the fact that I'd like to add the multilanguage
> descriptions back.  I documented it in bugzilla especially if someone
> else commits a rule with similar.

Yeah it would be good if the update system was resistant to this kind of
seemingly innocent changes.

>
> Regards,
>
> KAM
>
>

Re: Good 72_scores.cf

2017-11-03 Thread Merijn van den Kroonenberg 
> On 11/3/2017 7:50 AM, Dave Jones wrote:
>> Woo hoo!  Looks like we have a full 72_scores after the fix to
>> MILLION_USD.  Thanks Merijn!

Great! So fix 1 of 2 completed! With the truncation issue down we only
have the 'random removal' issue left. But I am confident the
merge-scoresets fix will take care of that.

> Now, the interesting thing is does changing the system to en_US fix the
> issue if we revert my change to that rule?

No need to revert the change. You can just test this by hand by running
the grep command on the old 72_active.cf from one of the tgz files (from
shell).

Re: Good 72_scores.cf

2017-11-03 Thread Kevin A. McGrail 

On 11/3/2017 7:50 AM, Dave Jones wrote:
Woo hoo!  Looks like we have a full 72_scores after the fix to 
MILLION_USD.  Thanks Merijn!
Now, the interesting thing is does changing the system to en_US fix the 
issue if we revert my change to that rule?



 Forwarded Message 
Subject: svn commit: r1814132 - in 
/spamassassin/trunk/rulesrc/scores: 72_scores.cf scores-set0 
scores-set1 scores-set2 scores-set3 stats-set0 stats-set1

Date: Fri, 03 Nov 2017 02:56:50 -
From: spamassassin_r...@apache.org
Reply-To: SpamAssassin Dev 
To: comm...@spamassassin.apache.org



Author: spamassassin_role
Date: Fri Nov  3 02:56:50 2017
New Revision: 1814132

URL: http://svn.apache.org/viewvc?rev=1814132=rev
Log:
updated scores for revision 1814048 active rules added since last 
mass-check


Modified:
    spamassassin/trunk/rulesrc/scores/72_scores.cf
    spamassassin/trunk/rulesrc/scores/scores-set0
    spamassassin/trunk/rulesrc/scores/scores-set1
    spamassassin/trunk/rulesrc/scores/scores-set2
    spamassassin/trunk/rulesrc/scores/scores-set3
    spamassassin/trunk/rulesrc/scores/stats-set0
    spamassassin/trunk/rulesrc/scores/stats-set1

Modified: spamassassin/trunk/rulesrc/scores/72_scores.cf
URL: 
http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?rev=1814132=1814131=1814132=diff
== 


--- spamassassin/trunk/rulesrc/scores/72_scores.cf (original)
+++ spamassassin/trunk/rulesrc/scores/72_scores.cf Fri Nov  3 02:56:50 
2017

@@ -1,48 +1,81 @@
-score AC_BR_BONANZA  0.001 2.676 0.001 2.676
-score AC_DIV_BONANZA 0.001 1.599 0.001 1.599
-score ADVANCE_FEE_2_NEW_MONEY    2.799 2.044 2.799 2.044
-score ADVANCE_FEE_3_NEW  1.799 1.520 1.799 1.520
-score ADVANCE_FEE_3_NEW_MONEY    3.099 3.100 3.099 3.100
-score ADVANCE_FEE_4_NEW  0.001 0.001 0.001 0.001
+score AC_BR_BONANZA  0.001 2.677 0.001 2.677
+score AC_DIV_BONANZA 0.351 1.599 0.351 1.599
+score ADVANCE_FEE_2_NEW_MONEY    1.675 2.041 1.675 2.041
+score ADVANCE_FEE_3_NEW  1.999 1.519 1.999 1.519
+score ADVANCE_FEE_3_NEW_MONEY    3.100 3.099 3.100 3.099
 score ADVANCE_FEE_4_NEW_MONEY    3.099 2.999 3.099 2.999
-score ADVANCE_FEE_5_NEW_FRM_MNY  4.099 0.264 4.099 0.264
-score ADVANCE_FEE_5_NEW_MONEY    2.526 1.766 2.526 1.766
+score ADVANCE_FEE_5_NEW_FRM_MNY  4.199 0.450 4.199 0.450
+score ADVANCE_FEE_5_NEW_MONEY    1.582 1.892 1.582 1.892
 score APOSTROPHE_TOCC    3.100 3.099 3.100 3.099
-score AXB_XMAILER_MIMEOLE_OL_024C2   2.397 0.586 2.397 0.586
-score AXB_X_AOL_SEZ_S    3.599 3.499 3.599 3.499
-score BODY_SINGLE_URI    0.261 1.014 0.261 1.014
-score BODY_SINGLE_WORD   0.339 2.563 0.339 2.563
-score BODY_URI_ONLY  3.199 1.551 3.199 1.551
-score BOGUS_MSM_HDRS 3.099 3.099 3.099 3.099
-score CK_HELO_DYNAMIC_SPLIT_IP   2.399 0.109 2.399 0.109
-score CK_HELO_GENERIC    2.601 2.822 2.601 2.822
-score DATE_IN_FUTURE_96_Q    2.999 2.559 2.999 2.559
-score DEAR_BENEFICIARY   3.099 3.099 3.099 3.099
-score FILL_THIS_FORM 0.198 1.631 0.198 1.631
-score FORM_FRAUD 2.899 1.149 2.899 1.149
-score FORM_FRAUD_3   2.800 1.295 2.800 1.295
-score FORM_FRAUD_5   2.471 2.756 2.471 2.756
-score FREEMAIL_FORGED_FROMDOMAIN 0.001 0.141 0.001 0.141
+score AXB_XMAILER_MIMEOLE_OL_024C2   0.001 0.587 0.001 0.587
+score AXB_X_AOL_SEZ_S    3.600 3.499 3.600 3.499
+score BODY_SINGLE_URI    0.124 1.719 0.124 1.719
+score BODY_SINGLE_WORD   0.477 1.857 0.477 1.857
+score BODY_URI_ONLY  3.299 1.552 3.299 1.552
+score BOGUS_MSM_HDRS 0.027 3.199 0.027 3.199
+score CK_HELO_DYNAMIC_SPLIT_IP   2.601 0.109 2.601 0.109
+score CK_HELO_GENERIC    2.601 2.818 2.601 2.818
+score DATE_IN_FUTURE_96_Q    3.099 2.599 3.099 2.599
+score DEAR_BENEFICIARY   3.299 3.099 3.299 3.099
+score FILL_THIS_FORM 0.197 1.645 0.197 1.645
+score FORM_FRAUD 2.178 1.133 2.178 1.133
+score FORM_FRAUD_3   2.899 1.703 2.899 1.703
+score FORM_FRAUD_5   1.528 1.827 1.528 1.827
+score FROM_IN_TO_AND_SUBJ    0.084 1.199 0.084 1.199
 score FROM_MISSPACED 0.001 0.001 0.001 0.001
-score FROM_MISSP_DYNIP   0.688 2.800 0.688 2.800
-score FROM_MISSP_EH_MATCH    2.785 3.013 2.785 3.013
-score FROM_MISSP_FREEMAIL    2.309 2.789 2.309 2.789
-score FROM_MISSP_MSFT    0.001 0.000 0.001 0.000
-score FROM_MISSP_REPLYTO 0.001 0.000 0.001 0.000
-score FROM_MISSP_TO_UNDISC   2.429 1.368 2.429 1.368
-score FROM_MISSP_USER  

Good 72_scores.cf

2017-11-03 Thread Dave Jones 
Woo hoo!  Looks like we have a full 72_scores after the fix to 
MILLION_USD.  Thanks Merijn!



 Forwarded Message 
Subject: 	svn commit: r1814132 - in /spamassassin/trunk/rulesrc/scores: 
72_scores.cf scores-set0 scores-set1 scores-set2 scores-set3 stats-set0 
stats-set1

Date:   Fri, 03 Nov 2017 02:56:50 -
From:   spamassassin_r...@apache.org
Reply-To:   SpamAssassin Dev 
To: comm...@spamassassin.apache.org



Author: spamassassin_role
Date: Fri Nov  3 02:56:50 2017
New Revision: 1814132

URL: http://svn.apache.org/viewvc?rev=1814132=rev
Log:
updated scores for revision 1814048 active rules added since last mass-check

Modified:
spamassassin/trunk/rulesrc/scores/72_scores.cf
spamassassin/trunk/rulesrc/scores/scores-set0
spamassassin/trunk/rulesrc/scores/scores-set1
spamassassin/trunk/rulesrc/scores/scores-set2
spamassassin/trunk/rulesrc/scores/scores-set3
spamassassin/trunk/rulesrc/scores/stats-set0
spamassassin/trunk/rulesrc/scores/stats-set1

Modified: spamassassin/trunk/rulesrc/scores/72_scores.cf
URL: 
http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?rev=1814132=1814131=1814132=diff
==
--- spamassassin/trunk/rulesrc/scores/72_scores.cf (original)
+++ spamassassin/trunk/rulesrc/scores/72_scores.cf Fri Nov  3 02:56:50 2017
@@ -1,48 +1,81 @@
-score AC_BR_BONANZA  0.001 2.676 0.001 2.676
-score AC_DIV_BONANZA 0.001 1.599 0.001 1.599
-score ADVANCE_FEE_2_NEW_MONEY2.799 2.044 2.799 2.044
-score ADVANCE_FEE_3_NEW  1.799 1.520 1.799 1.520
-score ADVANCE_FEE_3_NEW_MONEY3.099 3.100 3.099 3.100
-score ADVANCE_FEE_4_NEW  0.001 0.001 0.001 0.001
+score AC_BR_BONANZA  0.001 2.677 0.001 2.677
+score AC_DIV_BONANZA 0.351 1.599 0.351 1.599
+score ADVANCE_FEE_2_NEW_MONEY1.675 2.041 1.675 2.041
+score ADVANCE_FEE_3_NEW  1.999 1.519 1.999 1.519
+score ADVANCE_FEE_3_NEW_MONEY3.100 3.099 3.100 3.099
 score ADVANCE_FEE_4_NEW_MONEY3.099 2.999 3.099 2.999
-score ADVANCE_FEE_5_NEW_FRM_MNY  4.099 0.264 4.099 0.264
-score ADVANCE_FEE_5_NEW_MONEY2.526 1.766 2.526 1.766
+score ADVANCE_FEE_5_NEW_FRM_MNY  4.199 0.450 4.199 0.450
+score ADVANCE_FEE_5_NEW_MONEY1.582 1.892 1.582 1.892
 score APOSTROPHE_TOCC3.100 3.099 3.100 3.099
-score AXB_XMAILER_MIMEOLE_OL_024C2   2.397 0.586 2.397 0.586
-score AXB_X_AOL_SEZ_S3.599 3.499 3.599 3.499
-score BODY_SINGLE_URI0.261 1.014 0.261 1.014
-score BODY_SINGLE_WORD   0.339 2.563 0.339 2.563
-score BODY_URI_ONLY  3.199 1.551 3.199 1.551
-score BOGUS_MSM_HDRS 3.099 3.099 3.099 3.099
-score CK_HELO_DYNAMIC_SPLIT_IP   2.399 0.109 2.399 0.109
-score CK_HELO_GENERIC2.601 2.822 2.601 2.822
-score DATE_IN_FUTURE_96_Q2.999 2.559 2.999 2.559
-score DEAR_BENEFICIARY   3.099 3.099 3.099 3.099
-score FILL_THIS_FORM 0.198 1.631 0.198 1.631
-score FORM_FRAUD 2.899 1.149 2.899 1.149
-score FORM_FRAUD_3   2.800 1.295 2.800 1.295
-score FORM_FRAUD_5   2.471 2.756 2.471 2.756
-score FREEMAIL_FORGED_FROMDOMAIN 0.001 0.141 0.001 0.141
+score AXB_XMAILER_MIMEOLE_OL_024C2   0.001 0.587 0.001 0.587
+score AXB_X_AOL_SEZ_S3.600 3.499 3.600 3.499
+score BODY_SINGLE_URI0.124 1.719 0.124 1.719
+score BODY_SINGLE_WORD   0.477 1.857 0.477 1.857
+score BODY_URI_ONLY  3.299 1.552 3.299 1.552
+score BOGUS_MSM_HDRS 0.027 3.199 0.027 3.199
+score CK_HELO_DYNAMIC_SPLIT_IP   2.601 0.109 2.601 0.109
+score CK_HELO_GENERIC2.601 2.818 2.601 2.818
+score DATE_IN_FUTURE_96_Q3.099 2.599 3.099 2.599
+score DEAR_BENEFICIARY   3.299 3.099 3.299 3.099
+score FILL_THIS_FORM 0.197 1.645 0.197 1.645
+score FORM_FRAUD 2.178 1.133 2.178 1.133
+score FORM_FRAUD_3   2.899 1.703 2.899 1.703
+score FORM_FRAUD_5   1.528 1.827 1.528 1.827
+score FROM_IN_TO_AND_SUBJ0.084 1.199 0.084 1.199
 score FROM_MISSPACED 0.001 0.001 0.001 0.001
-score FROM_MISSP_DYNIP   0.688 2.800 0.688 2.800
-score FROM_MISSP_EH_MATCH2.785 3.013 2.785 3.013
-score FROM_MISSP_FREEMAIL2.309 2.789 2.309 2.789
-score FROM_MISSP_MSFT0.001 0.000 0.001 0.000
-score FROM_MISSP_REPLYTO 0.001 0.000 0.001 0.000
-score FROM_MISSP_TO_UNDISC   2.429 1.368 2.429 1.368
-score FROM_MISSP_USER0.001 0.000 0.001 0.000
+score FROM_MISSP_DYNIP   1.970 2.899 1.970 2.899
+score FROM_MISSP_EH_MATCH1.100 3.011 1.100 3.011
+score

Cron <automc@sa-vm1> ~/svn/trunk/build/mkupdates/run_nightly | /usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt

2017-11-03 Thread Cron Daemon 
+ promote_active_rules
+ pwd
/usr/local/spamassassin/automc/svn/trunk
+ /usr/bin/perl build/mkupdates/listpromotable
HTTP get: http://ruleqa.spamassassin.org/1-days-ago?xml=1
HTTP get: http://ruleqa.spamassassin.org/2-days-ago?xml=1
no 'mcviewing', 'mcsubmitters' microformats on day 2
URL: http://ruleqa.spamassassin.org/2-days-ago?xml=1
+ exit 25