On 05/17/2018 04:09 PM, John Hardin wrote:

I notice from the RuleQA website that the masschecks from giovanni and llanga are consistently reported separately from everybody else's.

I wonder whether this is affecting the quality of masscheck - is this perhaps causing it to bounce back and forth between scores (or do something else that's suboptimal) based on what appears to it to be two separate and different masscheck corpora?


It looks like they are always behind on the SVN revision pulled down so they are actually not being counted/included in the masscheck processing.

The ena-week* corpora is the majority of the masscheck data.

Is this because of how those masschecks are being run or submitted, or is the masscheck infrastructure too strict (filename matching, submission cutoffs, etc.)?


Maybe they are running old versions of the automasscheck script that has some sort of delay in it between the downloading of the SVN staging area and the masscheck local processing. I understand that some may want to delay processing until electricity costs. The downloading of the staging area needs to happen per the documentation to get the correct SVN version of rules else it's a waste of resources.


It *feels* like those result sets are not coming in by the cutoff.


It's the SVN revision that is making them show up behind in their own section.

Unfortunately the ruleQA website doesn't expose a tool to report when the results were submitted, just which DateRev the masscheck was based on.

I'd say "perhaps we need to extend the cutoff a bit", but I have no idea ATM when those result sets are coming in so I have no idea how the cutoff would be adjusted.


I have a script that runs on sa-vm1 to track submissions so the sysadmins list gets notifications when we don't have enough for masscheck to run. Here's the output right now -- note the No's that don't match the SVN tagged rev:


Corpus total: 162, Old: 86, Recent: 38, New: 38

SVN tagged rev in nightly_mass_check:  1831759

New masscheck submission listings in the past day:
   SVN rev (Match) File Name (Date)
   1831759 (Yes) - spam-darxus.log (May 17 09:09)
   1831759 (Yes) - ham-darxus.log (May 17 09:08)
   1831759 (Yes) - ham-grenier.log (May 17 09:03)
   1831759 (Yes) - ham-ena-week0.log (May 17 10:42)
   1831759 (Yes) - spam-ena-week0.log (May 17 10:42)
   1831759 (Yes) - spam-ena-week3.log (May 17 11:14)
   1831759 (Yes) - spam-ena-week1.log (May 17 11:10)
   1831759 (Yes) - spam-jbrooks.log (May 17 10:06)
   1831759 (Yes) - spam-axb-generic.log (May 17 11:35)
   1831759 (Yes) - ham-mmiroslaw-mails-ham.log (May 17 11:11)
   1831759 (Yes) - ham-ena-week2.log (May 17 12:13)
   1831759 (Yes) - spam-axb-ham-misc.log (May 17 11:35)
   1831684  (No) - ham-giovanni.log (May 17 08:38)
   1831684  (No) - spam-llanga.log (May 17 08:45)
   1831759 (Yes) - spam-grenier.log (May 17 09:03)
   1831759 (Yes) - ham-axb-ham-misc.log (May 17 11:34)
   1831759 (Yes) - ham-mmiroslaw-mails-spam.log (May 17 11:11)
   1831759 (Yes) - ham-axb-generic.log (May 17 11:34)
   1831759 (Yes) - ham-axb-ninja.log (May 17 11:34)
   1831759 (Yes) - spam-ena-week4.log (May 17 12:01)
   1831759 (Yes) - spam-mmiroslaw-mails-spam.log (May 17 11:11)
   1831759 (Yes) - spam-axb-ninja.log (May 17 11:35)
   1831759 (Yes) - ham-jbrooks.log (May 17 10:06)
   1831759 (Yes) - spam-jarif.log (May 17 09:28)
   1831759 (Yes) - ham-thendrikx.log (May 17 09:03)
   1831759 (Yes) - ham-jarif.log (May 17 09:28)
   1831759 (Yes) - spam-mmiroslaw-mails-ham.log (May 17 11:11)
   1831684  (No) - ham-llanga.log (May 17 08:45)
   1831759 (Yes) - spam-axb-coi-bulk.log (May 17 11:34)
   1831759 (Yes) - ham-ena-week3.log (May 17 11:14)
   1831759 (Yes) - ham-ena-week1.log (May 17 11:10)
   1831759 (Yes) - spam-thendrikx.log (May 17 09:03)
   1831759 (Yes) - ham-sihde.log (May 17 10:11)
   1831759 (Yes) - spam-sihde.log (May 17 10:11)
   1831759 (Yes) - ham-ena-week4.log (May 17 12:01)
   1831684  (No) - spam-giovanni.log (May 17 08:38)
   1831759 (Yes) - ham-axb-coi-bulk.log (May 17 11:34)
   1831759 (Yes) - spam-ena-week2.log (May 17 12:13)

34/38 matches (17 ham, 17 spam)

Recent masscheck submission listings in the past week:
   SVN rev (Match) File Name (Date)
   1831456 (Yes) - spam-net-ena-week0.log (May 12 16:22)
   1831456 (Yes) - spam-net-axb-ninja.log (May 12 12:10)
   1831456 (Yes) - spam-net-grenier.log (May 12 09:05)
   1831456 (Yes) - spam-net-jbrooks.log (May 12 10:44)
   1831456 (Yes) - ham-net-jarif.log (May 12 09:29)
   1831456 (Yes) - spam-net-thendrikx.log (May 12 09:29)
   1831456 (Yes) - ham-net-ena-week2.log (May 12 15:34)
   1831456 (Yes) - ham-net-ena-week1.log (May 12 12:42)
   1831456 (Yes) - spam-net-ena-week4.log (May 12 14:06)
   1831456 (Yes) - spam-net-sihde.log (May 12 11:13)
   1831456 (Yes) - ham-net-axb-ham-misc.log (May 12 12:10)
   1831456 (Yes) - spam-net-ena-week3.log (May 12 15:52)
   1831456 (Yes) - ham-net-axb-coi-bulk.log (May 12 12:10)
   1830957  (No) - spam-net-llanga.log (May 12 19:55)
   1831456 (Yes) - spam-net-axb-generic.log (May 12 12:10)
   1831456 (Yes) - ham-net-axb-ninja.log (May 12 12:10)
   1831456 (Yes) - ham-net-ena-week4.log (May 12 14:05)
   1830957  (No) - spam-net-giovanni.log (May 12 08:44)
   1831456 (Yes) - ham-net-grenier.log (May 12 09:05)
   1831456 (Yes) - ham-net-jbrooks.log (May 12 10:44)
   1831456 (Yes) - spam-net-mmiroslaw-mails-ham.log (May 12 11:42)
   1831456 (Yes) - ham-net-ena-week3.log (May 12 15:52)
   1831456 (Yes) - ham-net-mmiroslaw-mails-spam.log (May 12 11:42)
   1831456 (Yes) - ham-net-sihde.log (May 12 11:13)
   1830957  (No) - ham-net-llanga.log (May 12 19:55)
   1831456 (Yes) - ham-net-mmiroslaw-mails-ham.log (May 12 11:42)
   1830957  (No) - ham-net-giovanni.log (May 12 08:44)
   1831456 (Yes) - spam-net-ena-week2.log (May 12 15:35)
   1831456 (Yes) - ham-net-ena-week0.log (May 12 16:22)
   1831456 (Yes) - spam-net-darxus.log (May 12 10:32)
   1831456 (Yes) - spam-net-axb-ham-misc.log (May 12 12:10)
   1831456 (Yes) - ham-net-darxus.log (May 12 10:32)
   1831456 (Yes) - ham-net-thendrikx.log (May 12 09:29)
   1831456 (Yes) - spam-net-jarif.log (May 12 09:29)
   1831456 (Yes) - spam-net-axb-coi-bulk.log (May 12 12:10)
   1831456 (Yes) - spam-net-ena-week1.log (May 12 12:43)
   1831456 (Yes) - spam-net-mmiroslaw-mails-spam.log (May 12 11:42)
   1831456 (Yes) - ham-net-axb-generic.log (May 12 12:10)

34/38 matches (17 ham, 17 spam)

Reply via email to