Re: Body rules hit on Subject

[John Hardin]

On Sat, 3 Feb 2018, Alex wrote:


Hi,


The only "solution" I've ever come up with is to create a meta rule group to 
account for the Subject hit:

body __FOO /foo/
header __SUBJ_FOO  Subject =~ /foo/
meta FOO  __FOO && !__SUBJ_FOO

I have to admit it's annoyed me on occasion that I can't create a single simple 
rule that ONLY matches on the message body, but TBH it's never been important 
enough in context for me to even commit the above horror.


It seems the the number of times you want to match ONLY the body and not the 
body+subject is low enough math this workaround is reasonable.

I mean, you could have a new category bodyonly, or something, but I doubt it's 
necessary.

Certainly changing the behavior of body now would be a mistake.


I've also had a problem when trying to write rules that rely on or
otherwise measure the length of the body. A more complicated set of
rules are needed for that, if it's even possible/reliable.


Q'n'D:

  header  __SUBJ_LENGTHSubject =~ /./
  tflags  __SUBJ_LENGTHmultiple

  body__BODY_LENGTH/./
  tflags  __BODY_LENGTHmultiple

Inefficient as hell, but it should work.

Better to use eval:check_body_length() if you can, though.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  After ten years (1998-2008) of draconian gun control in the State
  of Massachusetts, the results are in: firearms-related assaults up
  78%, firearms-related homicides up 67%, assault-related emergency
  room visits up 331%. Gun Control does not reduce violent crime.
---
 3 days until the first Falcon Heavy test launch

Re: Body rules hit on Subject

[Bill Cole]

On 3 Feb 2018, at 16:37 (-0500), Bill Cole wrote:


On 2 Feb 2018, at 16:59 (-0500), Kevin A. McGrail wrote:

There is no solution at the moment.  The subject is appended to the 
body of the text for rule parsing. 


The 2nd sentence is wrong: the subject is *prepended* to the body. 
Also: the 1st sentence is wrong, there's no *PRETTY* solution.


If every rendered 'body' starts with an appended line containing the 
Subject (with '^Subject: ' stripped off) then one can solve the 
problem of matching body rules in the Subject header thus:


body__DOCUSIGN_BODY_1ST  /\A.*\bdocusign\b.*\n/mi

body__DOCUSIGN_BODY_NOT1ST  /(?!\A).*\bdocusign\b.*\n/mi

meta  DOCUSIGN_BODY  (HAS_SUBJECT && __DOCUSIGN_BODY_NOT1ST) || 
(__DOCUSIGN_BODY_1ST || __DOCUSIGN_BODY_NOT1ST)


make that:

meta  DOCUSIGN_BODY  (HAS_SUBJECT && __DOCUSIGN_BODY_NOT1ST) || 
(MISSING_SUBJECT && (__DOCUSIGN_BODY_1ST || __DOCUSIGN_BODY_NOT1ST))



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole

Re: Body rules hit on Subject

[Bill Cole]

On 2 Feb 2018, at 16:59 (-0500), Kevin A. McGrail wrote:

There is no solution at the moment.  The subject is appended to the 
body of the text for rule parsing. 


The 2nd sentence is wrong: the subject is *prepended* to the body. Also: 
the 1st sentence is wrong, there's no *PRETTY* solution.


If every rendered 'body' starts with an appended line containing the 
Subject (with '^Subject: ' stripped off) then one can solve the problem 
of matching body rules in the Subject header thus:


body__DOCUSIGN_BODY_1ST  /\A.*\bdocusign\b.*\n/mi

body__DOCUSIGN_BODY_NOT1ST  /(?!\A).*\bdocusign\b.*\n/mi

meta  DOCUSIGN_BODY  (HAS_SUBJECT && __DOCUSIGN_BODY_NOT1ST) || 
(__DOCUSIGN_BODY_1ST || __DOCUSIGN_BODY_NOT1ST)



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole

Re: Body rules hit on Subject

[Kevin A. McGrail]

On 2/3/2018 2:28 PM, Alex wrote:

I've also had a problem when trying to write rules that rely on or
otherwise measure the length of the body. A more complicated set of
rules are needed for that, if it's even possible/reliable.


Take a look at these rules for an example:

72_active.cf:    body __KAM_BODY_LENGTH_LT_1024   
eval:check_body_length('1024')
72_active.cf:   body    __KAM_BODY_LENGTH_LT_128 
eval:check_body_length('128')
72_active.cf:    body __KAM_BODY_LENGTH_LT_256    
eval:check_body_length('256')
72_active.cf:    body __KAM_BODY_LENGTH_LT_512    
eval:check_body_length('512')


Regards,
KAM

Re: Body rules hit on Subject

[Kevin A. McGrail]

On 2/2/2018 6:09 PM, John Hardin wrote:


There is no solution at the moment.  The subject is appended to the 
body of the text for rule parsing.  I've added a task I plan to 
submit for GSOC consideration to add a tflag to disable this behavior.


Globally, or per-rule? 


A tflag affect just one rule.

Regards,

KAM

Re: Body rules hit on Subject

[Alex]

Hi,

>> The only "solution" I've ever come up with is to create a meta rule group to 
>> account for the Subject hit:
>>
>> body __FOO /foo/
>> header __SUBJ_FOO  Subject =~ /foo/
>> meta FOO  __FOO && !__SUBJ_FOO
>>
>> I have to admit it's annoyed me on occasion that I can't create a single 
>> simple rule that ONLY matches on the message body, but TBH it's never been 
>> important enough in context for me to even commit the above horror.
>
> It seems the the number of times you want to match ONLY the body and not the 
> body+subject is low enough math this workaround is reasonable.
>
> I mean, you could have a new category bodyonly, or something, but I doubt 
> it's necessary.
>
> Certainly changing the behavior of body now would be a mistake.

I've also had a problem when trying to write rules that rely on or
otherwise measure the length of the body. A more complicated set of
rules are needed for that, if it's even possible/reliable.

Re: Body rules hit on Subject

[@lbutlr]

On 2 Feb 2018, at 14:27, Kris Deugau  wrote:
> The only "solution" I've ever come up with is to create a meta rule group to 
> account for the Subject hit:
> 
> body __FOO /foo/
> header __SUBJ_FOO  Subject =~ /foo/
> meta FOO  __FOO && !__SUBJ_FOO
> 
> I have to admit it's annoyed me on occasion that I can't create a single 
> simple rule that ONLY matches on the message body, but TBH it's never been 
> important enough in context for me to even commit the above horror.

It seems the the number of times you want to match ONLY the body and not the 
body+subject is low enough math this workaround is reasonable.

I mean, you could have a new category bodyonly, or something, but I doubt it's 
necessary.

Certainly changing the behavior of body now would be a mistake.

-- 
You start a conversation you can't even finish it
You're talkin' a lot, but you're not sayin' anything
When I have nothing to say, my lips are sealed
Say something once, why say it again?

Re: Body rules hit on Subject

[John Hardin]

On Fri, 2 Feb 2018, Kevin A. McGrail wrote:


On 2/2/2018 1:41 PM, Alex wrote:

Hi,
I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.

It's now affecting me again, and I hoped someone had some ideas on how
to prevent body rules from hitting on the subject too since it's
apparently considered part of the message body?


There is no solution at the moment.  The subject is appended to the body of 
the text for rule parsing.  I've added a task I plan to submit for GSOC 
consideration to add a tflag to disable this behavior.


Globally, or per-rule?

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Users mistake widespread adoption of Microsoft Office for
  the development of a document format standard.
---
 4 days until the first Falcon Heavy test launch

Re: Body rules hit on Subject

[Martin Gregorie]

On Fri, 2018-02-02 at 16:59 -0500, Kevin A. McGrail wrote:
> There is no solution at the moment.  The subject is appended to the
> body of the text for rule parsing.  I've added a task I plan to
> submit for GSOC consideration to add a tflag to disable this
> behavior.
> 
Would it sensible leave the body rule as at present and add two more
rules:

header   NAME  Subject =~ /regex/
bodytext NAME  /regex/

This would add flexibilty to rule creation. It may also be almost free
in execution time terms because the current body rule must already
require the subject text and body text to be extracted from the message
before concatenating them. 


Martin

Re: Body rules hit on Subject

[John Hardin]

On Fri, 2 Feb 2018, Alex wrote:


I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.


This behavior is by design.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  ...every time I sit down in front of a Windows machine I feel as
  if the computer is just a place for the manufacturers to put their
  advertising. -- fwadling on Y! SCOX
---
 4 days until the first Falcon Heavy test launch

Re: Body rules hit on Subject

[Kevin A. McGrail]

On 2/2/2018 1:41 PM, Alex wrote:

Hi,
I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.

It's now affecting me again, and I hoped someone had some ideas on how
to prevent body rules from hitting on the subject too since it's
apparently considered part of the message body?


There is no solution at the moment.  The subject is appended to the body 
of the text for rule parsing.  I've added a task I plan to submit for 
GSOC consideration to add a tflag to disable this behavior.



Regards,

KAM

Re: Body rules hit on Subject

[Kris Deugau]

Alex wrote:

Hi,
I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.

It's now affecting me again, and I hoped someone had some ideas on how
to prevent body rules from hitting on the subject too since it's
apparently considered part of the message body?


The only "solution" I've ever come up with is to create a meta rule 
group to account for the Subject hit:


body __FOO /foo/
header __SUBJ_FOO  Subject =~ /foo/
meta FOO  __FOO && !__SUBJ_FOO

I have to admit it's annoyed me on occasion that I can't create a single 
simple rule that ONLY matches on the message body, but TBH it's never 
been important enough in context for me to even commit the above horror.


-kgd

Re: Body rules hit on Subject

[David Jones]

On 02/02/2018 02:19 PM, Alex wrote:

Hi,


I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.

It's now affecting me again, and I hoped someone had some ideas on how
to prevent body rules from hitting on the subject too since it's
apparently considered part of the message body?



I don't think I have ever run across this problem on my SA instances. Can
you send an example of this via pastebin.com?


Save this text below which contains no "docusign" in the body in a
file and create a rule like:

body   __BODY_DOCUSIGN   /docusign/i

 From DUMMY-LINE Fri Feb  2 12:15:01 2018
Return-Path: 
To: otheru...@example.com
Subject: DocuSign
Content-Type: text/plain; charset=UTF-8
Message-Id: <20180202171501.e23486800c...@mail01.example.com>
Date: Fri,  2 Feb 2018 12:15:01 -0500 (EST)

this is a test

dbg: rules: ran body rule __BODY_DOCUSIGN ==> got hit: "DocuSign"
dbg: rules: ran header rule __DOCUSIGN_SUBJ ==> got hit: "DocuSign"



Yep.  Same results for me too.  Never noticed this before.

--
David Jones

Re: Body rules hit on Subject

[Alex]

Hi,

>> I recall a conversation in the past about body rules hitting on the
>> Subject of an email, but I don't believe there was ever a solution, or
>> I otherwise missed it.
>>
>> It's now affecting me again, and I hoped someone had some ideas on how
>> to prevent body rules from hitting on the subject too since it's
>> apparently considered part of the message body?
>>
>
> I don't think I have ever run across this problem on my SA instances. Can
> you send an example of this via pastebin.com?

Save this text below which contains no "docusign" in the body in a
file and create a rule like:

body   __BODY_DOCUSIGN   /docusign/i

>From DUMMY-LINE Fri Feb  2 12:15:01 2018
Return-Path: 
To: otheru...@example.com
Subject: DocuSign
Content-Type: text/plain; charset=UTF-8
Message-Id: <20180202171501.e23486800c...@mail01.example.com>
Date: Fri,  2 Feb 2018 12:15:01 -0500 (EST)

this is a test

dbg: rules: ran body rule __BODY_DOCUSIGN ==> got hit: "DocuSign"
dbg: rules: ran header rule __DOCUSIGN_SUBJ ==> got hit: "DocuSign"

Re: Body rules hit on Subject

[David Jones]

On 02/02/2018 12:41 PM, Alex wrote:

Hi,
I recall a conversation in the past about body rules hitting on the
Subject of an email, but I don't believe there was ever a solution, or
I otherwise missed it.

It's now affecting me again, and I hoped someone had some ideas on how
to prevent body rules from hitting on the subject too since it's
apparently considered part of the message body?



I don't think I have ever run across this problem on my SA instances. 
Can you send an example of this via pastebin.com?


--
David Jones