[sniffer] Re: gbudb source new

[John Tolmachoff]

Thanks Linda. I guess I should not have dismissed the "that would be too easy" 
thought next time.

-Original Message-
From: "Linda Pagillo" <lpad...@gmail.com>
Sent: Wednesday, July 26, 2017 12:50pm
To: "Message Sniffer Community" <sniffer@sortmonster.com>
Subject: [sniffer] Re: gbudb source new

HI John. The best way to do this would be to create a filter in Declude
with the following line and score it how you like by changing the 0 to a
value:

HEADERS  0 PCRE (?im:X-GBUdb-Analysis.+New)

Thanks!

On Tue, Jul 25, 2017 at 2:01 PM, John Tolmachoff <
johnl...@eservicesforyou.com> wrote:

> Using Message Sniffer as part of Declude on a SmarterMail install, I want
> to add weight to a source new when gbudb indicates such. What is the best
> way to do that?
>
> John T
> eServices For You
>
>
> #
> This message is sent to you because you are subscribed to
>   the mailing list <sniffer@sortmonster.com>.
> This list is for discussing Message Sniffer,
> Anti-spam, Anti-Malware, and related email topics.
> For More information see http://www.armresearch.com
> To unsubscribe, E-mail to: <sniffer-...@sortmonster.com>
> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com>
> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com>
> Send administrative queries to  <sniffer-requ...@sortmonster.com>
>
>



#
This message is sent to you because you are subscribed to
  the mailing list <sniffer@sortmonster.com>.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: <sniffer-...@sortmonster.com>
To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com>
To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com>
Send administrative queries to  <sniffer-requ...@sortmonster.com>

[sniffer] Re: gbudb source new

[Linda Pagillo]

HI John. The best way to do this would be to create a filter in Declude
with the following line and score it how you like by changing the 0 to a
value:

HEADERS  0 PCRE (?im:X-GBUdb-Analysis.+New)

Thanks!

On Tue, Jul 25, 2017 at 2:01 PM, John Tolmachoff <
johnl...@eservicesforyou.com> wrote:

> Using Message Sniffer as part of Declude on a SmarterMail install, I want
> to add weight to a source new when gbudb indicates such. What is the best
> way to do that?
>
> John T
> eServices For You
>
>
> #
> This message is sent to you because you are subscribed to
>   the mailing list .
> This list is for discussing Message Sniffer,
> Anti-spam, Anti-Malware, and related email topics.
> For More information see http://www.armresearch.com
> To unsubscribe, E-mail to: 
> To switch to the DIGEST mode, E-mail to 
> To switch to the INDEX mode, E-mail to 
> Send administrative queries to  
>
>

[sniffer] Re: GBUdb Tool

[Darin Cox]

Hi Pete,

Would you mind sharing your calculations of confidence and probability?  I'm 
looking at the stats for p=1.0 and curious about the low confidence values. 
I would have expected high confidence where there were no good samples and a 
lot of bad... or do I have something backwards?


Also, while it's easy to parse, it might be nice if the output had one 
delimiter between fields instead of being both tab and comma delimited. 
Makes importing into a database for analysis much easier.


Appreciate it,

Darin.

-Original Message- 
From: Pete McNeil

Sent: Friday, November 23, 2012 3:43 PM
To: Message Sniffer Community
Subject: [sniffer] GBUdb Tool

Hello Sniffer Folks,

We have been playing with a new utility that some of you may enjoy.

http://www.armresearch.com/message-sniffer/download/GBUDBTool-V0.1.zip

GBUDB Tool allows you to create a list of IP addresses from your GBUdb
snapshots (.gbx files). You can select IPs that are blacker or
whiter than a provided probability figure and confidence figure. It
outputs one IP per line, optionally with details about the statistics
for the IP. This can be useful for feeding-forward blacklists to block
at your firewall or for other research purposes.

Run GBUDBTool without any parameters and it will tell you about it's
command line options.

Please let us know if there is more we can do.

Best,

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com

[sniffer] Re: GBUdb

[Pete McNeil]

Hello Richard,

Wednesday, December 31, 2008, 11:49:35 AM, you wrote:







Does the snf XML command interface for GBUdb work? I was considering pumping in bad IPs as I find them into the GBUdb and also short-circuiting spam processing by calling the GBUdb to determine the status of an IP to reduce workload. Is this something that sounds like a workable idea?





It does work:

http://www.armresearch.com/support/articles/software/snfClient/commandLine.jsp

There are some considerations:

* If you mark an IP as bad in GBUdb then it will remain tagged that way until you change it. Instead you might want to give IPs some relatively high bad count so that those IPs can be forgotten over time, or refreshed if you determine they are bad at a later time.

* Blocking connections based on GBUdb has the effect of extending the black-list entry to something on the order of 24 hours if the IP record is based on statistics (ugly, not bad). This is because any good messages that might be received from the IP will not be heard by SNF while the IP is blocked up front. After some period of time the GBUdb statistics will be condensed and the IP may fall back into a zone where messages will be allowed. Condensation happens once per day by default. If a bad message is received again and pushes the IP into the black range then the IP will again be blocked for the remainder of that day (assuming you intend to block connections based on "black" GBUdb results.

* If you are building code to do up-front blocking (as you say short-circuiting processing) then you might want to consider talking to SNFServer directly via XCI instead of using the SNFClient. The SNFClient essentially translates command line parameters into XCI requests. If you make those requests with your code then you can skip the overhead of starting another child process.

* On gateway/proxy systems a very effective method of processing is to use SNF as the first scanner in the chain (during SMTP if possible) and to drive a dynamic local blacklist with it's results to block connections.

In this configuration:

--- SNF can perform full content scans during SMTP without slowing or overloading your system.
--- SNF can provide X- headers for later processing in case you want to add weights for some scores.
--- SNF content scans integrate with GBUdb so that content scanning work is truncated when IPs are bad.
--- SNF results caused by GBUdb overrides are unique and exposed so you only need a single operation.
--- Typically bad content scan results can be used to reject messages and add 30 minute black-list entries.
--- Typically caution results can be used to drive gray-listing for the IP.
--- Typically black results can be used to reject messages and add 1 hour black-list entries.
--- Typically truncate results can be used to reject connections and add 1 hour black-list entries.

In a more advanced system:

* Caution and Black results from SNF scans are very accurate indicators of Spam Storm conditions. If your filtering system can be configured with more than one mode then you can use any Caution or Black entry within the last 5 minutes to indicate a Spam Storm and change your system's condition to the more aggressive configuration. This can be a very effective way to turn on more aggressive gray listing and checking functions for short periods while normally running your system without them to avoid unnecessary support traffic.

* It is possible to integrate the SNF engine directly in your own MTA or Gateway software if you are making modifications at that level or writing your own as some of our larger filtering customers and OEMs have done. This can provide significant reductions in overhead at the expense of isolation. Often other elements in the system represent an overwhelmingly large part of the system load compared to SNF so this extra step may not be needed but on extremely efficient "carrier grade" systems it can represent a useful improvement in performance since child processes and IO operations are eliminated.

Please let us know if you have any other questions.

Thanks!

_M



--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.



#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com

[sniffer] Re: GBUdb

[Pete McNeil]

Hello Richard,

Thursday, December 4, 2008, 3:27:51 PM, you wrote:







Is the GBUdb currently sharing information as described in the documentation?





Yes.







 Do the GBUdb XCI commands detailed within snf_xci.xml work through the tcp interface?





Yes.

The SNFClient utility simply translates your command line parameters into XCI requests.

Best,

_M



--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.



#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb

[Richard Stupek]

Ok.  We are seeing a large amount of spam lately that is not being picked up
through snf and most of it has the from and the to set the same. Are you
seeing anything similar?

On Thu, Dec 4, 2008 at 2:37 PM, Pete McNeil [EMAIL PROTECTED]wrote:

  Hello Richard,


 Thursday, December 4, 2008, 3:27:51 PM, you wrote:




 Is the GBUdb currently sharing information as described in the
 documentation?


 Yes.




   Do the GBUdb XCI commands detailed within snf_xci.xml work through the
 tcp interface?


 Yes.


 The SNFClient utility simply translates your command line parameters into
 XCI requests.


 Best,


 _M




 --

 Pete McNeil

 Chief Scientist,

 Arm Research Labs, LLC.

 #
 This message is sent to you because you are subscribed to
   the mailing list sniffer@sortmonster.com.
 To unsubscribe, E-mail to: [EMAIL PROTECTED]
 To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
 To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
 Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb

[Pete McNeil]

Hello Richard,

Thursday, December 4, 2008, 3:41:34 PM, you wrote:







Ok. We are seeing a large amount of spam lately that is not being picked up through snf and most of it has the "from" and the "to" set the same. Are you seeing anything similar?





We have seen a lot of spam formed that way -- the tactic isn't new. It capitalizes on systems that automatically white-list messages from local addresses (to one extent or another).

I am showing good capture rates at the moment.

If you have some examples, please zip them and attach the zips in a message to support@ with "chronic spam" in your subject line. I will take a look and see if there is anything special about them.

Also please include your license ID so that I can check on the telemetry from your system and your account settings.

Thanks,

_M


--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.



#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb False Positives vs. Rule IDs

[Andy Schmidt]

Hi Pete,

 You can drop the record for the IP from GBUdb with SNFClient -drop IP,
but if the system is not configured properly then the IP will quickly rise
back into the truncate list. 

The IP address in question was a third party IP address, not related to us,
not a gateway. It was not in the ignore list and shouldn't be - does that
qualify as configured properly?

 If that is being caused by a pattern rule then you need to discover the
pattern rule from logs first and then panic that rule and report the FP.

Hm - so if we have such a GBUdb FP issue, we would need to first go into the
log for the message ID in question and locate the IP address. THEN, we have
to search the log files to find where this IP address may have occurred
(possibly several days of logs, before someone noticed legitimate email from
being missing) in hopes of eventually still finding some log entry that
relates to the original rule-ID, before we can add it to the panic list?

I suppose it would be technically impossible to include the underlying rule
in the GBUdb, so that it can be properly reported when messages are blocked?


 Are you reporting such an FP?

Yes, your FP support identified the underlying rule and reported it back to
me. Of course, I need to have a panic procedure in place that doesn't rely
on outside assistance.  Doesn't happen often, but better ask the questions
now than when the brown matter hits to air circulation enhancer.

Best Regards,

Andy

From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf
Of Pete McNeil
Sent: Tuesday, October 07, 2008 1:59 PM
To: Message Sniffer Community
Subject: [sniffer] Re: How to deal with False Positives and other
Documentation Issues

 

Hello Andy,

 

Thanks for this -- I will address the documentation issues shortly.

 

Regarding GBUdb FP issues-- to date we've not had a truncate (result code
20) false positive report from any system that was configured properly.

 

Are you reporting such an FP?

 

Depending upon the circumstances you may want to add the IP to your ignore
list.

 

You can drop the record for the IP from GBUdb with SNFClient -drop IP, but
if the system is not configured properly then the IP will quickly rise back
into the truncate list.

 

If that is being caused by a pattern rule then you need to discover the
pattern rule from logs first and then panic that rule and report the FP.

 

Hope this helps,

 

_M

 

Remainder for reference...

 

Tuesday, October 7, 2008, 12:58:43 PM, you wrote:

 


 

Hi,

 

1.   I read this page:

 http://www.armresearch.com/support/articles/procedures/falsePositives.jsp
http://www.armresearch.com/support/articles/procedures/falsePositives.jsp

and it seems to be the same.

 

However, should this chapter be expanded to contain information about what
to do if some of the new technologies are responsible for the false
positive? The panic rule instructions don't really apply in cases like
this where there IS no rule:

 

s u='20081007153730' m='D:\IMail\spool\proc\work\D822c0199026c.smd'
s='20' r='0'

p s='0' t='0' l='10306' d='0'/

g o='0' i='207.45.161.16' t='u' c='0.226425' p='1'
r='Truncate'/

/s

 

Instead you should have some ready-made sample that shows how to except an
IP that has ended up on the Truncate list, or at least move it to the
caution list?

 

2.   The explanation of the Log files is incomplete:

 
http://www.armresearch.com/support/articles/software/snfServer/logFiles/act
ivityLogs.jsp
http://www.armresearch.com/support/articles/software/snfServer/logFiles/acti
vityLogs.jsp

As you can see from the log snippet I posted, there is a node s:r=0.
However, s:r is not in the documentation.

 

Best Regards,

Andy

 

 

 

 

-- 

Pete McNeil

Chief Scientist,

Arm Research Labs, LLC.

#
 
This message is sent to you because you are subscribed to
 
  the mailing list sniffer@sortmonster.com.
 
To unsubscribe, E-mail to: [EMAIL PROTECTED]
 
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
 
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
 
Send administrative queries to  [EMAIL PROTECTED]
 
 

[sniffer] Re: GBUdb False Positives vs. Rule IDs

[Pete McNeil]

Hello Andy,

Tuesday, October 7, 2008, 2:40:01 PM, you wrote:







Hi Pete,
You can drop the record for the IP from GBUdb with SNFClient -drop IP, but if the system is not configured properly then the IP will quickly rise back into the truncate list. 
The IP address in question was a third party IP address, not related to us, not a gateway It was not in the ignore list and shouldnt be  does that qualify as configured properly?





Yes.







If that is being caused by a pattern rule then you need to discover the pattern rule from logs first and then panic that rule and report the FP.
Hm  so if we have such a GBUdb FP issue, we would need to first go into the log for the message ID in question and locate the IP address. THEN, we have to search the log files to find where this IP address may have occurred (possibly several days of logs, before someone noticed legitimate email from being missing) in hopes of eventually still finding some log entry that relates to the original rule-ID, before we can add it to the panic list?





Yes-- more or less.

It's not as bad as it seems though.

In order for an IP to get into the truncate range in GBUdb it has to consistently send messages that match pattern rules. That is 95% of the time if a message is sent from this IP it matches a pattern rule AND it has to send a bunch of them.

If the messages come in over separate days the statistics condense every day -- so on any given day it is likely a number of messages would have to come in and match pattern rules.

That means that a message matching the offending pattern rule is likely to be listed in the same log file and previous days (if any).

It also means that if you find that IP in that log you are virtually guaranteed that the message you find will have either matched the pattern rule or been truncated.

In this case the probability figure is 1 indicating that all messages from this IP have matched pattern rules. GBUdb override results (caution, black, truncate) do not change IP statistics... so the only way for an IP to get into the truncate range is by consistently producing messages that match pattern rules.

Presumably if substantially all messages from this legitimate source were to be tagged as spam then they would be reported as false positives.

Even if they were not immediately reported as false positives then the daily condensation of GBUdb statistics would force the IP out of the truncate range until more messages were tagged by the pattern rule -- and presumably one or more of those would be reported as false positives.

Bottom line -- it should not be difficult to find log records associated with this IP that are also associated with the pattern rules that pushed it into the truncate range.







I suppose it would be technically impossible to include the underlying rule in the GBUdb, so that it can be properly reported when messages are blocked?





Yes. The GBUdb engine only stores the statistics about the IPs and the data needed to index and access these records quickly. However, as I've said, information on the pattern rules should be relatively easy to find -- especially for truncate cases.








Are you reporting such an FP?
Yes, your FP support identified the underlying rule and reported it back to me. Of course, I need to have a panic procedure in place that doesnt rely on outside assistance Doesnt happen often, but better ask the questions now than when the brown matter hits to air circulation enhancer.





This case is somewhat unique. The pattern rule has been around for a very long time -- so it is extremely unlikely that a similar case would arise again.

A short-term and immediate fix for such a case -- while figuring out what is really going on -- is to reset the statistics on the IP so that they are not in the truncate range and so that it would take a large effort to get them there.

For example, you could SNFClient -set IP ugly 0 32

This would move the IPs statistics far toward the white so that a truly large number of hits would be required to push it back into truncate even if every message failed a pattern rule. In the mean time the IP would be in the "normal" range.

This gives you immediate relieve with a "fire and forget" command. The GBUdb statistics for the IP will eventually return to the correct value for the IP and by the time that happens you will have resolved the underlying pattern rule issue or made some other decision regarding the IP.

Hope this helps,

_M

--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.



#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb False Positives vs. Rule IDs

[Andy Schmidt]

Thanks Pete - I'll save that command.

I also suggest that some of your instructions might be helpful to see in the
documentation in the chapters on how to deal with false positives.

From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf
Of Pete McNeil
Sent: Tuesday, October 07, 2008 3:41 PM
To: Message Sniffer Community
Subject: [sniffer] Re: GBUdb False Positives vs. Rule IDs

 

Hello Andy,

 

Tuesday, October 7, 2008, 2:40:01 PM, you wrote:

 


 

Hi Pete,

 You can drop the record for the IP from GBUdb with SNFClient -drop IP,
but if the system is not configured properly then the IP will quickly rise
back into the truncate list. 

The IP address in question was a third party IP address, not related to us,
not a gateway. It was not in the ignore list and shouldn't be - does that
qualify as configured properly?

 

Yes.

 


 

 If that is being caused by a pattern rule then you need to discover the
pattern rule from logs first and then panic that rule and report the FP.

Hm - so if we have such a GBUdb FP issue, we would need to first go into the
log for the message ID in question and locate the IP address. THEN, we have
to search the log files to find where this IP address may have occurred
(possibly several days of logs, before someone noticed legitimate email from
being missing) in hopes of eventually still finding some log entry that
relates to the original rule-ID, before we can add it to the panic list?

 

Yes-- more or less.

 

It's not as bad as it seems though.

 

In order for an IP to get into the truncate range in GBUdb it has to
consistently send messages that match pattern rules. That is 95% of the time
if a message is sent from this IP it matches a pattern rule AND it has to
send a bunch of them.

 

If the messages come in over separate days the statistics condense every day
-- so on any given day it is likely a number of messages would have to come
in and match pattern rules.

 

That means that a message matching the offending pattern rule is likely to
be listed in the same log file and previous days (if any).

 

It also means that if you find that IP in that log you are virtually
guaranteed that the message you find will have either matched the pattern
rule or been truncated.

 

In this case the probability figure is 1 indicating that all messages from
this IP have matched pattern rules. GBUdb override results (caution, black,
truncate) do not change IP statistics... so the only way for an IP to get
into the truncate range is by consistently producing messages that match
pattern rules.

 

Presumably if substantially all messages from this legitimate source were to
be tagged as spam then they would be reported as false positives.

 

Even if they were not immediately reported as false positives then the daily
condensation of GBUdb statistics would force the IP out of the truncate
range until more messages were tagged by the pattern rule -- and presumably
one or more of those would be reported as false positives.

 

Bottom line -- it should not be difficult to find log records associated
with this IP that are also associated with the pattern rules that pushed it
into the truncate range.

 


 

I suppose it would be technically impossible to include the underlying rule
in the GBUdb, so that it can be properly reported when messages are blocked?

 

Yes. The GBUdb engine only stores the statistics about the IPs and the data
needed to index and access these records quickly. However, as I've said,
information on the pattern rules should be relatively easy to find --
especially for truncate cases.

 


 

 

 Are you reporting such an FP?

Yes, your FP support identified the underlying rule and reported it back to
me. Of course, I need to have a panic procedure in place that doesn't rely
on outside assistance.  Doesn't happen often, but better ask the questions
now than when the brown matter hits to air circulation enhancer.

 

This case is somewhat unique. The pattern rule has been around for a very
long time -- so it is extremely unlikely that a similar case would arise
again.

 

A short-term and immediate fix for such a case -- while figuring out what is
really going on -- is to reset the statistics on the IP so that they are not
in the truncate range and so that it would take a large effort to get them
there.

 

For example, you could SNFClient -set IP ugly 0 32

 

This would move the IPs statistics far toward the white so that a truly
large number of hits would be required to push it back into truncate even if
every message failed a pattern rule. In the mean time the IP would be in the
normal range. 

 

This gives you immediate relieve with a fire and forget command. The GBUdb
statistics for the IP will eventually return to the correct value for the IP
and by the time that happens you will have resolved the underlying pattern
rule issue or made some other decision regarding the IP.

 

Hope this helps,

 

_M

 

-- 

Pete McNeil

Chief Scientist,

Arm

[sniffer] Re: GBUdb dump

[Pete McNeil]

Hello Michael,

Tuesday, June 17, 2008, 4:48:54 PM, you wrote:

 Pete,

 How soon should we expect to see a new gbx file after a dump?

If you are using the default settings then it should appear after
about an hour. By default GBUdb creates a snapshot of it's database
every 3600 seconds.

gbudb
  database
...
checkpoint on-off='on' secs='3600'/


_M

-- 
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.


#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb question

[Pi-Web - Frank Jensen]

Hi Rob,

You can add the IPs to GBUdbIgnoreList.txt if you want sniffer to ignore the 
IPs.


Pete,

I have some questions about GBUdb

FIRST QUESTION:

I have several clients who forward over e-mails from ISP accounts. I 
have a system whereby I can pick out the original sending server IP. I 
then add that IP to the message in a special header. (this can vary by 
ISP and situation, but I've programmed my system to appropriately 
determine which IP is the original sending server IP. Next, I add a 
special custom header which points out that IP.


Would it be possible for MessageSniffer to grab the IP from a particular 
header (perhaps this header could be added as a node in the XML config 
file?). That way,  if/when that header is available in the message, 
Sniffer would then treat *that* IP as the sender's IP?


SECOND QUESTION:

Is it possible to tell Sniffer to NOT allow the possibility of 
truncating on a message-by-message basis, where this would be 
determined if a special command line switch were present. In fact, can 
Sniffer be further instructed to ONLY run pattern matching scanning 
and ignore the GBUdb for that particular message?


THIRD QUESTION:

Much of the spam I block doesn't run through Sniffer. Additionally, many 
of the messages that Sniffer blocks are spams sent via established ISPs 
whereas I already have those IPs in an extensive whitelist that I've 
built up over the years.


A 4% sampling of this whitelist can be found here:
http://invaluement.com/fourpercentofwhitelist.txt
(multiple the size of that by 25 to get an idea of the massive size of 
my IP whitelist)


Here is what I'd like to do which I believe would make my contribution 
to sniffer most effective:


(A) Have sniffer NOT automatically input data into GBUdb with each 
sniffer scan. (Is that possible?)


(B) Alternatively, whenever my spam filter marks a message as spam, it 
will issue the following command (but ONLY if that IP is NOT on my IP 
whitelist, and regardless of whether or not the message was run through 
sniffer):


SNFClient.exe -bad IP4Address

(If on my IP whitelist, it just won't do anything here.)

(C) If my spam filter marks a message as ham, then it will issue the 
following command (again, regardless of whether or not the message was 
run through sniffer)


SNFClient.exe -good IP4Address

**
**
I know that this puts more trust on me and my system, but I have also 
know that the quality of stats you'd receive from my system would vastly 
improved due to my abilities in this area and this would be a huge 
contribution to other Sniffer users over the norm. (I run one of the 
best RBLs and URI blacklists in the world... I know what I'm doing here!)


Can these things be done?

Rob McEwen



#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]





--
Mvh. Frank Jensen
[EMAIL PROTECTED]
www.pi.dk



Imponerende, fascinerende og kæmpe
Plakater f.eks. 149 x 149 = 629 kr
Vi kan også lave plakat fra dit digitale foto

www.plakatkunst.dk



#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb question

[Pete McNeil]

Hello Rob,

Tuesday, January 22, 2008, 11:09:10 AM, you wrote:

 Pete,

 I have some questions about GBUdb

This may help:

http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.GBUdb

 FIRST QUESTION:

 I have several clients who forward over e-mails from ISP accounts. I 
 have a system whereby I can pick out the original sending server IP. I
 then add that IP to the message in a special header. (this can vary by
 ISP and situation, but I've programmed my system to appropriately 
 determine which IP is the original sending server IP. Next, I add a 
 special custom header which points out that IP.

We are developing an auto-drill-down feature for GBUdb to assist in
automatically training GBUdb in this way. The auto drill feature will
add IPs of intermediate systems to the local ignore list based on
header directives. The theory is that GBUdb will be able to
automatically learn to ignore the intermediate nodes of mixed-source
ISPs in order to identify the original source of the message.

There is still some development work to do on this experimental
feature but we hope to include it in the upcoming release. Any
insights you can provide on reliably identifying these intermediate
servers would be very useful.

The current plan is to locate a specific tell tale string in the
Received header that is likely to be the source (based on current
knowledge). If the string is found then that header is disqualified
(and it's IP added to the ignore list) so that the next header becomes
the source candidate.

The tell tale string is presumed to be the domain portion (or
similar fragment) of the reverse DNS data in the Received header. So,
for example, if the top Received header contains .troublesome.isp.com
[ then that header would be disqualified as the source of the message
(for GBUdb purposes), it's IP would be added to the ignore
(infrastructure) list, and the next Received header would be
considered. Once all of the .troublesome.isp.com [ or similar
headers are exhausted then the next header is likely to be the actual
source (so the theory goes).

 Would it be possible for MessageSniffer to grab the IP from a particular
 header (perhaps this header could be added as a node in the XML config
 file?). That way,  if/when that header is available in the message, 
 Sniffer would then treat *that* IP as the sender's IP?

I will consider adding this to the feature request list. It probably
won't be added to the first version though -- we have a request freeze
in effect to ensure we get the production version out in Q1.

This is also a highly specialized request -- there aren't a lot of
systems out there that can accurately drill through delivery chains to
identify the original source of the message with any great accuracy --
so the number of folks who could use this feature would be pretty
small (if not one). Your use of the command line utility (described
below) seems more appropriate since in effect you want to eliminate
GBUdb's source detection features.

That said - I am anxious to support your work -

Please share an example of the header you would inject.

If it is possible to implement the feature quickly and reliably then I
will see what I can do to add it to the header directives engine.

 SECOND QUESTION:

 Is it possible to tell Sniffer to NOT allow the possibility of 
 truncating on a message-by-message basis, where this would be 
 determined if a special command line switch were present. In fact, can
 Sniffer be further instructed to ONLY run pattern matching scanning 
 and ignore the GBUdb for that particular message?

It is not possible to turn off truncate on a message by message basis.

It is possible to turn off truncate for all messages but not on a
message by message basis.

You can also create a header directive to cause GBUdb training to
ignore a message with a specific header (or specifically, if it finds
a specific string in a specific header).

 THIRD QUESTION:

 Much of the spam I block doesn't run through Sniffer. Additionally, many
 of the messages that Sniffer blocks are spams sent via established ISPs
 whereas I already have those IPs in an extensive whitelist that I've 
 built up over the years.

 A 4% sampling of this whitelist can be found here:
 http://invaluement.com/fourpercentofwhitelist.txt (multiple the size
 of that by 25 to get an idea of the massive size of my IP whitelist)

 Here is what I'd like to do which I believe would make my contribution
 to sniffer most effective:

 (A) Have sniffer NOT automatically input data into GBUdb with each 
 sniffer scan. (Is that possible?)

You could create header directives to selectively disable GBUdb
training.

You can also disable GBUdb training for all messages.

training on-off='off'

 (B) Alternatively, whenever my spam filter marks a message as spam, it
 will issue the following command (but ONLY if that IP is NOT on my IP 
 whitelist, and regardless of whether or not the message was run through
 sniffer):

 

[sniffer] Re: GBUdb question

[Rob McEwen]

Pete McNeil wrote:

This may help:

http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.GBUdb

  

I did read that first. It was helpful. I'll keep referring back.

We are developing an auto-drill-down feature for GBUdb to assist in
automatically training GBUdb in this way. The auto drill feature will
add IPs of intermediate systems to the local ignore list based on
header directives. The theory is that GBUdb will be able to
automatically learn to ignore the intermediate nodes of mixed-source
ISPs in order to identify the original source of the message.

There is still some development work to do on this experimental
feature but we hope to include it in the upcoming release. Any
insights you can provide on reliably identifying these intermediate
servers would be very useful.
I'm not confident that this will handle the forwarded messages 
scenarios that I described, which I have ready custom programmed for the 
specific narrow range of ways that this currently happens with my server.

Please share an example of the header you would inject.
  

Currently, I'm using the following:

X-RegEx-Original-IP: 127.0.0.1

(But X-RegEx-Original-IP was arbitrary. This was inherited by an 
antiquated anti-spam utility I used years ago. The X-RegEx-Original-IP 
part can change at any time. This would even be a header custom 
designated by Sniffer.)


Even better, another option would be for the IP to be passed to sniffer 
via the command line where sniffer would know to use that one and not 
bother trying to grab this from the header. Please consider that as a 
feature request.

It is not possible to turn off truncate on a message by message basis.

It is possible to turn off truncate for all messages but not on a
message by message basis.
  

that will suffice


Here is what I'd like to do which I believe would make my contribution
to sniffer most effective:

(A) Have sniffer NOT automatically input data into GBUdb with each 
sniffer scan. (Is that possible?)



You could create header directives to selectively disable GBUdb
training.

You can also disable GBUdb training for all messages.

training on-off='off'

  
That will work. But will this disable the SNFClient.exe -bad and 
SNFClient.exe -good tools?? and will this disable sharing of the data? 
Can data accumulated via these manual reportings be shared even if 
training is off?

That sounds very much like what these tools were designed for. However
the effect may not be what you intend.

If the IPs you track are not detected as the source IP by GBUdb then
it is likely to ignore the data during it's scans. It will evaluate
the statistics of the IP it believes to be the source. When it gets
that right it will find your data. When it gets that wrong it will
find no data (most likely) so GBUdb will be effectively inert in those
cases.

If your intent is simply to input this data into the GBUdb system so
that it is available as a resource then that will work - somewhat.

One other thought that I have is that you could use the command line
(or the ignore list) to mark the IPs on your internal white-list as
Infrastructure (ignore flag). This might effectively train GBUdb to
skip those IPs when finding the source of the message - and in any
case would render GBUdb inert for those IPs.
  
There are too many IPs on that whitelist (it might have been possible 
were it not that many of these entries are massive blocks of IPs).


Follow-up question...

If, therefore, I cannot stop GBUdb-processing for a particular message, 
but I turn off truncate for all messages, the way I see it, couldn't I 
simply ignore the GBUdb reporting for some particular messages? (might 
not be as efficient, but I'd get the same result I seek!) But in a case 
where truncate is turned off, if GBUdb reports a message as spam, AND 
content rules ALSO mark that message as spam, will the return code tell 
me that both GBUdb *and *rules caught the spam? Or do I get one code 
instead of the other (if so, which one?)


Thanks!

Rob McEwen



#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to  [EMAIL PROTECTED]

[sniffer] Re: GBUdb question

[Pete McNeil]

Hello Rob,

Tuesday, January 22, 2008, 1:11:00 PM, you wrote:

snip... about auto-drill-down/

 I'm not confident that this will handle the forwarded messages
 scenarios that I described, which I have ready custom programmed for
 the specific narrow range of ways that this currently happens with
 my server.

We're hopeful it will work for many cases. If you can identify cases
where it won't work please let us know.

 Please share an example of the header you would inject.
   
 Currently, I'm using the following:

 X-RegEx-Original-IP: 127.0.0.1

 (But X-RegEx-Original-IP was arbitrary. This was inherited by an 
 antiquated anti-spam utility I used years ago. The X-RegEx-Original-IP
 part can change at any time. This would even be a header custom 
 designated by Sniffer.)

That seems straight forward enough. Thanks.

 Even better, another option would be for the IP to be passed to sniffer
 via the command line where sniffer would know to use that one and not 
 bother trying to grab this from the header. Please consider that as a 
 feature request.

I will add that to the list.

snip about GBUdb training options (disabled training)/

 That will work. But will this disable the SNFClient.exe -bad and
 SNFClient.exe -good tools?? and will this disable sharing of the
 data? Can data accumulated via these manual reportings be shared
 even if  training is off?

The command line tools always work. When you report a good or bad
hit it has the same effect as GBUdb learning from a message scan.

The information will be stored and shared in exactly the same way.

When you turn off training you are only disabling the system's ability
to learn automatically from scanned messages. Inputs from the command
line utility are still retained.

snip/

 One other thought that I have is that you could use the command
 line (or the ignore list) to mark the IPs on your internal
 white-list as Infrastructure (ignore flag). This might effectively
 train GBUdb to skip those IPs when finding the source of the
 message - and in any case would render GBUdb inert for those IPs.
 There are too many IPs on that whitelist (it might have been possible 
 were it not that many of these entries are massive blocks of IPs).

Perhaps - that's up to you. However, the GBUdb system is designed to
handle large numbers of IPs without slowing down. It is not uncommon
to have significantly more than half a million IPs in GBUdb on systems
that handle 500 msg/min or more.

The ignore list file is intended to handle local infrastructure so
that if you lose your GBUdb data you can be assured that your local
resources are not tagged as bad sources accidentally.

Other IP records (ignore, good, bad, or ugly) can be entered via the
command line utility with the only real limit being the amount of RAM
you want to commit to the GBUdb.

To give you an idea of scalability, one of our spamtrap processors is
currently (typ) handling about 3000 msg/minute and has the following
GBUdb statistics:

gbudb
size bytes='109051904'/
records count='479671'/
utilization percent='96.7379'/
/gbudb


 Follow-up question...

 If, therefore, I cannot stop GBUdb-processing for a particular message,
 but I turn off truncate for all messages, the way I see it, couldn't I
 simply ignore the GBUdb reporting for some particular messages? (might
 not be as efficient, but I'd get the same result I seek!) But in a case
 where truncate is turned off, if GBUdb reports a message as spam, AND 
 content rules ALSO mark that message as spam, will the return code tell
 me that both GBUdb *and *rules caught the spam? Or do I get one code 
 instead of the other (if so, which one?)

If you turn off truncate then you will see the following results by
default in a conventional command-line implementation:

* For messages that match pattern rules you will see the pattern rule
result.

* If a message fails to match a pattern rule but would have been
truncated then it will be treated as black and you will get result
code 40.

* If a message fails to match a pattern rule but the IP falls in the
black range then you will get the black result code 40.

* If the message fails to match a pattern rule and the IP falls in the
caution range then you will get an bad IP result code 63. This is the
same result code you get from SNF when an IP pattern rule has matched.
IP pattern rules are deprecated and will be phased out over time -
GBUdb replaces them.

If you call SNF directly via XCI, or use the command line utility with
the -xhdr and capture the output then you also have the ability to
configure SNF to provide detailed information about the scan including
the GBUdb data and all available pattern matches. You could also mine
this data from the log files if you wish.

Note that you can set the x-header option to api and it will be
available to the XCI and command line interfaces without being
injected into the message.

--- One other thing ---

You can