[sniffer] Re: xci scanner command
Richard Stupek wrote: A question about using the XCI bad command. Assume an email passes through sniffer and does not trigger any rules, I then run it through and determine it is in fact spam. I send a bad command to let sniffer know the IP address had a bad event. Won't the good event that would occur due the spam passing through nullify the bad event? It won't nullify it, but it will balance it off. Should I post 2 bad events for each mail that is caught after sniffer? That may actually work against you because each event increases the confidence figure. One of the surprises we discovered as we began deploying GBUdb was that some legitimate sources can produce spam at rates well above 85%!! In order to avoid false positives our conservative settings have to be extremely high. They can be adjusted by each system administrator but this is not commonly done (yet) as far as I know. http://www.armresearch.com/support/articles/technology/GBUdb/referenceSettings.jsp We have researched this even rewrite scenario and have discussed creating flip and flop events to switch good events to bad and bad events to good respectively. In the mean time you can simulate that functionality by using the test and set functions. Test the IP to get the good and bad count data. Then calculate the values you want to see by subtracting a point from the good event count and adding a point to the bad count. Then set the counts for the IP with the new values. http://www.armresearch.com/support/articles/software/snfClient/commandLine.jsp#performIPTesting http://www.armresearch.com/support/articles/software/snfClient/commandLine.jsp#updateGBUdb Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
A question on GBUDB utilization. I show a current utilization of 95% (from the log file) which I assume means the amount of memory used from what is set aside for gbudb entries. Is that correct? What happens when more entries are added? Does the GBUdb grow or does it get pruned out? Will gbudb XCI commands (like bad) show up in a log file? On Fri, Feb 13, 2009 at 3:27 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: So there would not be a real benefit to passing the IP over when it is the is already in the mail having been added by the mail server? Correct. The vast majority of the time a properly configured SNF + GBUdb can learn the original source of the IP even if you have multiple gateways in your infrastructure. You can even teach SNF + GBUdb to learn to see past the infrastructure of other ISPs in many cases. For example you might teach it to see past a DSL provider's outbound servers so that it can map IP reputations based on individual message sources on their network provided they include Received headers you can understand and predict (to some extent). This way GBUdb can provide pinpoint accuracy instead of a rough average of every source on that network. That said, there are still some times where you might want to explicitly define the source IP even if it is present in the Received headers. For example, one of our larger customers has a complex infrastructure. They found that it was easier to explicitly provide the source IP than to train SNF + GBUdb to understand their structure and the inevitable changes that go on through time. Another large customer has developed a very complex system for determining the precise original source for a message even when it is relayed through many large ISPs. They chose to provide that IP rather than have SNF + GBUdb attempt to duplicate that effort. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Richard Stupek wrote: A question on GBUDB utilization. I show a current utilization of 95% (from the log file) which I assume means the amount of memory used from what is set aside for gbudb entries. Is that correct? Yes. What happens when more entries are added? Does the GBUdb grow or does it get pruned out? Both. When more space is needed the ram allocated to GBUdb will grow to accommodate the need. This should happen infrequently. When entries are no longer active they are dropped to make room for additional entries. In practice the GBUdb data size stabilizes quickly and then doesn't change much. Once per day or as otherwise specified by GBUdb condensation triggers the GBUdb data will be condensed. This means that all of the good and bad event counts are divided in half. This has the effect of retaining the ratios (probability of spam) while reducing the event counts (confidence figure). Any ugly entries that drop to zero are dropped from the system (forgotten). The remaining live entries are placed in a new GBUdb and the old one is thrown away. If the GBUdb size can shrink during a condensation run then it will. Will gbudb XCI commands (like bad) show up in a log file? No. These are administrative entries so they don't get reported in scan activity. We may change this later but at present there are no requests for it. Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Thanks for the info. Is there any diagnostic information available when a gbudb sync occurs? On Tue, Feb 17, 2009 at 4:35 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: A question on GBUDB utilization. I show a current utilization of 95% (from the log file) which I assume means the amount of memory used from what is set aside for gbudb entries. Is that correct? Yes. What happens when more entries are added? Does the GBUdb grow or does it get pruned out? Both. When more space is needed the ram allocated to GBUdb will grow to accommodate the need. This should happen infrequently. When entries are no longer active they are dropped to make room for additional entries. In practice the GBUdb data size stabilizes quickly and then doesn't change much. Once per day or as otherwise specified by GBUdb condensation triggers the GBUdb data will be condensed. This means that all of the good and bad event counts are divided in half. This has the effect of retaining the ratios (probability of spam) while reducing the event counts (confidence figure). Any ugly entries that drop to zero are dropped from the system (forgotten). The remaining live entries are placed in a new GBUdb and the old one is thrown away. If the GBUdb size can shrink during a condensation run then it will. Will gbudb XCI commands (like bad) show up in a log file? No. These are administrative entries so they don't get reported in scan activity. We may change this later but at present there are no requests for it. Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Richard Stupek wrote: Thanks for the info. Is there any diagnostic information available when a gbudb sync occurs? You can always see the current status of GBUdb in your status.* files. If you append these logs you can follow the state of the system through time using pre-compiled statistics including the size of GBUdb. This can be done with one entry per minute (status.minute) or one entry per hour (status.hour) or both. In the same status log you can see when the last GBUdb condensation event occurred. No other useful data is produced by a condensation run so none is recorded. If there is something you would like to see then please let us know and we will consider adding features to support your request(s). Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
A question about using the XCI bad command. Assume an email passes through sniffer and does not trigger any rules, I then run it through and determine it is in fact spam. I send a bad command to let sniffer know the IP address had a bad event. Won't the good event that would occur due the spam passing through nullify the bad event? Should I post 2 bad events for each mail that is caught after sniffer? On Tue, Feb 17, 2009 at 5:02 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: Thanks for the info. Is there any diagnostic information available when a gbudb sync occurs? You can always see the current status of GBUdb in your status.* files. If you append these logs you can follow the state of the system through time using pre-compiled statistics including the size of GBUdb. This can be done with one entry per minute (status.minute) or one entry per hour (status.hour) or both. In the same status log you can see when the last GBUdb condensation event occurred. No other useful data is produced by a condensation run so none is recorded. If there is something you would like to see then please let us know and we will consider adding features to support your request(s). Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Richard Stupek wrote: Which of the 2 scan commands should we use to scan a message? Does sending the IP address help improve scanning? snfxciscannerscan file='filepath'//scanner/xci/snf OR snfxciscannerscan file='filepath' xhdr='no' log='no' ip='12.34.56.78'//scanner/xci/snf That depends on your needs. If you want SNF + GBUdb to learn IPs by reading through the Received headers then you would NOT include the ip= attribute. If you want to tell SNF + GBUdb what the source IP was for the message explicitly then you DO include the ip= attribute. See: http://www.armresearch.com/support/articles/software/snfServer/xci/scanner.jsp The ip='12.34.56.78' attribute is optional and indicates that the source IP for the message has already been determined as 12.34.56.78. If this option is used then GBUdb training directives may not function as expected. This option is best used when the calling application has already determined the correct source IP for the message and wishes to force the source IP value rather than have SNF+GBUdb discover it from Received headers in the message. Note: It is often best to let SNF + GBUdb determine the source IP for a given message based on the Received headers. If the connecting IP is not already represented in the top Received header for the message then the calling application should create one in the top of the temporary file SNF is going to scan. Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
So there would not be a real benefit to passing the IP over when it is the is already in the mail having been added by the mail server? On Fri, Feb 13, 2009 at 2:56 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: Which of the 2 scan commands should we use to scan a message? Does sending the IP address help improve scanning? snfxciscannerscan file='filepath'//scanner/xci/snf OR snfxciscannerscan file='filepath' xhdr='no' log='no' ip='12.34.56.78'//scanner/xci/snf That depends on your needs. If you want SNF + GBUdb to learn IPs by reading through the Received headers then you would NOT include the ip= attribute. If you want to tell SNF + GBUdb what the source IP was for the message explicitly then you DO include the ip= attribute. See: http://www.armresearch.com/support/articles/software/snfServer/xci/scanner.jsp The ip='12.34.56.78' attribute is optional and indicates that the source IP for the message has already been determined as 12.34.56.78. If this option is used then GBUdb training directives may not function as expected. This option is best used when the calling application has already determined the correct source IP for the message and wishes to force the source IP value rather than have SNF+GBUdb discover it from Received headers in the message. Note: It is often best to let SNF + GBUdb determine the source IP for a given message based on the Received headers. If the connecting IP is not already represented in the top Received header for the message then the calling application should create one in the top of the temporary file SNF is going to scan. Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Richard Stupek wrote: So there would not be a real benefit to passing the IP over when it is the is already in the mail having been added by the mail server? Correct. The vast majority of the time a properly configured SNF + GBUdb can learn the original source of the IP even if you have multiple gateways in your infrastructure. You can even teach SNF + GBUdb to learn to see past the infrastructure of other ISPs in many cases. For example you might teach it to see past a DSL provider's outbound servers so that it can map IP reputations based on individual message sources on their network provided they include Received headers you can understand and predict (to some extent). This way GBUdb can provide pinpoint accuracy instead of a rough average of every source on that network. That said, there are still some times where you might want to explicitly define the source IP even if it is present in the Received headers. For example, one of our larger customers has a complex infrastructure. They found that it was easier to explicitly provide the source IP than to train SNF + GBUdb to understand their structure and the inevitable changes that go on through time. Another large customer has developed a very complex system for determining the precise original source for a message even when it is relayed through many large ISPs. They chose to provide that IP rather than have SNF + GBUdb attempt to duplicate that effort. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
