[jira] [Commented] (METRON-442) Incorrect/Approximated threat triage level is set when the score is configured to some max value

2016-10-07 Thread Casey Stella (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554632#comment-15554632
 ] 

Casey Stella commented on METRON-442:
-

I looked into this, this is an artifact of using Doubles as threat triage rule 
scores.  There is a fundamental limitation to the precision that IEEE 64-bit 
floating point values can hold and you've hit it.  IEEE 754 gives you 15 to 17 
significant decimal digits and you're at 18.

Consider the following experiment, taking Metron out of the equation:
 
Assert.assertEquals(9.223372036854776E18, 9223372036854775807d, 
Double.MIN_VALUE);
Assert.assertEquals(9223372036854775808d, 9223372036854775807d, 
Double.MIN_VALUE);

Now, we could change the threat triage rules to use BigDecimal instead of 
Double, but, frankly, I'm not sure that I see much of a reason to take the 
performance hit to gain further expressibility.  Also, as soon as you get into 
the indices, they will be treated as doubles again in elasticsearch, losing any 
gain that you had in precision.

I am inclined to view this as an artifact of the representation of doubles in 
Java and not worth fixing.  Thoughts?

> Incorrect/Approximated threat triage level is set when the score is 
> configured to some max value
> 
>
> Key: METRON-442
> URL: https://issues.apache.org/jira/browse/METRON-442
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.2.2BETA
>Reporter: Neha Sinha
>
> Hi,
> I have specified the following threat config for snort sensor  :-
> 
> "threatIntel" : {
> "triageConfig" : {
>   "riskLevelRules" : {
> "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))" : 9223372036854775807
>   }
> }
>   }
> ===
> Expected threat.triage.level = 9223372036854775807
> Actual threat.triage.level = 9223372036854776000
> *Enrichments log*
> ===
> 2016-08-22 09:42:57.509 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: Found 
> sensor enrichment config.
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: Found 
> threat triage config: 
> ThreatTriageConfig{riskLevelRules={not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807}, aggregator=MAX, aggregationConfig={}}
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] Marked snort as 
> triage level 9.223372036854776E18 with rules not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807
> 2016-08-22 09:42:57.510 o.a.m.w.BulkWriterComponent [DEBUG] Acking 1 
> tuples2016-08-22 09:42:57.509 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: 
> Found sensor enrichment config.
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: Found 
> threat triage config: 
> ThreatTriageConfig{riskLevelRules={not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807}, aggregator=MAX, aggregationConfig={}}
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] Marked snort as 
> triage level 9.223372036854776E18 with rules not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807
> 2016-08-22 09:42:57.510 o.a.m.w.BulkWriterComponent [DEBUG] Acking 1 tuples
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-442) Incorrect/Approximated threat triage level is set when the score is configured to some max value

2016-09-20 Thread Neha Sinha (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508748#comment-15508748
 ] 

Neha Sinha commented on METRON-442:
---

The content of enrichment json file in hadoop is this :-
Command :- hadoop fs -cat 
/apps/metron/enrichment/indexed/bro/enrichment-null-0-0-1471938799700.json

===
{"adapter.threatinteladapter.end.ts":"1471939557742","bro_timestamp":"1.471939556638758E9","status_code":404,"ip_dst_port":80,"enrichmentsplitterbolt.splitter.end.ts":"1471939557740","enrichments.geo.ip_dst_addr.city":"Phoenix","enrichments.geo.ip_dst_addr.latitude":"33.4499","adapter.hostfromjsonlistadapter.end.ts":"1471939557741","enrichmentsplitterbolt.splitter.begin.ts":"1471939557740","enrichments.geo.ip_dst_addr.country":"US","enrichments.geo.ip_dst_addr.locID":"3886","adapter.geoadapter.begin.ts":"1471939557740","enrichments.geo.ip_dst_addr.postalCode":"85004","uid":"CgrsLeHSOZRGMJdSa","resp_mime_types":["text\/html"],"trans_depth":1,"protocol":"http","original_string":"HTTP
 | id.orig_p:49199 status_code:404 method:POST request_body_len:96 id.resp_p:80 
orig_mime_types:[\"text\\\/plain\"] 
uri:\/wp-content\/themes\/twentyfifteen\/img5.php?l=8r1gf1b2t1kuq42 tags:[] 
uid:CgrsLeHSOZRGMJdSa resp_mime_types:[\"text\\\/html\"] trans_depth:1 
orig_fuids:[\"FlINOb2WXgQZh3YG0j\"] host:runlove.us status_msg:Not Found 
id.orig_h:192.168.138.158 response_body_len:357 user_agent:Mozilla\/4.0 
(compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) 
ts:1.471939556638758E9 id.resp_h:204.152.254.221 
resp_fuids:[\"F6LnXgRnQD51mbhfb\"]","ip_dst_addr":"204.152.254.221","threat.triage.level":9.223372036854776E18,"threatinteljoinbolt.joiner.ts":"1471939557742","enrichments.geo.ip_dst_addr.dmaCode":"753","host":"runlove.us","enrichmentjoinbolt.joiner.ts":"1471939557741","adapter.hostfromjsonlistadapter.begin.ts":"1471939557741","threatintelsplitterbolt.splitter.begin.ts":"1471939557741","enrichments.geo.ip_dst_addr.longitude":"-112.0712","ip_src_addr":"192.168.138.158","user_agent":"Mozilla\/4.0
 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
6.0)","resp_fuids":["F6LnXgRnQD51mbhfb"],"timestamp":1471939556638,"method":"POST","request_body_len":96,"is_alert":"true","orig_mime_types":["text\/plain"],"uri":"\/wp-content\/themes\/twentyfifteen\/img5.php?l=8r1gf1b2t1kuq42","source.type":"bro","tags":[],"adapter.geoadapter.end.ts":"1471939557740","adapter.threatinteladapter.begin.ts":"1471939557742","threatintelsplitterbolt.splitter.end.ts":"1471939557741","orig_fuids":["FlINOb2WXgQZh3YG0j"],"ip_src_port":49199,"enrichments.geo.ip_dst_addr.location_point":"33.4499,-112.0712","status_msg":"Not
 Found","response_body_len":357}

===
Threat triage level :- "threat.triage.level":9.223372036854776E18
Note:-I tested the max value with Bro enrichment in this case.


> Incorrect/Approximated threat triage level is set when the score is 
> configured to some max value
> 
>
> Key: METRON-442
> URL: https://issues.apache.org/jira/browse/METRON-442
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.2.2BETA
>Reporter: Neha Sinha
>
> Hi,
> I have specified the following threat config for snort sensor  :-
> 
> "threatIntel" : {
> "triageConfig" : {
>   "riskLevelRules" : {
> "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))" : 9223372036854775807
>   }
> }
>   }
> ===
> Expected threat.triage.level = 9223372036854775807
> Actual threat.triage.level = 9223372036854776000
> *Enrichments log*
> ===
> 2016-08-22 09:42:57.509 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: Found 
> sensor enrichment config.
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: Found 
> threat triage config: 
> ThreatTriageConfig{riskLevelRules={not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807}, aggregator=MAX, aggregationConfig={}}
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] Marked snort as 
> triage level 9.223372036854776E18 with rules not(IN_SUBNET(ip_dst_addr, 
> '192.168.0.0/24'))=9223372036854775807
> 2016-08-22 09:42:57.510 o.a.m.w.BulkWriterComponent [DEBUG] Acking 1 
> tuples2016-08-22 09:42:57.509 o.a.m.e.b.ThreatIntelJoinBolt [DEBUG] snort: 
> Found sensor enrichment config.
> 2016-08-22 09:42:57.510 o.a.m.e.b.ThreatIntelJoinBolt