[
https://issues.apache.org/jira/browse/METRON-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644410#comment-15644410
]
ASF GitHub Bot commented on METRON-554:
---------------------------------------
GitHub user mmiklavc opened a pull request:
https://github.com/apache/incubator-metron/pull/346
METRON-554: Require proper error handling when invalid input is fed to
Threat triage rules
This completes https://issues.apache.org/jira/browse/METRON-554
Non-boolean-returning functions used as predicates in the triage rules will
cause generic exceptions like the following:
```
java.lang.ClassCastException: Cannot cast java.lang.String to
java.lang.Boolean
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_60]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:53)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
```
This fix makes it clear where the problem is.
**Testing**
Edit the bro enrichment config in
$METRON_HOME/config/zookeeper/enrichments/bro.json
Add a predicate to the riskLevelRules that does not return a boolean value,
e.g. `"TO_UPPER(protocol)" : 0.92"` as shown in the example below.
```
{
"index": "bro",
"batchSize": 5,
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : {
"exists(ip_dst_addr)" : 0.10,
"TO_UPPER(protocol) == 'HTTP'" : 0.91,
"TO_UPPER(protocol)" : 0.92,
"exists(ip_dst_port)" : 0.20,
"exists(ip_src_port)" : 0.30000000000
},
"aggregator" : "MAX",
"aggregationConfig":
{
"NEGATIVE_VALUES_TRUMP_CONF" : "false"
}
}
}
}
```
Load the new configuration in zookeeper doing the following:
```
$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i
$METRON_HOME/config/zookeeper/
```
The configuration should push out to the enrichment topology. You will need
to wait a bit for new bro messages to percolate through the system. Verify the
new, more specific error message in the storm worker logs, an example of which
is shown below.
example path for enrichment -
/var/log/storm/workers-artifacts/enrichment-7-1478449668/6700/worker.log
Revised Storm worker error message:
```
2016-11-06 16:47:56.325 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join
messages:
{"adapter.threatinteladapter.end.ts":"1478450876296","adapter.threatinteladapter.begin.ts":"1478450876296","source.type":"bro"}
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not
return a boolean value.
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2016-11-06 16:47:56.326 o.a.s.d.executor [ERROR]
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not
return a boolean value.
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
I went with an IllegalArgumentException because, while it's a runtime
exception, this is more of a configuration error.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/incubator-metron METRON-554
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/346.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #346
----
commit fc331b30f18473278884d58714ec3b3451bb2467
Author: Michael Miklavcic <[email protected]>
Date: 2016-11-06T16:17:32Z
Throw meaningful exception when predicate return value fails to cast to
boolean
----
> Require proper error handling when invalid input is fed to Threat triage rules
> ------------------------------------------------------------------------------
>
> Key: METRON-554
> URL: https://issues.apache.org/jira/browse/METRON-554
> Project: Metron
> Issue Type: Improvement
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
