GitHub user mmiklavc opened a pull request:
https://github.com/apache/incubator-metron/pull/346
METRON-554: Require proper error handling when invalid input is fed to
Threat triage rules
This completes https://issues.apache.org/jira/browse/METRON-554
Non-boolean-returning functions used as predicates in the triage rules will
cause generic exceptions like the following:
```
java.lang.ClassCastException: Cannot cast java.lang.String to
java.lang.Boolean
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_60]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:53)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
```
This fix makes it clear where the problem is.
**Testing**
Edit the bro enrichment config in
$METRON_HOME/config/zookeeper/enrichments/bro.json
Add a predicate to the riskLevelRules that does not return a boolean value,
e.g. `"TO_UPPER(protocol)" : 0.92"` as shown in the example below.
```
{
"index": "bro",
"batchSize": 5,
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : {
"exists(ip_dst_addr)" : 0.10,
"TO_UPPER(protocol) == 'HTTP'" : 0.91,
"TO_UPPER(protocol)" : 0.92,
"exists(ip_dst_port)" : 0.20,
"exists(ip_src_port)" : 0.30000000000
},
"aggregator" : "MAX",
"aggregationConfig":
{
"NEGATIVE_VALUES_TRUMP_CONF" : "false"
}
}
}
}
```
Load the new configuration in zookeeper doing the following:
```
$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i
$METRON_HOME/config/zookeeper/
```
The configuration should push out to the enrichment topology. You will need
to wait a bit for new bro messages to percolate through the system. Verify the
new, more specific error message in the storm worker logs, an example of which
is shown below.
example path for enrichment -
/var/log/storm/workers-artifacts/enrichment-7-1478449668/6700/worker.log
Revised Storm worker error message:
```
2016-11-06 16:47:56.325 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join
messages:
{"adapter.threatinteladapter.end.ts":"1478450876296","adapter.threatinteladapter.begin.ts":"1478450876296","source.type":"bro"}
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not
return a boolean value.
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2016-11-06 16:47:56.326 o.a.s.d.executor [ERROR]
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not
return a boolean value.
at
org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55)
~[stormjar.jar:?]
at
org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38)
~[stormjar.jar:?]
at
org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
I went with an IllegalArgumentException because, while it's a runtime
exception, this is more of a configuration error.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/incubator-metron METRON-554
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/346.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #346
----
commit fc331b30f18473278884d58714ec3b3451bb2467
Author: Michael Miklavcic <[email protected]>
Date: 2016-11-06T16:17:32Z
Throw meaningful exception when predicate return value fails to cast to
boolean
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
