[
https://issues.apache.org/jira/browse/METRON-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878651#comment-15878651
]
ASF GitHub Bot commented on METRON-686:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/438
I'm of the opinion that the flattening should be writer-specific and that
should be a function of the writer config with the default to be specified by
the writer implementation. This way we can have our cake and eat it too.
Also, we could ALREADY be in a situation where messages aren't flat (imagine a
situation where a stellar function returns a map or a list and it get assigned
to a field). The *only* safe way to do this is to enforce it at the writer,
IMO. This is one of the stated benefits to extracting writer configs into
their own structures.
Regarding existing conventions, this one was around when I joined the
project. I might be wrong, but it was an early convention. The reasoning, as
I recall, was multi-fold:
* Solr didn't handle it
* Interacting with complex structures was deemed to be difficult
* Indexing nested structures had some performance implications
As to how to move forward, my suggestion is conform to convention for this
JIRA and flatten. I created a JIRA to track the flattening effort at
[METRON-736](https://issues.apache.org/jira/browse/METRON-736)
> Record Rule Set that Fired During Threat Triage
> -----------------------------------------------
>
> Key: METRON-686
> URL: https://issues.apache.org/jira/browse/METRON-686
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> h3. Problem
> There is little transparency into the Threat Triage process itself. When
> Threat Triage runs, all I get is a score. I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused a
> rule to trigger.
> More specifically, there is no way to generate a message that looks like "The
> host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold
> of '202'". This makes it difficult for an analyst to action the alert.
> h3. Proposed Solution
> To improve the transparency of the Threat Triage process, I am proposing
> these enhancements.
> (1) Threat Triage should attach to each message all of the rules that fired
> in addition to the total calculated threat triage score.
> (2) Threat Triage should allow a custom message to be generated for each
> rule. The custom message would allow for some form of string interpolation
> so that I can add specific values from each message to the generated alert.
> We could allow this in one or both of the new fields that Casey just added,
> name and comment.
> (3) The specific method of string interpolation will be implemented under a
> separate issue.
> h3. Example
> (1) In this example, we have a telemetry message with a field called 'value'
> that we need to monitor. In Enrichment, I calculate some sort of value
> threshold, over which an alert should be generated.
> (2) In Threat Triage, I use the calculated value threshold to alert on any
> message that has a value exceeding this threshold.
> (3) By leveraging a new field called 'reason', I can embed values from the
> message, like the hostname, value, and value threshold, into the alert
> produced by Threat Triage.
> {code}
> "triageConfig" : {
> "riskLevelRules" : [ {
> "name" : "Abnormal DNS Port",
> "rule" : "source.type == 'bro' and protocol == 'dns' and ip_dst_port
> != 53",
> "score" : 10.0,
> "reason" : "FORMAT('Abnormal DNS Port: expected: 53, found: %s:%d',
> ip_dst_addr, ip_dst_port)"
> } ],
> "aggregator" : "MAX",
> "aggregationConfig" : { }
> }
> {code}
> (4) The Threat Triage process today would add only the total calculated score.
> {code}
> "threat.triage.level": 10.0
> {code}
> With this proposal, Threat Triage would add the following to the message.
> {code}
> "threat.triage.level":{
> "score":10.0,
> "rules":[
> {
> "name":"Abnormal DNS Port",
> "comment":null
> "score":10.0,
> "reason":"Abnormal DNS Port: expected: 53, found: 224.0.0.251:5353",
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
