[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 +1 from me as well. Great job @justinleet ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 +1 This was more challenging than meets the eye. Thanks for working through this @justinleet . We have some use cases to figure out, but this is a good first step toward metaalerts. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 Master is merged in, so this Travis run should be legit ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 Travis failure should be resolved once https://github.com/apache/metron/pull/816 is in, and I'll merge in master and push again once it is. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen I added testing around the status stuff, and a couple docs. Let me know if there's anything else that should be updated (or isn't clear or whatever else). ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/811
### Testing
Testing purely based on the REST API. Having said that, if you want to do
like @nickwallen did and pull in https://github.com/apache/metron/pull/803 into
the same branch, you should be able to follow his examples that he posted
screenshots for. Having done this myself, there should be no merge conflicts.
Throughout the following instructions, make sure to substitute GUIDs and
index appropriately.
Create a meta alert.
```
{
"groups": [
"string"
],
"guidToIndices": {
"c4060f38-aa20-4670-a85f-9d6e9fe0931a":"bro_index_2017.10.25.13"
}
}
```
Retrieve the meta alert. Either use curl or use `findOne` endpoint on the
returned GUID from the previous response.
```
curl -XGET
"node1:9200/metaalert_index/metaalert_doc/54f979ec-1491-4c8d-9292-db9202cdba86?pretty"
```
Ensure the contained alert has the appropriate guid, e.g.
```
...
"guid": "c4060f38-aa20-4670-a85f-9d6e9fe0931a",
...
```
Retrieve the child alert, again through curl or `findOne`
```
curl -XGET
"node1:9200/bro_index_2017.10.25.13/bro_doc/c4060f38-aa20-4670-a85f-9d6e9fe0931a?pretty"
```
Ensure the meta alerts field contains a link to the meta alert we created.
```
"metaalerts": [
"54f979ec-1491-4c8d-9292-db9202cdba86"
]
```
Run a search that should match both and ensure only the meta alert is
returned
```
{
"fields": [
"*"
],
"from": 0,
"size": 10,
"indices": [
"metaalert",
"bro"
],
"query": "guid:\"c4060f38-aa20-4670-a85f-9d6e9fe0931a\" OR
alert.guid:\"c4060f38-aa20-4670-a85f-9d6e9fe0931a\""
}
```
returns
```
{
"total": 1,
"results": [
{
"id": "54f979ec-1491-4c8d-9292-db9202cdba86",
"source": {
"_timestamp": 1508942592157
},
"score": 1,
"index": "metaalert_index"
}
]
}
```
This can also be run where multiple matches happen
```
{
"fields": [
"*"
],
"from": 0,
"size": 10,
"indices": [
"metaalert",
"bro"
],
"query": "ip_dst_addr:192.168.66.121 OR alert.ip_dst_addr:192.168.66.121"
}
```
Results are omitted for size, but the child alert should not show up, but
the meta alert should. Just do a find on the results if there are a lot or
modify the query.
Updating Status
A status update will carry through for all the child alerts. I.e. going
from "active" to "inactive" and vice-versa will manage the links.
```
{
"guid": "54f979ec-1491-4c8d-9292-db9202cdba86",
"index": "metaalert",
"patch": [{
"op": "replace"
, "path": "/status"
, "value": "inactive"
}],
"sensorType": "metaalert",
"source": {}
}
```
Retrieving the meta alert will now give an "inactive" status
Retrieving the alert itself again will now have the metaalerts field as
follows (assuming this alert isn't in any other meta alerts):
```
"metaalerts": []
```
It should also show up in our search again (but the metaalert won't!)
Try changing it back to active and ensure the child is linked again and is
hidden from the results.
```
{
"guid": "54f979ec-1491-4c8d-9292-db9202cdba86",
"index": "metaalert",
"patch": [{
"op": "replace"
, "path": "/status"
, "value": "active"
}],
"sensorType": "metaalert",
"source": {}
}
```
The link in the alert should also be back.
---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 Thanks @justinleet . This is working well with @iraghumitra UI work in #803. The metalerts show-up in the left-side "Filters" panel, which provides a decent short-cut to retrieve the metaalerts. For example, here I created 4 metaalerts. If I click on "metaalerts" on that left panel, it filters everything down to metaalerts.  Filtering does nicely retrieve both alerts and metaalerts.  ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen It's not a perfect solution, but I added the "source:type" field for consistency and to allow for filtering and retrieval of just meta alerts in an obvious way. Should just be "source:type" of "metaalert" ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/811 Kick Travis ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/811
Another thing I noticed is that there does not seem to be a way to retrieve
meta-alerts that you have created.
For example, I created a meta-alert where `host:ip-addr.es`. Assuming we
exclude metaalerts from all the nice "group by" functionality, then I need to
find it using search. But if I search with `host:ip-addr.es` or even
`alert.host.ip-addr.es` it does not return any results.
When using `alert.host` the UI submits the following which returns no
results.
```
{
"indices": [
"websphere",
"snort",
"asa",
"bro",
"yaf",
"metaalert"
],
"facetFields": [
"source:type",
"ip_src_addr",
"ip_dst_addr",
"host",
"enrichments:geo:ip_dst_addr:country"
],
"query": "(host:ip\\-addr.es OR alert.host:ip\\-addr.es)",
"from": 0,
"size": 25,
"sort": [
{
"field": "host",
"sortOrder": "asc"
}
]
}
```
I ran this by @justinleet and we found that it is the `sort` field that
prevents the meta-alerts from being returned. Removing the sort field like the
following query does actually return the meta-alert.
```
{
"indices": [
"websphere",
"snort",
"asa",
"bro",
"yaf",
"metaalert"
],
"facetFields": [
"source:type",
"ip_src_addr",
"ip_dst_addr",
"host",
"enrichments:geo:ip_dst_addr:country"
],
"query": "(host:ip\\-addr.es OR alert.host:ip\\-addr.es)",
"from": 0,
"size": 25
}
```
---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen to avoid scope creep on this PR I created a follow-on PR to figure out how to represent meta alerts in the facet panel. https://issues.apache.org/jira/browse/METRON-1276 I agreee with @merrimanr that meta alerts contains multiple alerts each with with distinct meta data and faceting them with the non-grouped alerts doesn't make sense IMO. We do need to figure out a way to visualize metadata of the meta alert (maybe have a separate facet panel for meta alerts?), but I think this will add scope to this PR and therefore should be resolved outside of this PR. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 > @nickwallen suppose you have a metaalert that contains 2 alerts. Then suppose each alert has a different value for the host field. If you grouped on host, which group would you expect the metaalert to appear in? I would say both. Let me try and explain that. To me, the "group by" functionality is super-powerful for digging down deep into the alerts data. When I group by hostname and open that accordian, I expect to see all the alerts related to that hostname. That would include a meta-alert that has even 1 contained alert related to that hostname. Maybe there are other corner cases that I am not considering, but that's what I would expect. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 > I find that extremely confusing as a user of the tool. To explain that a bit more (and continuing with that same basic example)... As a user I created a meta-alert where the hostname is "ip-addr.es". Since I created a meta-alert around that specific hostname, that must be a pretty important host name. It is probably something I am investigating right now. Now imagine I ask my Tier III to take a look at that weird hostname. He's going to do something like this to attempt to find that problematic hostname.  It appears that the hostname "ip-addr.es" is completely missing. It is as if we lost data. This is the kind of work flow that I think is very confusing. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen suppose you have a metaalert that contains 2 alerts. Then suppose each alert has a different value for the host field. If you grouped on host, which group would you expect the metaalert to appear in? ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 @james-sirota I find that extremely confusing as a user of the tool. We need to document that fact and some reasoning behind it at the very least. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen what you are looking at is a desired behavior. If the alerts are a part of the meta alert they do not appear in the facets ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/811 I believe excluding metaalerts from the group by view is the desired behavior. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/811
I am seeing another issue that may or may not be related. It seems that
when I am using the "group by" functionality, I cannot see meta-alerts at all.
(1) If I am not using the "group by", I can see the meta-alert perfectly
fine.
(2) Now I want to group by host. I click the "host" group by widget, but
there are no results. I am left thinking... Where did the meta-alert go? The
only way I can see the meta-alert is to not use the "group by" functionality.
(3) And the UI screenshots match what is returned by the underlying API.
In the case of the missing meta-alert, this is the request/response.
Request:
```
{
"indices": [
"websphere",
"snort",
"asa",
"bro",
"yaf",
"metaalert"
],
"scoreField": "threat:triage:score",
"groups": [
{
"field": "host",
"order": {
"sortOrder": "desc",
"groupOrderType": "term"
}
}
],
"query": "(host:ip\\-addr.es OR alert.host:ip\\-addr.es)"
}
```
Response:
```
{"groupedBy":"host","groupResults":[]}
```
---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 It appears to me that the alerts contained within a meta-alert are not contributing to the facet counts returned by a search request. I think we still do want that to happen. Let me explain with an example and screenshots. Note: I am testing this together with the UI changes in #803 . 1. First, I isolate 10 alerts with a specific host name that I want to work with. I have turned off ingest so that no additional alerts should appear during the running of this example. I can see that the facet counts are what I would expect.  2. Next, I group by host so that I can create my meta-alert.   3. Immediately after creating the meta-alert, I do not immediately see it. I think this is a problem with the UI itself not refreshing after creating the alert. This might need fixed in #803 .  4. If I then trigger another search, I do see the meta-alert. Great!  5. Next I just expand the meta-alert to validate that the 10 original alerts were added. You can see from this screenshot that the facet counts all show 0. This tells me that the facet counts are not including meta-alerts.  ---
