[
https://issues.apache.org/jira/browse/METRON-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259588#comment-16259588
]
ASF GitHub Bot commented on METRON-1321:
----------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/845
METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices
After creating Metaalerts in the Alerts UI, I am unable to sort by threat
triage score. The exception that is logged is shown in the issue. When
opening the Developer view in Chrome, I can see the API responds with an error.
An error message is logged under `/var/log/metron/metron-rest.log`.
## What happened?
An overall threat score is added to a Metaalert whenever it has any child
alerts. This overall threat score is a summary of the threat scores from each
of the child alerts.
This field is also named the same as the threat score fields from the
sensor indices; 'threat:triage:score'. The type of this field must be the same
as the 'threat:triage:score' fields from each of the sensor indices. Otherwise
the Alerts UI cannot properly sort alerts and metaalerts in the same table/view
in the UI.
## Changes
I updated the `ElasticsearchMetaalertDao` so that when it creates the
overall threat score, it is added as a Float to match the type expected from
the other sensor indices.
## Testing
1. Spin up Full Dev
2. Open up the Alerts UI
3. Sort by triage score and assure no errors occur
4. Create a Metaalert.
5. Sort by triage score and assure no errors occur
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1321
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/845.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #845
----
commit c780058aad1611edb0ea7a4fae84038ee07d50fd
Author: Nick Allen <[email protected]>
Date: 2017-11-20T18:05:37Z
METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices
----
> Metaalert Threat Score Type Does Not Match Sensor Indices
> ---------------------------------------------------------
>
> Key: METRON-1321
> URL: https://issues.apache.org/jira/browse/METRON-1321
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
>
> Open up the Alerts UI, then sort by threat triage score.
> The following request is sent.
> {code}
> {
> "indices": [],
> "facetFields": [
> "source:type",
> "ip_src_addr",
> "ip_dst_addr",
> "host",
> "enrichments:geo:ip_dst_addr:country"
> ],
> "query": "*",
> "from": 0,
> "size": 25,
> "sort": [
> {
> "field": "threat:triage:score",
> "sortOrder": "asc"
> }
> ]
> }
> {code}
> The following error is logged from the API side.
> {code}
> 17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
> mismatch: snort_index_2017.11.20.16.threat:triage:score has type float while
> metaalert_index.threat:triage:score has type double. Defaulting type to
> other.
> 17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
> mismatch: bro_index_2017.11.20.16.id has type string while
> snort_index_2017.11.20.16.id has type integer. Defaulting type to other.
> 17/11/20 16:55:40 ERROR dao.ElasticsearchRequestSubmitter: Failed to execute
> search; error='NotSerializableExceptionWrapper: class_cast_exception:
> java.lang.Double cannot be cast to java.lang.Float',
> search='{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"}}}}'
> Failed to execute phase [query], [reduce] ; shardFailures
> {[0KqVPgyOT2KKCjTKZYIl3Q][bro_index_2017.11.20.16][0]:
> RemoteTransportException[[node1][192.168.66.121:9300][indices:data/read/search[phase/query]]];
> nested: SearchParseException[failed to parse search source
> [{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"}}}}]];
> nested: IllegalArgumentException[No mapper found for type [other]]; }
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:176)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:147)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:144)
> at
> org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:41)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:819)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:803)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:793)
> at
> org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
> at
> org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
> at
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:369)
> at
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
> at
> org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
> at
> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
> at
> org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: NotSerializableExceptionWrapper[class_cast_exception:
> java.lang.Double cannot be cast to java.lang.Float]
> at java.lang.Float.compareTo(Float.java:49)
> at
> org.apache.lucene.search.FieldComparator.compareValues(FieldComparator.java:132)
> at
> org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:170)
> at
> org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:119)
> at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263)
> at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140)
> at org.apache.lucene.search.TopDocs.mergeAux(TopDocs.java:255)
> at org.apache.lucene.search.TopDocs.merge(TopDocs.java:232)
> at
> org.elasticsearch.search.controller.SearchPhaseController.sortDocs(SearchPhaseController.java:253)
> at
> org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.moveToSecondPhase(SearchQueryThenFetchAsyncAction.java:72)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.innerMoveToSecondPhase(AbstractSearchAsyncAction.java:374)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:171)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
