[
https://issues.apache.org/jira/browse/METRON-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259632#comment-16259632
]
ASF GitHub Bot commented on METRON-1321:
----------------------------------------
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152075990
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+ // add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
- metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+ // the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
The calculations were done as Double and given to ES. However, there's no
definition of the field in ES (It just used automatic mapping), so it was given
the ES double.
> Metaalert Threat Score Type Does Not Match Sensor Indices
> ---------------------------------------------------------
>
> Key: METRON-1321
> URL: https://issues.apache.org/jira/browse/METRON-1321
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
>
> Open up the Alerts UI, then sort by threat triage score.
> The following request is sent.
> {code}
> {
> "indices": [],
> "facetFields": [
> "source:type",
> "ip_src_addr",
> "ip_dst_addr",
> "host",
> "enrichments:geo:ip_dst_addr:country"
> ],
> "query": "*",
> "from": 0,
> "size": 25,
> "sort": [
> {
> "field": "threat:triage:score",
> "sortOrder": "asc"
> }
> ]
> }
> {code}
> The following error is logged from the API side.
> {code}
> 17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
> mismatch: snort_index_2017.11.20.16.threat:triage:score has type float while
> metaalert_index.threat:triage:score has type double. Defaulting type to
> other.
> 17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
> mismatch: bro_index_2017.11.20.16.id has type string while
> snort_index_2017.11.20.16.id has type integer. Defaulting type to other.
> 17/11/20 16:55:40 ERROR dao.ElasticsearchRequestSubmitter: Failed to execute
> search; error='NotSerializableExceptionWrapper: class_cast_exception:
> java.lang.Double cannot be cast to java.lang.Float',
> search='{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"}}}}'
> Failed to execute phase [query], [reduce] ; shardFailures
> {[0KqVPgyOT2KKCjTKZYIl3Q][bro_index_2017.11.20.16][0]:
> RemoteTransportException[[node1][192.168.66.121:9300][indices:data/read/search[phase/query]]];
> nested: SearchParseException[failed to parse search source
> [{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"}}}}]];
> nested: IllegalArgumentException[No mapper found for type [other]]; }
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:176)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:147)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:144)
> at
> org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:41)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:819)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:803)
> at
> org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:793)
> at
> org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
> at
> org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
> at
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:369)
> at
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
> at
> org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
> at
> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
> at
> org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: NotSerializableExceptionWrapper[class_cast_exception:
> java.lang.Double cannot be cast to java.lang.Float]
> at java.lang.Float.compareTo(Float.java:49)
> at
> org.apache.lucene.search.FieldComparator.compareValues(FieldComparator.java:132)
> at
> org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:170)
> at
> org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:119)
> at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263)
> at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140)
> at org.apache.lucene.search.TopDocs.mergeAux(TopDocs.java:255)
> at org.apache.lucene.search.TopDocs.merge(TopDocs.java:232)
> at
> org.elasticsearch.search.controller.SearchPhaseController.sortDocs(SearchPhaseController.java:253)
> at
> org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.moveToSecondPhase(SearchQueryThenFetchAsyncAction.java:72)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.innerMoveToSecondPhase(AbstractSearchAsyncAction.java:374)
> at
> org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:171)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
