Anand Subramanian created METRON-1369:
-----------------------------------------
Summary: squid events are not seen in Alerts UI
Key: METRON-1369
URL: https://issues.apache.org/jira/browse/METRON-1369
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
On a metron cluster, I have added the squid telemetry by following the [Wiki
instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
I am able to see the squid indices present in elasticsearch, but not able to
see them in the alerts UI. On further checking, I found out that this is due to
the missing nested "alert" field in the index mappings--which is required for
the Alerts UI to pick up the index.
This was confirmed in the rest logs as well.
{code}
17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search failure
[1/1]; reason=IllegalStateException: [nested] failed to find nested object
under path [alert], index=squid_index_2017.12.18.10, shard=0,
status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
[squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed
to create query: {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{
"bool" : {
"should" : [
{
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
{
"nested" : {
"query" : {
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
"path" : "alert",
"ignore_unmapped" : false,
"score_mode" : "none",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
{
"bool" : {
"should" : [
{
"term" : {
"status" : {
"value" : "active",
"boost" : 1.0
}
}
},
{
"bool" : {
"must_not" : [
{
"exists" : {
"field" : "status",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"must_not" : [
{
"exists" : {
"field" : "metaalerts",
"boost" : 1.0
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"boost" : 1.0
}
}]; nested: IllegalStateException[[nested] failed to find nested object under
path [alert]];
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:617)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:485)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
at
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340)
at
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337)
at
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1533)
at
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: [nested] failed to find nested
object under path [alert]
at
org.elasticsearch.index.query.NestedQueryBuilder.doToQuery(NestedQueryBuilder.java:281)
at
org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
at
org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
at
org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:420)
at
org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
at
org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
at
org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418)
at
org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
at
org.elasticsearch.index.query.AbstractQueryBuilder.toFilter(AbstractQueryBuilder.java:118)
at
org.elasticsearch.index.query.ConstantScoreQueryBuilder.doToQuery(ConstantScoreQueryBuilder.java:146)
at
org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
at
org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313)
at
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325)
... 14 more
{code}
Now, I was trying to find a way to add this field to the mappings, so I can get
the squid alerts listed. We have a documented process for including this filed
in the [Elasticsearch README
doc|https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x],
but this did not work for squid, since there is no template file present for
squid. The following command returns empty:
{code}
curl -XGET 'http://<eshost>:9200/_template/squid*'
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
