[
https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-1369:
----------------------------------
Assignee: Nick Allen
> squid events are not seen in Alerts UI
> --------------------------------------
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
> Issue Type: Bug
> Reporter: Anand Subramanian
> Assignee: Nick Allen
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki
> instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to
> see them in the alerts UI. On further checking, I found out that this is due
> to the missing nested "alert" field in the index mappings--which is required
> for the Alerts UI to pick up the index.
> This was confirmed in the rest logs as well.
> {code}
> 17/12/18 10:39:40 ERROR dao.ElasticsearchRequestSubmitter: Shard search
> failure [1/1]; reason=IllegalStateException: [nested] failed to find nested
> object under path [alert], index=squid_index_2017.12.18.10, shard=0,
> status=BAD_REQUEST, nodeId=NOjEpjy0RxCV1ltcqp4Erw
> [squid_index_2017.12.18.10/a7-5TaroSRCHMrJnCWkYww] QueryShardException[failed
> to create query: {
> "constant_score" : {
> "filter" : {
> "bool" : {
> "must" : [
> {
> "bool" : {
> "should" : [
> {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 10000,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> {
> "nested" : {
> "query" : {
> "query_string" : {
> "query" : "*",
> "fields" : [ ],
> "use_dis_max" : true,
> "tie_breaker" : 0.0,
> "default_operator" : "or",
> "auto_generate_phrase_queries" : false,
> "max_determinized_states" : 10000,
> "enable_position_increments" : true,
> "fuzziness" : "AUTO",
> "fuzzy_prefix_length" : 0,
> "fuzzy_max_expansions" : 50,
> "phrase_slop" : 0,
> "escape" : false,
> "split_on_whitespace" : true,
> "boost" : 1.0
> }
> },
> "path" : "alert",
> "ignore_unmapped" : false,
> "score_mode" : "none",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> },
> {
> "bool" : {
> "should" : [
> {
> "term" : {
> "status" : {
> "value" : "active",
> "boost" : 1.0
> }
> }
> },
> {
> "bool" : {
> "must_not" : [
> {
> "exists" : {
> "field" : "status",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "must_not" : [
> {
> "exists" : {
> "field" : "metaalerts",
> "boost" : 1.0
> }
> }
> ],
> "disable_coord" : false,
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> },
> "boost" : 1.0
> }
> }]; nested: IllegalStateException[[nested] failed to find nested object under
> path [alert]];
> at
> org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
> at
> org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
> at
> org.elasticsearch.search.SearchService.parseSource(SearchService.java:617)
> at
> org.elasticsearch.search.SearchService.createContext(SearchService.java:485)
> at
> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461)
> at
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
> at
> org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340)
> at
> org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:337)
> at
> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
> at
> org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1533)
> at
> org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: [nested] failed to find nested
> object under path [alert]
> at
> org.elasticsearch.index.query.NestedQueryBuilder.doToQuery(NestedQueryBuilder.java:281)
> at
> org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at
> org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
> at
> org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:420)
> at
> org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at
> org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
> at
> org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418)
> at
> org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at
> org.elasticsearch.index.query.AbstractQueryBuilder.toFilter(AbstractQueryBuilder.java:118)
> at
> org.elasticsearch.index.query.ConstantScoreQueryBuilder.doToQuery(ConstantScoreQueryBuilder.java:146)
> at
> org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
> at
> org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313)
> at
> org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325)
> ... 14 more
> {code}
> Now, I was trying to find a way to add this field to the mappings, so I can
> get the squid alerts listed. We have a documented process for including this
> filed in the [Elasticsearch README
> doc|https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x],
> but this did not work for squid, since there is no template file present for
> squid. The following command returns empty:
> {code}
> curl -XGET 'http://<eshost>:9200/_template/squid*'
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
