GitHub user nickwallen reopened a pull request: https://github.com/apache/incubator-metron/pull/305
METRON-403 Bro Elasticsearch index item fails when DNS response includes CNAME ### Problem [METRON-403](https://issues.apache.org/jira/browse/METRON-403) When the Bro DNS logs contain FQDNs in the 'answers' field, writing to the Elasticsearch index fails with the following exception. ``` 2016-09-21 09:13:28.360 o.a.m.w.BulkWriterComponent [ERROR] Failing 5 tuples java.lang.Exception: failure in bulk execution: [1]: index [bro_index_2016.09.21.09], type [bro_doc], id [AVdMBXQSDU1LLSTQd0pk], message [MapperParsingException[failed to parse [answers]]; nested: IllegalArgumentException[failed to parse ip [clients.l.google.com], not a valid ip address];] at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:173) ~[stormjar.jar:?] at org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:108) [stormjar.jar:?] at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:98) [stormjar.jar:?] at ``` ### Solution The template for the Bro index is constructed expecting that the 'answers' field is one or more IPv4 addresses. This is not always the case. The field is [described as a vector of strings](https://www.bro.org/sphinx/scripts/base/protocols/dns/main.bro.html#type-DNS::Info) and can include an IPv4, IPv6 or a hostname depending on the type of DNS response message. ``` {"dns": {"uid": "C05DzHkyLOXv22Cgl", ..., "answers": ["daysofyorr.com", "50.28.53.156"], ...} ``` I altered the default Elasticsearch Bro index template to expect a string, rather than simply IPv4. This has no impact to Metron's default dashboard. ### Testing This change was tested by running an end-to-end test on the "Quick Dev" platform. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-403 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #305 ---- commit 0c983b8be80adf7ba63a69bb2f7f3d429c0cb5aa Author: Nick Allen <n...@nickallen.org> Date: 2016-10-11T21:01:27Z METRON-403 Bro Elasticsearch index item fails when DNS response includes CNAME ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---