GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/305
METRON-403 Bro Elasticsearch index item fails when DNS response includes
CNAME
### Problem
When the Bro DNS logs contain FQDNs in the 'answers' field, writing to the
Elasticsearch index fails with the following exception.
```
2016-09-21 09:13:28.360 o.a.m.w.BulkWriterComponent [ERROR] Failing 5 tuples
java.lang.Exception: failure in bulk execution:
[1]: index [bro_index_2016.09.21.09], type [bro_doc], id
[AVdMBXQSDU1LLSTQd0pk], message [MapperParsingException[failed to parse
[answers]]; nested: IllegalArgumentException[failed to parse ip
[clients.l.google.com], not a valid ip address];]
at
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:173)
~[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:108)
[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:98)
[stormjar.jar:?]
at
```
### Solution
The template for the Bro index is constructed expecting that the 'answers'
field is one or more IPv4 addresses. This is not always the case. The field is
[described as a vector of
strings](https://www.bro.org/sphinx/scripts/base/protocols/dns/main.bro.html#type-DNS::Info)
and can include an IPv4, IPv6 or a hostname depending on the type of DNS
response message.
```
{"dns": {"uid": "C05DzHkyLOXv22Cgl", ..., "answers": ["daysofyorr.com",
"50.28.53.156"], ...}
```
I altered the default Elasticsearch Bro index template to expect a string,
rather than simply IPv4. This has no impact to Metron's default dashboard.
### Testing
This change was tested by running an end-to-end test on the "Quick Dev"
platform.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-403
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/305.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #305
----
commit 0c983b8be80adf7ba63a69bb2f7f3d429c0cb5aa
Author: Nick Allen <[email protected]>
Date: 2016-10-11T21:01:27Z
METRON-403 Bro Elasticsearch index item fails when DNS response includes
CNAME
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
