GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/175
METRON-243 Ensure templates are installed on all search hosts
### [METRON-243](https://issues.apache.org/jira/browse/METRON-243)
I have a working theory on this one. I will need some additional testing,
but I wanted to get some eyeballs on this to validate my thought process. I
would love it if others were to run this through full AWS deployments and
report their results.
#### Problem
Kibana reports this error when access the Metron Dashboard. Does not occur
on Vagrant deployment. Occurs sometimes on AWS deployments.
```
[unsupported_operation_exception] custom format isn't supported
```
#### Root Cause
The dashboard expects certain data types for the Snort, Bro, and YAF
indices. If the indices were created WITHOUT the index template definitions,
the dashboard will throw this error.
In some cases, the index templates that appear to be properly installed
during deployment, are later dropped when Elasticsearch is restarted by Monit.
This causes Kibana to report this error. Now what might cause the index
templates to go missing? Here are the steps that can cause this to occur.
* The 'elasticsearch' role installs Elasticsearch on each node in the
'search' host group. After installation, Elasticsearch is not yet started.
* In the next step, the 'metron_elasticsearch_templates' role uses
Ansible's 'run_once' functionality. The effect of using 'run_once' is that
Ansible chooses a single node in the 'search' host group; let's call this host
ES1.
* Only on ES1, Elasticsearch is started, the index templates are installed,
and then the index templates are validated. We effectively have a single node
Elasticsearch cluster at this point.
* After all components have been installed, the 'monit-start' role ensures
that all services are stopped across the cluster. At this point the single-node
Elasticsearch cluster, which was left running, is shutdown.
* The 'monit-start' role then determines which services need started and
starts them. At this point all of the Elasticsearch nodes are started which
creates a multi-node Elasticsearch cluster.
* Here is the sticky wicket...
* If ES1 is elected master, then the index template that was previously
configured is propagated to the other nodes in the cluster.
* If ES1 is not elected master, then the index template is forgotten as
the master knows nothing about those index templates.
This explains why it occurs sometimes, but not all the time for AWS
deployments.
This would also explain why the problem never occurs on Vagrant deployments.
#### Fix
This PR ensures that ALL Elasticsearch nodes are started when the index
templates are configured. A side effect is that the index templates are
installed multiple times, once per host in the cluster. This does not cause a
problem necessarily, but is duplicative.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-243
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/175.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #175
----
commit bb1665fd7e8944687e8cfbc400a49c37b12a0c1b
Author: Nick Allen <[email protected]>
Date: 2016-06-23T21:25:48Z
METRON-243 Ensure templates are installed on all search hosts
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
