Repository: metron Updated Branches: refs/heads/master 3fc8c84b9 -> eddbda88e
METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/eddbda88 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/eddbda88 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/eddbda88 Branch: refs/heads/master Commit: eddbda88e4872479f1a78c3000cc6cf56f3567a6 Parents: 3fc8c84 Author: justinleet <[email protected]> Authored: Fri Oct 6 09:33:23 2017 -0400 Committer: leet <[email protected]> Committed: Fri Oct 6 09:33:23 2017 -0400 ---------------------------------------------------------------------- README.md | 6 +++ Upgrading.md | 41 ++++++++++++++++ metron-platform/metron-elasticsearch/README.md | 54 +++++++++++++++++++++ metron-platform/metron-indexing/README.md | 4 ++ metron-platform/metron-parsers/README.md | 6 +++ 5 files changed, 111 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 8599f0f..f3b2765 100644 --- a/README.md +++ b/README.md @@ -118,3 +118,9 @@ Some useful utilities that cross all of these parts of the architecture: * [Model as a Service](metron-analytics/metron-maas-service) : A Yarn application which can deploy machine learning and statistical models onto the cluster along with the associated Stellar functions to be able to call out to them in a scalable manner. * [Data management](metron-platform/metron-data-management) : A set of data management utilities aimed at getting data into HBase in a format which will allow data flowing through metron to be enriched with the results. Contains integrations with threat intelligence feeds exposed via TAXII as well as simple flat file structures. * [Profiler](metron-analytics/metron-profiler) : A feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior. + +# Notes on Adding a New Sensor +In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, +it is necessary to add an additional field to the templates and mapping for existing sensors. + +Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/Upgrading.md ---------------------------------------------------------------------- diff --git a/Upgrading.md b/Upgrading.md index 6fb2486..dd68de0 100644 --- a/Upgrading.md +++ b/Upgrading.md @@ -2,6 +2,47 @@ This document constitutes a per-version listing of changes of configuration which are non-backwards compatible. +## 0.4.1 to 0.4.2 + +### [METRON-1158: Build backend for grouping alerts into meta alerts](https://issues.apache.org/jira/browse/METRON-1158) +In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, +it is necessary to add an additional field to the templates and mapping for existing sensors. + +Two steps must be done for each sensor, but not on each index for each sensor. + +First is to update the Elasticsearch template for each sensor, so any new indices have the field: + +``` +export ELASTICSEARCH="node1" +export SENSOR="bro" +curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true"; -o "${SENSOR}.template" +sed -i '' '2d;$d' ./${SENSOR}.template +sed -i '' '/"properties" : {/ a\ +"alert": { "type": "nested"},' ${SENSOR}.template +curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index"; -d @${SENSOR}.template +``` + +To update existing indexes, update Elasticsearch mappings with the new field for each sensor. Make sure to set the ELASTICSEARCH variable appropriately. + +``` +curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc"; -d ' +{ + "properties" : { + "alert" : { + "type" : "nested" + } + } +} +' +rm ${SENSOR}.template +``` + +For a more detailed description, please see metron-platform/metron-elasticsearch/README.md + +### Description + +In the 0.4.2 release, + ## 0.3.1 to 0.4.0 ### [METRON-671: Refactor existing Ansible deployment to use Ambari MPack](https://issues.apache.org/jira/browse/METRON-671) http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-elasticsearch/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-elasticsearch/README.md b/metron-platform/metron-elasticsearch/README.md new file mode 100644 index 0000000..2d2b139 --- /dev/null +++ b/metron-platform/metron-elasticsearch/README.md @@ -0,0 +1,54 @@ +# Elasticsearch in Metron + +## Introduction + +Elasticsearch can be used as the real-time portion of the datastore resulting from [metron-indexing](../metron-indexing.README.md). + +## Using Metron with Elasticsearch 2.x + +With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information + +Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service's logs. + +Exception seen: +``` +QueryParsingException[[nested] failed to find nested object under path [alert]]; +``` + +There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field. + +Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it's not necessary to do each index individually. The example here appends `index*` to get all indexes for a the provided sensor. + +``` +export ELASTICSEARCH="node1" +export SENSOR="bro" +curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true"; -o "${SENSOR}.template" +sed -i '' '2d;$d' ./${SENSOR}.template +sed -i '' '/"properties" : {/ a\ +"alert": { "type": "nested"},' ${SENSOR}.template +``` + +To manually verify this, you can optionally pretty print it again with: +``` +python -m json.tool bro.template +``` + +We'll want to put the template back into Elasticsearch: +``` +curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index"; -d @${SENSOR}.template +``` + +To update existing indexes, update Elasticsearch mappings with the new field for each sensor. + +``` +curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc"; -d ' +{ + "properties" : { + "alert" : { + "type" : "nested" + } + } +} +' +rm ${SENSOR}.template +``` http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-indexing/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md index e65152c..6f47507 100644 --- a/metron-platform/metron-indexing/README.md +++ b/metron-platform/metron-indexing/README.md @@ -46,6 +46,10 @@ If unspecified, or set to `0`, it defaults to a system-determined duration which parameter `topology.message.timeout.secs`. Ignored if batchSize is `1`, since this disables batching. * `enabled` : Whether the writer is enabled (default `true`). + +### Elasticsearch +Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field `alert` of type `nested` for Elasticsearch 2.x installs. This is detailed at [Using Metron with Elasticsearch 2.x](../metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x) + ### Indexing Configuration Examples For a given sensor, the following scenarios would be indicated by the following cases: http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-parsers/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md index 141e232..01eae71 100644 --- a/metron-platform/metron-parsers/README.md +++ b/metron-platform/metron-parsers/README.md @@ -434,6 +434,12 @@ and pass `--extra_topology_options custom_config.json` to `start_parser_topology Default installed Metron is untuned for production deployment. There are a few knobs to tune to get the most out of your system. +# Notes on Adding a New Sensor +In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, +it is necessary to add an additional field to the templates and mapping for existing sensors. + +Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) + ## Kafka Queue The kafka queue associated with your parser is a collection point for all of the data sent to your parser. As such, make sure that the number of partitions in
