METRON-1629 Update Solr documentation (merrimanr via justinleet) closes apache/metron#1072
Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/6159c6f9 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/6159c6f9 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/6159c6f9 Branch: refs/heads/feature/METRON-1554-pcap-query-panel Commit: 6159c6f997851fabe1b3c21cc0e06bf49c49d9f7 Parents: ed4dfb9 Author: merrimanr <[email protected]> Authored: Fri Jun 22 07:48:27 2018 -0400 Committer: leet <[email protected]> Committed: Fri Jun 22 07:48:27 2018 -0400 ---------------------------------------------------------------------- metron-interface/metron-alerts/README.md | 3 ++- metron-interface/metron-rest/README.md | 1 + metron-platform/metron-indexing/README.md | 14 ++++++++++++++ metron-platform/metron-parsers/README.md | 2 ++ metron-platform/metron-solr/README.md | 26 ++++++++++++++++++++++++++ 5 files changed, 45 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-interface/metron-alerts/README.md ---------------------------------------------------------------------- diff --git a/metron-interface/metron-alerts/README.md b/metron-interface/metron-alerts/README.md index 2d34505..486a885 100644 --- a/metron-interface/metron-alerts/README.md +++ b/metron-interface/metron-alerts/README.md @@ -40,7 +40,8 @@ Sorting has a similar caveat, in that if we are matching on multiple alerts, the Alerts that are contained in a a meta alert are generally excluded from search results, because a user has already grouped them in a meaningful way. ## Prerequisites -* The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies +* The Metron REST application should be up and running +* Elasticsearch or Solr should have some alerts populated by Metron topologies, depending on which real-time store is enabled * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-interface/metron-rest/README.md ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md index 2a6a0e0..44594f7 100644 --- a/metron-interface/metron-rest/README.md +++ b/metron-interface/metron-rest/README.md @@ -22,6 +22,7 @@ This module provides a RESTful API for interacting with Metron. ## Prerequisites * A running Metron cluster +* A running real-time store, either Elasticsearch or Solr depending on which one is enabled * Java 8 installed * Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed * A relational database http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-indexing/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md index 5a35d62..46e511b 100644 --- a/metron-platform/metron-indexing/README.md +++ b/metron-platform/metron-indexing/README.md @@ -77,6 +77,20 @@ Alerts can be grouped, after appropriate searching, into a set of alerts called ### Elasticsearch Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field `alert` of type `nested` for Elasticsearch 2.x installs. This is detailed at [Using Metron with Elasticsearch 2.x](../metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x) +### Solr + +Metron comes with built-in schemas for the default sensors for Solr. When adding a new sensor, it will be necessary to add a new schema defining the output fields appropriately. In addition, these fields are used internally by Metron and also required: + +* `<field name="guid" type="string" indexed="true" stored="true" required="true" multiValued="false" />` +* `<field name="source.type" type="string" indexed="true" stored="true" />` +* `<field name="timestamp" type="timestamp" indexed="true" stored="true" />` +* `<field name="comments" type="string" indexed="true" stored="true" multiValued="true"/>` +* `<field name="metaalerts" type="string" multiValued="true" indexed="true" stored="true"/>` + +The unique key should be set to `guid` by including `<uniqueKey>guid</uniqueKey>` in the schema. + +It is strongly suggested the `fieldTypes` match those in the built-in schemas. + ### Indexing Configuration Examples For a given sensor, the following scenarios would be indicated by the following cases: http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-parsers/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md index 8254baf..d79b9ce 100644 --- a/metron-platform/metron-parsers/README.md +++ b/metron-platform/metron-parsers/README.md @@ -561,6 +561,8 @@ it is necessary to add an additional field to the templates and mapping for exis Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](../../metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) +If Solr is selected as the real-time store, it is also necessary to add additional fields. See the [Solr](../metron-indexing#solr) section in metron-indexing for more details. + ## Kafka Queue The kafka queue associated with your parser is a collection point for all of the data sent to your parser. As such, make sure that the number of partitions in http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-solr/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-solr/README.md b/metron-platform/metron-solr/README.md index 159779c..ca90c73 100644 --- a/metron-platform/metron-solr/README.md +++ b/metron-platform/metron-solr/README.md @@ -92,6 +92,32 @@ Navigate to `$METRON_HOME/bin` and spin up Solr Cloud by running `install_solr.s Elasticsearch and Kibana will have been stopped and you should now have an instance of Solr Cloud up and running at http://localhost:8983/solr/#/~cloud. This manner of starting Solr will also spin up an embedded Zookeeper instance at port 9983. More information can be found [here](https://lucene.apache.org/solr/guide/6_6/getting-started-with-solrcloud.html) +Solr can also be installed using [HDP Search 3](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_solr-search-installation/content/ch_hdp_search_30.html). HDP Search 3 sets the Zookeeper root to +`/solr` so this will need to be added to each url in the comma-separated list in Ambari UI -> Services -> Metron -> Configs -> Index Settings -> Solr Zookeeper Urls. For example, in full dev +this would be `node1:2181/solr`. + +## Enabling Solr + +Elasticsearch is the real-time store used by default in Metron. Solr can be enabled following these steps: + +1. Stop the Metron Indexing component in Ambari. +1. Update Ambari UI -> Services -> Metron -> Configs -> Index Settings -> Solr Zookeeper Urls to match the Solr installation described in the previous section. +1. Change Ambari UI -> Services -> Metron -> Configs -> Indexing -> Index Writer - Random Access -> Random Access Search Engine to `Solr`. +1. Set the `source.type.field` property to `source.type` in the [Global Configuration](../metron-common#global-configuration). +1. Set the `threat.triage.score.field` property to `threat.triage.score` in the [Global Configuration](../metron-common#global-configuration). +1. Start the Metron Indexing component in Ambari. +1. Restart Metron REST and the Alerts UI in Ambari. + +This will automatically create collections for the schemas shipped with Metron: + +* bro +* snort +* yaf +* error (used internally by Metron) +* metaalert (used internall by Metron) + +Any other collections must be created manually before starting the Indexing component. Alerts should be present in the Alerts UI after enabling Solr. + ## Schemas As of now, we have mapped out the Schemas in `src/main/config/schema`.
