Repository: metron Updated Branches: refs/heads/master 830e5ccf4 -> b47c9a572
METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b47c9a57 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b47c9a57 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b47c9a57 Branch: refs/heads/master Commit: b47c9a57260faa006173980a765cd565af7f12dd Parents: 830e5cc Author: cstella <[email protected]> Authored: Mon Oct 9 09:39:04 2017 -0400 Committer: cstella <[email protected]> Committed: Mon Oct 9 09:39:04 2017 -0400 ---------------------------------------------------------------------- metron-platform/metron-common/README.md | 27 ++++++++++++++++++- metron-platform/metron-elasticsearch/README.md | 29 +++++++++++++++++++++ metron-platform/metron-enrichment/README.md | 21 ++++++++++++++- metron-platform/metron-indexing/README.md | 13 +++++++++ metron-platform/metron-parsers/README.md | 17 +++++++++++- 5 files changed, 104 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-common/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md index 54738f8..5f9fec6 100644 --- a/metron-platform/metron-common/README.md +++ b/metron-platform/metron-common/README.md @@ -47,7 +47,7 @@ This configuration is stored in zookeeper, but looks something like "es.ip": "node1", "es.port": "9300", "es.date.format": "yyyy.MM.dd.HH", - "parser.error.topic": "indexing" + "parser.error.topic": "indexing", "fieldValidations" : [ { "input" : [ "ip_src_addr", "ip_dst_addr" ], @@ -60,6 +60,31 @@ This configuration is stored in zookeeper, but looks something like } ``` +Various parts of our stack uses the global config are documented throughout the Metron documentation, +but a convenient index is provided here: + +| Property Name | Subsystem | Type | Ambari Property | +|---------------------------------------------------------------------------------------------------------------------|---------------|------------|----------------------------| +| [`es.clustername`](../metron-elasticsearch#esclustername) | Indexing | String | `es_cluster_name` | +| [`es.ip`](../metron-elasticsearch#esip) | Indexing | String | `es_hosts` | +| [`es.port`](../metron-elasticsearch#esport) | Indexing | String | `es_port` | +| [`es.date.format`](../metron-elasticsearch#esdateformat) | Indexing | String | `es_date_format` | +| [`fieldValidations`](#validation-framework) | Parsing | Object | N/A | +| [`parser.error.topic`](../metron-parsers#parsererrortopic) | Parsing | String | N/A | +| [`stellar.function.paths`](../../metron-stellar/stellar-common#stellarfunctionpaths) | Stellar | CSV String | N/A | +| [`stellar.function.resolver.includes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A | +| [`stellar.function.resolver.excludes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A | +| [`profiler.period.duration`](../../metron-analytics/metron-profiler#profilerperiodduration) | Profiler | Integer | `profiler_period_duration` | +| [`profiler.period.duration.units`](../../metron-analytics/metron-profiler#profilerperioddurationunits) | Profiler | String | `profiler_period_units` | +| [`update.hbase.table`](../metron-indexing#updatehbasetable) | REST/Indexing | String | `update_hbase_table` | +| [`update.hbase.cf`](../metron-indexing#updatehbasecf) | REST/Indexing | String | `update_hbase_cf` | +| [`geo.hdfs.file`](../metron-enrichment#geohdfsfile) | Enrichment | String | `geo_hdfs_file` | + +## Note Configs in Ambari +If a field is managed via ambari, you should change the field via +ambari. Otherwise, upon service restarts, you may find your update +overwritten. + # Validation Framework Inside of the global configuration, there is a validation framework in http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-elasticsearch/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-elasticsearch/README.md b/metron-platform/metron-elasticsearch/README.md index 2d2b139..1e39691 100644 --- a/metron-platform/metron-elasticsearch/README.md +++ b/metron-platform/metron-elasticsearch/README.md @@ -4,6 +4,35 @@ Elasticsearch can be used as the real-time portion of the datastore resulting from [metron-indexing](../metron-indexing.README.md). +## Properties + +### `es.clustername` + +The name of the elasticsearch Cluster. See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name) + +### `es.ip` + +Specifies the nodes in the elasticsearch cluster to use for writing. +The format is one of the following: +* A hostname or IP address with a port (e.g. `hostname1:1234`), in which case `es.port` is ignored. +* A hostname or IP address without a port (e.g. `hostname1`), in which case `es.port` is used. +* A string containing a CSV of hostnames without ports (e.g. `hostname1,hostname2,hostname3`) without spaces between. `es.port` is assumed to be the port for each host. +* A string containing a CSV of hostnames with ports (e.g. `hostname1:1234,hostname2:1234,hostname3:1234`) without spaces between. `es.port` is ignored. +* A list of hostnames with ports (e.g. `[ "hostname1:1234", "hostname2:1234"]`). Note, `es.port` is NOT used in this construction. + +### `es.port` + +The port for the elasticsearch hosts. This will be used in accordance with the discussion of `es.ip`. + +### `es.date.format` + +The date format to use when constructing the indices. For every message, the date format will be applied +to the current time and that will become the last part of the index name where the message is written to. + +For instance, an `es.date.format` of `yyyy.MM.dd.HH` would have the consequence that the indices would +roll hourly, whereas an `es.date.format` of `yyyy.MM.dd` would have the consequence that the indices would +roll daily. + ## Using Metron with Elasticsearch 2.x With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-enrichment/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md index 10f2cd4..70bf832 100644 --- a/metron-platform/metron-enrichment/README.md +++ b/metron-platform/metron-enrichment/README.md @@ -25,9 +25,27 @@ defined by JSON documents stored in zookeeper. There are two types of configurations at the moment, `global` and `sensor` specific. + ## Global Configuration -See the "[Global Configuration](../metron-common)" section. +There are a few enrichments which have independent configurations, such +as from the global config. + +Also, see the "[Global Configuration](../metron-common)" section for +more discussion of the global config. + +### GeoIP +Metron supports enrichment of IP information using +[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The +location of the file is managed in the global config. + +#### `geo.hdfs.file` + +The location on HDFS of the GeoLite2 database file to use for GeoIP +lookups. This file will be localized on the storm supervisors running +the topology and used from there. This is lazy, so if this property +changes in a running topology, the file will be localized from HDFS upon first +time the file is used via the geo enrichment. ## Sensor Enrichment Configuration @@ -269,6 +287,7 @@ An example configuration for the YAF sensor is as follows: ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10. + # Example Enrichment via Stellar Let's walk through doing a simple enrichment using Stellar on your cluster using the Squid topology. http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-indexing/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md index 6f47507..6dbcb98 100644 --- a/metron-platform/metron-indexing/README.md +++ b/metron-platform/metron-indexing/README.md @@ -150,6 +150,19 @@ in parallel. This enables a flexible strategy for specifying your backing store For instance, currently the REST API supports the update functionality and may be configured with a list of IndexDao implementations to use to support the updates. +### The `HBaseDao` + +Updates will be written to HBase. The key structure is the GUID and +for each new version, a new column is created with value as the message. + +The HBase table and column family are configured via fields in the global configuration. + +#### `update.hbase.table` +The HBase table to use for message updates. + +#### `update.hbase.cf` +The HBase column family to use for message updates. + ### The `MetaAlertDao` The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-parsers/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md index 01eae71..75559c0 100644 --- a/metron-platform/metron-parsers/README.md +++ b/metron-platform/metron-parsers/README.md @@ -76,7 +76,22 @@ So putting it all together a typical Metron message with all 5-tuple fields pres ## Global Configuration -See the "[Global Configuration](../metron-common)" section. +There are a few properties which can be managed in the global configuration that have pertinence to +parsers and parsing in general. + +### `parser.error.topic` + +The topic where messages which were unable to be parsed due to error are sent. +Error messages will be indexed under a sensor type of `error` and the messages will have +the following fields: +* `sensor.type`: `error` +* `failed_sensor_type` : The sensor type of the message which wasn't able to be parsed +* `error_type` : The error type, in this case `parser`. +* `stack` : The stack trace of the error +* `hostname` : The hostname of the node where the error happened +* `raw_message` : The raw message in string form +* `raw_message_bytes` : The raw message bytes +* `error_hash` : A hash of the error message ## Parser Configuration
