Repository: metron Updated Branches: refs/heads/master 7ef929ae4 -> 08f95dc8f
METRON-959 Trivial documentation formatting bugs (JonZeolla) closes apache/metron#591 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/08f95dc8 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/08f95dc8 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/08f95dc8 Branch: refs/heads/master Commit: 08f95dc8f5588dc9d50e204410ca3d09f84f3c8b Parents: 7ef929a Author: JonZeolla <zeo...@gmail.com> Authored: Thu May 25 18:15:41 2017 -0400 Committer: jonzeolla <jonzeo...@apache.org> Committed: Thu May 25 18:15:41 2017 -0400 ---------------------------------------------------------------------- metron-analytics/metron-maas-service/README.md | 6 +++--- metron-platform/metron-indexing/README.md | 4 ++-- metron-platform/metron-management/README.md | 2 +- metron-platform/metron-parsers/README.md | 20 ++++++++++---------- site-book/README.md | 8 ++++---- 5 files changed, 20 insertions(+), 20 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-analytics/metron-maas-service/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-maas-service/README.md b/metron-analytics/metron-maas-service/README.md index 6b1663b..65f1a01 100644 --- a/metron-analytics/metron-maas-service/README.md +++ b/metron-analytics/metron-maas-service/README.md @@ -1,6 +1,6 @@ # Model Management Infrastructure -##Introduction +## Introduction One of the main features envisioned and requested is the ability to augment the threat intelligence and enrichment processes with insights derived from machine learning or statistical models. The challenges with this sort of infrastructure are * Applying the model may be sufficiently computationally/resource intensive that we need to support scaling via load balancing, which will require service discovery and management. @@ -12,7 +12,7 @@ To support a high throughput environment that is manageable, it is evident that * Deployment should happen using Yarn to manage resources * Clients should have new model endpoints pushed to them -##Architecture +## Architecture  @@ -95,7 +95,7 @@ usage: ModelSubmission -zr,--zk_root <arg> Zookeeper Root ``` -##Stellar Integration +## Stellar Integration Two Stellar functions have been added to provide the ability to call out to models deployed via Model as a Service. One aimed at recovering a load balanced endpoint of a deployed model given the name and, optionally, the version. http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-indexing/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md index a536d87..0cf3a66 100644 --- a/metron-platform/metron-indexing/README.md +++ b/metron-platform/metron-indexing/README.md @@ -1,4 +1,4 @@ -#Indexing +# Indexing ## Introduction @@ -26,7 +26,7 @@ and sent to By default, errors during indexing are sent back into the `indexing` kafka queue so that they can be indexed and archived. -##Sensor Indexing Configuration +## Sensor Indexing Configuration The sensor specific configuration is intended to configure the indexing used for a given sensor type (e.g. `snort`). http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-management/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-management/README.md b/metron-platform/metron-management/README.md index 0cfcac5..b920460 100644 --- a/metron-platform/metron-management/README.md +++ b/metron-platform/metron-management/README.md @@ -1,4 +1,4 @@ -#Stellar REPL Management Utilities +# Stellar REPL Management Utilities In order to augment the functionality of the Stellar REPL, a few management functions surrounding the management of the configurations http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-parsers/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md index cc45834..97e8e5f 100644 --- a/metron-platform/metron-parsers/README.md +++ b/metron-platform/metron-parsers/README.md @@ -1,4 +1,4 @@ -#Parsers +# Parsers ## Introduction @@ -32,7 +32,7 @@ topology in kafka. Errors are collected with the context of the error `error` queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an `error` queue. -##Message Format +## Message Format All Metron messages follow a specific format in order to ingest a message. If a message does not conform to this format it will be dropped and put onto an error queue for further examination. The message must be of a JSON format and must have a JSON tag message like so: @@ -70,11 +70,11 @@ So putting it all together a typical Metron message with all 5-tuple fields pres } ``` -##Global Configuration +## Global Configuration See the "[Global Configuration](../metron-common)" section. -##Parser Configuration +## Parser Configuration The configuration for the various parser topologies is defined by JSON documents stored in zookeeper. @@ -103,7 +103,7 @@ transformation which can be done to a message. This transformation can * Add new fields given the values of existing fields of a message * Remove existing fields of a message -###`fieldTransformation` configuration +### `fieldTransformation` configuration The format of a `fieldTransformation` is as follows: * `input` : An array of fields or a single field representing the input. This is optional; if unspecified, then the whole message is passed as input. @@ -201,7 +201,7 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )" Note that the `dc2tz` map is in the parser config, so it is accessible in the functions. -###An Example Configuration for a Sensor +### An Example Configuration for a Sensor Consider the following example configuration for the `yaf` sensor: ``` @@ -225,12 +225,12 @@ Consider the following example configuration for the `yaf` sensor: } ``` -##Parser Adapters +## Parser Adapters Parser adapters are loaded dynamically in each Metron topology. They are defined in the Parser Config (defined above) JSON file in Zookeeper. -###Java Parser Adapters +### Java Parser Adapters Java parser adapters are indended for higher-velocity topologies and are not easily changed or extended. As the adoption of Metron continues we plan on extending our library of Java adapters to process more log formats. As of this moment the Java adapters included with Metron are: * org.apache.metron.parsers.ise.BasicIseParser : Parse ISE messages @@ -238,7 +238,7 @@ Java parser adapters are indended for higher-velocity topologies and are not eas * org.apache.metron.parsers.sourcefire.BasicSourcefireParser : Parse Sourcefire messages * org.apache.metron.parsers.lancope.BasicLancopeParser : Parse Lancope messages -###Grok Parser Adapters +### Grok Parser Adapters Grok parser adapters are designed primarly for someone who is not a Java coder for quickly standing up a parser adapter for lower velocity topologies. Grok relies on Regex for message parsing, which is much slower than purpose-built Java parsers, but is more extensible. Grok parsers are defined via a config file and the topplogy does not need to be recombiled in order to make changes to them. An example of a Grok perser is: * org.apache.metron.parsers.GrokParser @@ -247,7 +247,7 @@ For more information on the Grok project please refer to the following link: https://github.com/thekrakken/java-grok -#Starting the Parser Topology +# Starting the Parser Topology Starting a particular parser topology on a running Metron deployment is as easy as running the `start_parser_topology.sh` script located in http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/site-book/README.md ---------------------------------------------------------------------- diff --git a/site-book/README.md b/site-book/README.md index 6c6b7bd..bf579de 100644 --- a/site-book/README.md +++ b/site-book/README.md @@ -32,22 +32,22 @@ open target/site/index.html ``` -##Key Components: +## Key Components: -###bin/generate-md.sh +### bin/generate-md.sh - Copies all .md files from the code directory tree into the site tree - Performs some transformations on them - Generates the nav tree structure and labels - Happens during the site:pre-site phase of Maven. -###bin/fix-md-dialect.py +### bin/fix-md-dialect.py - Called by 'generate-md.sh' - Does transforms within the text of each file - Converts the Github-MD dialect of markdown into the doxia-markdown dialect -###pom.xml and src/site/site.xml +### pom.xml and src/site/site.xml - [Doxia](https://maven.apache.org/doxia/) boilerplate, tweaked for our specific needs
