http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html b/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html index 5cf9775..f3d8dff 100644 --- a/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html +++ b/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid">
http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html b/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html index 499e89b..8a59770 100644 --- a/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html +++ b/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/elasticsearch-mpack/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/elasticsearch-mpack/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/ambari/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/ambari/index.html b/current-book/metron-deployment/packaging/ambari/index.html index bac5758..7acd537 100644 --- a/current-book/metron-deployment/packaging/ambari/index.html +++ b/current-book/metron-deployment/packaging/ambari/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Ambari Management Pack Development</title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Ambari Management Pack Development</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html b/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html index dd97717..ddf0663 100644 --- a/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html +++ b/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/docker/ansible-docker/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html index 8145c8f..4c85d3f 100644 --- a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html +++ b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/ansible-docker/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/ansible-docker/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/docker/deb-docker/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/docker/deb-docker/index.html b/current-book/metron-deployment/packaging/docker/deb-docker/index.html index a83a363..8dd1e3f 100644 --- a/current-book/metron-deployment/packaging/docker/deb-docker/index.html +++ b/current-book/metron-deployment/packaging/docker/deb-docker/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/deb-docker/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/deb-docker/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/docker/rpm-docker/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html index d684c0d..4d0cee3 100644 --- a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html +++ b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/rpm-docker/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/rpm-docker/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-deployment/packaging/packer-build/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/packaging/packer-build/index.html b/current-book/metron-deployment/packaging/packer-build/index.html index fb4b71b..63a85b0 100644 --- a/current-book/metron-deployment/packaging/packer-build/index.html +++ b/current-book/metron-deployment/packaging/packer-build/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/packer-build/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/packer-build/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Build Metron Images</title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Build Metron Images</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-interface/metron-alerts/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-interface/metron-alerts/index.html b/current-book/metron-interface/metron-alerts/index.html index 9459bd1..f4c3d04 100644 --- a/current-book/metron-interface/metron-alerts/index.html +++ b/current-book/metron-interface/metron-alerts/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-alerts/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-alerts/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active "></li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -118,7 +118,8 @@ limitations under the License. <h2><a name="Prerequisites"></a>Prerequisites</h2> <ul> -<li>The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies</li> +<li>The Metron REST application should be up and running</li> +<li>Elasticsearch or Solr should have some alerts populated by Metron topologies, depending on which real-time store is enabled</li> <li>The Management UI should be installed (which includes <a class="externalLink" href="https://expressjs.com/";>Express</a>)</li> <li>The alerts can be populated using Full Dev or any other setup</li> <li>UI is developed using angular4 and uses angular-cli</li> @@ -202,7 +203,10 @@ rest: <h2><a name="Global_Configuration_Properties"></a>Global Configuration Properties</h2> <div class="section"> <h3><a name="source.type.field"></a><tt>source.type.field</tt></h3> -<p>The source type format used. Defaults to <tt>source:type</tt>.</p></div></div> +<p>The source type field name used in the real-time store. Defaults to <tt>source:type</tt>.</p></div> +<div class="section"> +<h3><a name="threat.triage.score.field"></a><tt>threat.triage.score.field</tt></h3> +<p>The threat triage score field name used in the real-time store. Defaults to <tt>threat:triage:score</tt>.</p></div></div> <div class="section"> <h2><a name="Usage"></a>Usage</h2> <p>After configuration is complete, the Management UI can be managed as a service:</p> @@ -237,12 +241,34 @@ npm install <p><b>NOTE</b>: <i>In the development mode ui by default connects to REST at <a class="externalLink" href="http://node1:8082";>http://node1:8082</a> for fetching data. If you wish to change it you can change the REST url at metron/metron-interface/metron-alerts/proxy.conf.json</i></p></div> <div class="section"> <h2><a name="E2E_Tests"></a>E2E Tests</h2> -<p>An expressjs server is available for mocking the elastic search api.</p> +<div class="section"> +<h3><a name="Caveats"></a>Caveats</h3> <ol style="list-style-type: decimal"> <li> -<p>Run e2e webserver :</p> +<p>E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @<a class="externalLink" href="http://node1:8082";>http://node1:8082</a>. It is recommended to shutdown all other Metron services while running the E2E tests including Parsers, Enrichment, Indexing and the Profiler.</p> +</li> +<li> + +<p>E2E tests are run on headless chrome. To see the chrome browser in action, remove the ‘–headless’ parameter of chromeOptions in metron/metron-interface/metron-alerts/protractor.conf.js file</p> +</li> +<li> + +<p>E2E tests delete all the data in HBase table ‘metron_update’ and Elastic search index ‘meta_alerts_index’ for testing against its test data</p> +</li> +<li> + +<p>E2E tests use <a class="externalLink" href="https://github.com/NickTomlin/protractor-flake";>protractor-flake</a> to re-run flaky tests.</p> +</li> +</ol></div> +<div class="section"> +<h3><a name="Steps_to_run"></a>Steps to run</h3> +<ol style="list-style-type: decimal"> + +<li> + +<p>An Express.js server is available for accessing the rest api. Run the e2e webserver:</p> <div> <div> @@ -252,7 +278,7 @@ sh ./scripts/start-server-for-e2e.sh </li> <li> -<p>run e2e test using the following command</p> +<p>Run e2e tests using the following command:</p> <div> <div> @@ -260,12 +286,8 @@ sh ./scripts/start-server-for-e2e.sh npm run e2e </pre></div></div> </li> -<li> - -<p>E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @<a class="externalLink" href="http://node1:8082";>http://node1:8082</a></p> -</li> </ol> -<p><b>NOTE</b>: <i>e2e tests covers all the general workflows and we will extend them as we need</i></p></div> +<p><b>NOTE</b>: <i>e2e tests cover all the general workflows and we will extend them as we need</i></p></div></div> </div> </div> </div> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-interface/metron-config/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-interface/metron-config/index.html b/current-book/metron-interface/metron-config/index.html index 0c4c073..c46f751 100644 --- a/current-book/metron-interface/metron-config/index.html +++ b/current-book/metron-interface/metron-config/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-config/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-config/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Management UI</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Metron Management UI</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-interface/metron-rest/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-interface/metron-rest/index.html b/current-book/metron-interface/metron-rest/index.html index eb378b3..b0f91d7 100644 --- a/current-book/metron-interface/metron-rest/index.html +++ b/current-book/metron-interface/metron-rest/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-rest/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-rest/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron REST</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Metron REST</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -97,6 +97,7 @@ limitations under the License. <ul> <li>A running Metron cluster</li> +<li>A running real-time store, either Elasticsearch or Solr depending on which one is enabled</li> <li>Java 8 installed</li> <li>Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed</li> <li>A relational database</li> @@ -429,6 +430,23 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" </pre></div></div> </div> <div class="section"> +<h2><a name="Pcap_Query"></a>Pcap Query</h2> +<p>The REST application exposes endpoints for querying Pcap data. For more information about filtering options see <a href="../../metron-platform/metron-pcap-backend/index.html#Query_Filter_Utility">Query Filter Utility</a>.</p> +<p>There is an endpoint available that will return Pcap data in <a class="externalLink" href="https://wiki.wireshark.org/PDML";>PDML</a> format. <a class="externalLink" href="https://www.wireshark.org/";>Wireshark</a> must be installed for this feature to work. Installing wireshark in CentOS can be done with <tt>yum -y install wireshark</tt>.</p> +<p>The REST application uses a Java Process object to call out to the <tt>pcap_to_pdml.sh</tt> script. This script is installed at <tt>$METRON_HOME/bin/pcap_to_pdml.sh</tt> by default. Out of the box it is a simple wrapper around the tshark command to transform raw pcap data to PDML. However it can be extended to do additional processing as long as the expected input/output is maintained. REST will supply the script with raw pcap data through standard in and expects PDML data serialized as XML.</p> +<p>Pcap query jobs can be configured for submission to a YARN queue. This setting is exposed as the Spring property <tt>pcap.yarn.queue</tt>. If configured, the REST application will set the <tt>mapreduce.job.queuename</tt> Hadoop property to that value. It is highly recommended that a dedicated YARN queue be created and configured for Pcap queries to prevent a job from consuming too many cluster resources. More information about setting up YARN queues can be found <a class="externalLink" href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setting_up_queues";>here</a>.</p> +<p>Pcap query results are stored in HDFS. The location of query results when run through the REST app is determined by a couple factors. The root of Pcap query results defaults to <tt>/apps/metron/pcap/output</tt> but can be changed with the Spring property <tt>pcap.final.output.path</tt>. Assuming the default Pcap query output directory, the path to a result page will follow this pattern:</p> + +<div> +<div> +<pre class="source">/apps/metron/pcap/output/{username}/MAP_REDUCE/{job id}/page-{page number}.pcap +</pre></div></div> + +<p>Over time Pcap query results will accumulate in HDFS. Currently these results are not cleaned up automatically so cluster administrators should be aware of this and monitor them. It is highly recommended that a process be put in place to periodically delete files and directories under the Pcap query results root.</p> +<p>Users should also be mindful of date ranges used in queries so they don’t produce result sets that are too large. Currently there are no limits enforced on date ranges.</p> +<p>Queries can also be configured on a global level for setting the number of results per page via a Spring property <tt>pcap.page.size</tt>. By default, this value is set to 10 pcaps per page, but you may choose to set this value higher based on observing frequenetly-run query result sizes. This setting works in conjunction with the property for setting finalizer threadpool size when optimizing query performance.</p> +<p>Pcap query jobs have a finalization routine that writes their results out to HDFS in pages. Depending on the size of your pcaps, the number or results typically returned, page sizing (described above), and available CPU cores for running your REST application, your performance can be improved by adjusting the number of files that can be written to HDFS in parallel. To this end, there is a threadpool used for this finalization step that can be configured to use a specified number of threads. This setting is exposed as the Spring property <tt>pcap.finalizer.threadpool.size</tt>. A default value of “1” is used if not specified by the user. Generally speaking, you should see a performance gain when this value is set to anything higher than 1. A sizeable increase in performance can be achieved, especially for larger numbers of files of smaller size, by increasing the number of threads. It should be noted that this property is parsed as a String to allow for more complex parallelism values. In addition to normal integer values, you can specify a multiple of the number of cores. If it’s a string and ends with “C”, then strip the C and treat it as an integral multiple of the number of cores. If it’s a string and does not end with a C, then treat it as a number in string form.</p></div> +<div class="section"> <h2><a name="API"></a>API</h2> <p>Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.</p> <table border="0" class="table table-striped"> @@ -439,7 +457,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" </thead><tbody> <tr class="b"> -<td> <a href="#get-apiv1alertsuiescalate"> <tt>POST /api/v1/alerts/ui/escalate</tt></a></td></tr> +<td> <a href="#POST_apiv1alertsuiescalate"> <tt>POST /api/v1/alerts/ui/escalate</tt></a></td></tr> <tr class="a"> <td> <a href="#GET_apiv1alertsuisettings"> <tt>GET /api/v1/alerts/ui/settings</tt></a></td></tr> <tr class="b"> @@ -491,11 +509,27 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" <tr class="a"> <td> <a href="#GET_apiv1metaalertupdatestatusguidstatus"> <tt>GET /api/v1/metaalert/update/status/{guid}/{status}</tt></a></td></tr> <tr class="b"> +<td> <a href="#POST_apiv1pcapfixed"> <tt>POST /api/v1/pcap/fixed</tt></a></td></tr> +<tr class="a"> +<td> <a href="#POST_apiv1pcapquery"> <tt>POST /api/v1/pcap/query</tt></a></td></tr> +<tr class="b"> +<td> <a href="#GET_apiv1pcap"> <tt>GET /api/v1/pcap</tt></a></td></tr> +<tr class="a"> +<td> <a href="#GET_apiv1pcapjobId"> <tt>GET /api/v1/pcap/{jobId}</tt></a></td></tr> +<tr class="b"> +<td> <a href="#GET_apiv1pcapjobIdpdml"> <tt>GET /api/v1/pcap/{jobId}/pdml</tt></a></td></tr> +<tr class="a"> +<td> <a href="#GET_apiv1pcapjobIdraw"> <tt>GET /api/v1/pcap/{jobId}/raw</tt></a></td></tr> +<tr class="b"> +<td> <a href="#DELETE_apiv1pcapkilljobId"> <tt>DELETE /api/v1/pcap/kill/{jobId}</tt></a></td></tr> +<tr class="a"> +<td> <a href="#GET_apiv1pcapjobIdconfig"> <tt>GET /api/v1/pcap/{jobId}/config</tt></a></td></tr> +<tr class="b"> <td> <a href="#GET_apiv1searchsearch"> <tt>GET /api/v1/search/search</tt></a></td></tr> <tr class="a"> -<td> <a href="#get-apiv1searchsearch"> <tt>POST /api/v1/search/search</tt></a></td></tr> +<td> <a href="#POST_apiv1searchsearch"> <tt>POST /api/v1/search/search</tt></a></td></tr> <tr class="b"> -<td> <a href="#get-apiv1searchgroup"> <tt>POST /api/v1/search/group</tt></a></td></tr> +<td> <a href="#POST_apiv1searchgroup"> <tt>POST /api/v1/search/group</tt></a></td></tr> <tr class="a"> <td> <a href="#GET_apiv1searchfindOne"> <tt>GET /api/v1/search/findOne</tt></a></td></tr> <tr class="b"> @@ -593,7 +627,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" <tr class="b"> <td> <a href="#PATCH_apiv1updatepatch"> <tt>PATCH /api/v1/update/patch</tt></a></td></tr> <tr class="a"> -<td> <a href="#patch-apiv1updatereplace"> <tt>PUT /api/v1/update/replace</tt></a></td></tr> +<td> <a href="#PUT_apiv1updatereplace"> <tt>PUT /api/v1/update/replace</tt></a></td></tr> <tr class="b"> <td> <a href="#GET_apiv1user"> <tt>GET /api/v1/user</tt></a></td></tr> </tbody> @@ -1051,6 +1085,156 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" </li> </ul></div> <div class="section"> +<h3><a name="POST_.2Fapi.2Fv1.2Fpcap.2Ffixed"></a><tt>POST /api/v1/pcap/fixed</tt></h3> +<ul> + +<li>Description: Executes a Fixed Filter Pcap Query.</li> +<li>Input: +<ul> + +<li>fixedPcapRequest - A Fixed Pcap Request which includes fixed filter fields like ip source address and protocol</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns a job status with job ID.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="POST_.2Fapi.2Fv1.2Fpcap.2Fquery"></a><tt>POST /api/v1/pcap/query</tt></h3> +<ul> + +<li>Description: Executes a Query Filter Pcap Query.</li> +<li>Input: +<ul> + +<li>queryPcapRequest - A Query Pcap Request which includes Stellar query field</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns a job status with job ID.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="GET_.2Fapi.2Fv1.2Fpcap"></a><tt>GET /api/v1/pcap</tt></h3> +<ul> + +<li>Description: Gets a list of job statuses for Pcap query jobs that match the requested state.</li> +<li>Input: +<ul> + +<li>state - Job state</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns a list of job statuses for jobs that match the requested state.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="GET_.2Fapi.2Fv1.2Fpcap.2F.7BjobId.7D"></a><tt>GET /api/v1/pcap/{jobId}</tt></h3> +<ul> + +<li>Description: Gets job status for Pcap query job.</li> +<li>Input: +<ul> + +<li>jobId - Job ID of submitted job</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns a job status for the Job ID.</li> +<li>404 - Job is missing.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="GET_.2Fapi.2Fv1.2Fpcap.2F.7BjobId.7D.2Fpdml"></a><tt>GET /api/v1/pcap/{jobId}/pdml</tt></h3> +<ul> + +<li>Description: Gets Pcap Results for a page in PDML format.</li> +<li>Input: +<ul> + +<li>jobId - Job ID of submitted job</li> +<li>page - Page number</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns PDML in json format.</li> +<li>404 - Job or page is missing.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="GET_.2Fapi.2Fv1.2Fpcap.2F.7BjobId.7D.2Fraw"></a><tt>GET /api/v1/pcap/{jobId}/raw</tt></h3> +<ul> + +<li>Description: Download Pcap Results for a page.</li> +<li>Input: +<ul> + +<li>jobId - Job ID of submitted job</li> +<li>page - Page number</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns Pcap as a file download.</li> +<li>404 - Job or page is missing.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_.2Fapi.2Fv1.2Fpcap.2Fkill.2F.7BjobId.7D"></a><tt>DELETE /api/v1/pcap/kill/{jobId}</tt></h3> +<ul> + +<li>Description: Kills running job.</li> +<li>Input: +<ul> + +<li>jobId - Job ID of submitted job</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Kills passed job.</li> +</ul> +</li> +</ul></div> +<div class="section"> +<h3><a name="GET_.2Fapi.2Fv1.2Fpcap.2F.7BjobId.7D.2Fconfig"></a><tt>GET /api/v1/pcap/{jobId}/config</tt></h3> +<ul> + +<li>Description: Gets job configuration for Pcap query job.</li> +<li>Input: +<ul> + +<li>jobId - Job ID of submitted job</li> +</ul> +</li> +<li>Returns: +<ul> + +<li>200 - Returns a map of job properties for the Job ID.</li> +<li>404 - Job is missing.</li> +</ul> +</li> +</ul></div> +<div class="section"> <h3><a name="POST_.2Fapi.2Fv1.2Fsearch.2Fsearch"></a><tt>POST /api/v1/search/search</tt></h3> <ul> @@ -1865,8 +2049,8 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" <li>Returns: <ul> -<li>200 - nothing</li> -<li>404 - document not found</li> +<li>200 - Nothing</li> +<li>404 - Document not found</li> </ul> </li> </ul></div> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/Performance-tuning-guide.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/Performance-tuning-guide.html b/current-book/metron-platform/Performance-tuning-guide.html index 00d2907..b9134de 100644 --- a/current-book/metron-platform/Performance-tuning-guide.html +++ b/current-book/metron-platform/Performance-tuning-guide.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/Performance-tuning-guide.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/Performance-tuning-guide.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Performance Tuning Guide</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Metron Performance Tuning Guide</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li><a href="../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li class="active"><a href="#"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> <li><a href="../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li><a href="../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -781,7 +782,7 @@ enrichments enrichments 43 29754331 297 <div> <div> -<pre class="source">/usr/metron/0.5.0/bin/start_parser_topology.sh \ +<pre class="source">/usr/metron/0.6.0/bin/start_parser_topology.sh \ -e ~metron/.storm/storm-bro.config \ -esc ~/.storm/spout-bro.config \ -k $BROKERLIST \ @@ -966,7 +967,7 @@ export KAFKA_HOME=$HDP_HOME/kafka-broker export STORM_UI=http://node1:8744 export ELASTIC=http://node1:9200 export ZOOKEEPER=node1:2181 -export METRON_VERSION=0.5.0 +export METRON_VERSION=0.6.0 export METRON_HOME=/usr/metron/${METRON_VERSION} </pre></div></div> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/index.html b/current-book/metron-platform/index.html index 7819640..9fa8d70 100644 --- a/current-book/metron-platform/index.html +++ b/current-book/metron-platform/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Current Build</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Current Build</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li class="active"><a href="#"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> <li><a href="../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li><a href="../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -104,7 +105,7 @@ limitations under the License. --> <h1>Current Build</h1> <p><a name="Current_Build"></a></p> -<p>The latest build of metron-platform is 0.5.0.</p> +<p>The latest build of metron-platform is 0.6.0.</p> <p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p> <p><a name="metron-platform"></a></p> <h1>metron-platform</h1> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-common/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-common/index.html b/current-book/metron-platform/metron-common/index.html index 8d0dc0e..caa5a3b 100644 --- a/current-book/metron-platform/metron-common/index.html +++ b/current-book/metron-platform/metron-common/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-common/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-common/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Contents</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Contents</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li class="active"><a href="#"><span class="none"></span>Common</a></li> <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -231,6 +232,16 @@ limitations under the License. <td> String </td> <td> <tt>profiler_period_units</tt> </td></tr> <tr class="a"> +<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.writer.batchSize"><tt>profiler.writer.batchSize</tt></a> </td> +<td> Profiler </td> +<td> Integer </td> +<td> N/A </td></tr> +<tr class="b"> +<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.writer.batchTimeout"><tt>profiler.writer.batchTimeout</tt></a> </td> +<td> Profiler </td> +<td> Integer </td> +<td> N/A </td></tr> +<tr class="a"> <td> <a href="../metron-indexing/index.html#update.hbase.table"><tt>update.hbase.table</tt></a> </td> <td> REST/Indexing </td> <td> String </td> @@ -246,10 +257,30 @@ limitations under the License. <td> String </td> <td> <tt>geo_hdfs_file</tt> </td></tr> <tr class="b"> +<td> <a href="../metron-enrichment/index.html#enrichment.writer.batchSize"><tt>enrichment.writer.batchSize</tt></a> </td> +<td> Enrichment </td> +<td> Integer </td> +<td> N/A </td></tr> +<tr class="a"> +<td> <a href="../metron-enrichment/index.html#enrichment.writer.batchTimeout"><tt>enrichment.writer.batchTimeout</tt></a> </td> +<td> Enrichment </td> +<td> Integer </td> +<td> N/A </td></tr> +<tr class="b"> +<td> <a href="../metron-enrichment/index.html#geo.hdfs.file"><tt>geo.hdfs.file</tt></a> </td> +<td> Enrichment </td> +<td> String </td> +<td> <tt>geo_hdfs_file</tt> </td></tr> +<tr class="a"> <td> <a href="../../metron-interface/metron-alerts/index.html#source.type.field"><tt>source.type.field</tt></a> </td> <td> UI </td> <td> String </td> -<td> N/A </td></tr> +<td> <tt>source_type_field</tt> </td></tr> +<tr class="b"> +<td> <a href="../../metron-interface/metron-alerts/index.html#threat.triage.score.field"><tt>threat.triage.score.field</tt></a> </td> +<td> UI </td> +<td> String </td> +<td> <tt>threat_triage_score_field</tt> </td></tr> </tbody> </table> <div class="section"> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-data-management/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-data-management/index.html b/current-book/metron-platform/metron-data-management/index.html index dea600c..610c5c6 100644 --- a/current-book/metron-platform/metron-data-management/index.html +++ b/current-book/metron-platform/metron-data-management/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-data-management/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-data-management/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Resource Data Management</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Resource Data Management</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li class="active"><a href="#"><span class="none"></span>Data-management</a></li> <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-elasticsearch/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-elasticsearch/index.html b/current-book/metron-platform/metron-elasticsearch/index.html index cc360b1..bf3c630 100644 --- a/current-book/metron-platform/metron-elasticsearch/index.html +++ b/current-book/metron-platform/metron-elasticsearch/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-elasticsearch/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-elasticsearch/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Elasticsearch in Metron</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Elasticsearch in Metron</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li class="active"><a href="#"><span class="none"></span>Elasticsearch</a></li> <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -405,13 +406,13 @@ limitations under the License. </ul></div></div></div></div> <div class="section"> <h2><a name="Using_Metron_with_Elasticsearch_5.6.2"></a>Using Metron with Elasticsearch 5.6.2</h2> -<p>There is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields";>Ignoring Unmapped Fields</a> for more information</p> +<p>There is a requirement that all sensors templates have a nested <tt>metron_alert</tt> field defined. This field is a dummy field. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields";>Ignoring Unmapped Fields</a> for more information</p> <p>Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service’s logs.</p> <p>Exception seen:</p> <div> <div> -<pre class="source">QueryParsingException[[nested] failed to find nested object under path [alert]]; +<pre class="source">QueryParsingException[[nested] failed to find nested object under path [metron_alert]]; </pre></div></div> <p>There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.</p> @@ -424,7 +425,7 @@ export SENSOR="bro" curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template" sed -i '' '2d;$d' ./${SENSOR}.template sed -i '' '/"properties" : {/ a\ -"alert": { "type": "nested"},' ${SENSOR}.template +"metron_alert": { "type": "nested"},' ${SENSOR}.template </pre></div></div> <p>To manually verify this, you can optionally pretty print it again with:</p> @@ -448,7 +449,7 @@ sed -i '' '/"properties" : {/ a\ <pre class="source">curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d ' { "properties" : { - "alert" : { + "metron_alert" : { "type" : "nested" } } http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-enrichment/Performance.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-enrichment/Performance.html b/current-book/metron-platform/metron-enrichment/Performance.html index 136d939..0857ee4 100644 --- a/current-book/metron-platform/metron-enrichment/Performance.html +++ b/current-book/metron-platform/metron-enrichment/Performance.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/Performance.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/Performance.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Enrichment Performance</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Enrichment Performance</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,7 +55,6 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> @@ -65,9 +64,11 @@ </ul> </li> <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-enrichment/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-enrichment/index.html b/current-book/metron-platform/metron-enrichment/index.html index de1e4fa..e750946 100644 --- a/current-book/metron-platform/metron-enrichment/index.html +++ b/current-book/metron-platform/metron-enrichment/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Enrichment</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Enrichment</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,7 +55,6 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> @@ -65,9 +64,11 @@ </ul> </li> <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -162,14 +163,22 @@ limitations under the License. <p>There are two types of configurations at the moment, <tt>global</tt> and <tt>sensor</tt> specific.</p></div> <div class="section"> <h2><a name="Global_Configuration"></a>Global Configuration</h2> -<p>There are a few enrichments which have independent configurations, such as from the global config.</p> +<p>There are a few enrichments which have independent configurations, such as from the global config. You can also configure the enrichment topology’s writer batching settings.</p> <p>Also, see the “<a href="../metron-common/index.html">Global Configuration</a>” section for more discussion of the global config.</p> <div class="section"> <h3><a name="GeoIP"></a>GeoIP</h3> <p>Metron supports enrichment of IP information using <a class="externalLink" href="https://dev.maxmind.com/geoip/geoip2/geolite2/";>GeoLite2</a>. The location of the file is managed in the global config.</p> <div class="section"> <h4><a name="geo.hdfs.file"></a><tt>geo.hdfs.file</tt></h4> -<p>The location on HDFS of the GeoLite2 database file to use for GeoIP lookups. This file will be localized on the storm supervisors running the topology and used from there. This is lazy, so if this property changes in a running topology, the file will be localized from HDFS upon first time the file is used via the geo enrichment.</p></div></div></div> +<p>The location on HDFS of the GeoLite2 database file to use for GeoIP lookups. This file will be localized on the storm supervisors running the topology and used from there. This is lazy, so if this property changes in a running topology, the file will be localized from HDFS upon first time the file is used via the geo enrichment.</p></div></div> +<div class="section"> +<h3><a name="Writer_Batching"></a>Writer Batching</h3> +<div class="section"> +<h4><a name="enrichment.writer.batchSize"></a><tt>enrichment.writer.batchSize</tt></h4> +<p>The size of the batch that is written to Kafka at once. Defaults to <tt>15</tt> (size of 1 disables batching).</p></div> +<div class="section"> +<h4><a name="enrichment.writer.batchTimeout"></a><tt>enrichment.writer.batchTimeout</tt></h4> +<p>The timeout after which a batch will be flushed even if batchSize has not been met. Optional. If unspecified, or set to <tt>0</tt>, it defaults to a system-determined duration which is a fraction of the Storm parameter <tt>topology.message.timeout.secs</tt>. Ignored if batchSize is <tt>1</tt>, since this disables batching.</p></div></div></div> <div class="section"> <h2><a name="Sensor_Enrichment_Configuration"></a>Sensor Enrichment Configuration</h2> <p>The sensor specific configuration is intended to configure the individual enrichments and threat intelligence enrichments for a given sensor type (e.g. <tt>snort</tt>).</p> http://git-wip-us.apache.org/repos/asf/metron/blob/0bea5bdb/current-book/metron-platform/metron-indexing/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-indexing/index.html b/current-book/metron-platform/metron-indexing/index.html index 6d4b8d6..65c0bad 100644 --- a/current-book/metron-platform/metron-indexing/index.html +++ b/current-book/metron-platform/metron-indexing/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-indexing/index.md at 2018-06-07 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-indexing/index.md at 2018-09-12 | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta name="Date-Revision-yyyymmdd" content="20180912" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Indexing</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> @@ -32,8 +32,8 @@ <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> <li class="active ">Indexing</li> - <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> - <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-12</li> + <li id="projectVersion" class="pull-right">Version: 0.6.0</li> </ul> </div> <div class="row-fluid"> @@ -55,15 +55,16 @@ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> <ul class="nav nav-list"> <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> - <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> <li class="active"><a href="#"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> </ul> </li> @@ -139,15 +140,38 @@ limitations under the License. <li><tt>hdfs</tt></li> <li><tt>solr</tt></li> </ul> -<p>Depending on how you start the indexing topology, it will have either elasticsearch or solr and hdfs writers running.</p> -<p>The configuration for an individual writer-specific configuration is a JSON map with the following fields:</p> -<ul> +<p>Depending on how you start the indexing topology, it will have either Elasticsearch or Solr and HDFS writers running.</p> +<table border="0" class="table table-striped"> +<thead> -<li><tt>index</tt> : The name of the index to write to (defaulted to the name of the sensor).</li> -<li><tt>batchSize</tt> : The size of the batch that is written to the indices at once. Defaults to <tt>1</tt> (no batching).</li> -<li><tt>batchTimeout</tt> : The timeout after which a batch will be flushed even if batchSize has not been met. Optional. If unspecified, or set to <tt>0</tt>, it defaults to a system-determined duration which is a fraction of the Storm parameter <tt>topology.message.timeout.secs</tt>. Ignored if batchSize is <tt>1</tt>, since this disables batching.</li> -<li><tt>enabled</tt> : Whether the writer is enabled (default <tt>true</tt>).</li> -</ul> +<tr class="a"> +<th> Property </th> +<th> Description </th> +<th> Default Value </th></tr> +</thead><tbody> + +<tr class="b"> +<td> <tt>index</tt> </td> +<td> The name of the index to write to. </td> +<td> Defaults to the name of the sensor. </td></tr> +<tr class="a"> +<td> <tt>batchSize</tt> </td> +<td> The size of the batch that is written to the indices at once. </td> +<td> Defaults to <tt>1</tt>; no batching. </td></tr> +<tr class="b"> +<td> <tt>batchTimeout</tt> </td> +<td> The timeout after which a batch will be flushed even if <tt>batchSize</tt> has not been met. </td> +<td> Defaults to a duration which is a fraction of the Storm parameter <tt>topology.message.timeout.secs</tt>, if left undefined or set to 0. Ignored if batchSize is <tt>1</tt>, since this disables batching.</td></tr> +<tr class="a"> +<td> <tt>enabled</tt> </td> +<td> A boolean indicating whether the writer is enabled. </td> +<td> Defaults to <tt>true</tt> </td></tr> +<tr class="b"> +<td> <tt>fieldNameConverter</tt> </td> +<td> Defines how field names are transformed before being written to the index. Only applicable to <tt>elasticsearch</tt>. </td> +<td> Defaults to <tt>DEDOT</tt>. Acceptable values are <tt>DEDOT</tt> that replaces all ‘.’ with ‘:’ or <tt>NOOP</tt> that does not change the field names . </td></tr> +</tbody> +</table> <div class="section"> <h3><a name="Meta_Alerts"></a>Meta Alerts</h3> <p>Alerts can be grouped, after appropriate searching, into a set of alerts called a meta alert. A meta alert is useful for maintaining the context of searching and grouping during further investigations. Standard searches can return meta alerts, but grouping and other aggregation or sorting requests will not, because there’s not a clear way to aggregate in many cases if there are multiple alerts contained in the meta alert. All meta alerts will have the source type of metaalert, regardless of the contained alert’s origins.</p></div> @@ -155,6 +179,19 @@ limitations under the License. <h3><a name="Elasticsearch"></a>Elasticsearch</h3> <p>Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field <tt>alert</tt> of type <tt>nested</tt> for Elasticsearch 2.x installs. This is detailed at <a href="../metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using Metron with Elasticsearch 2.x</a></p></div> <div class="section"> +<h3><a name="Solr"></a>Solr</h3> +<p>Metron comes with built-in schemas for the default sensors for Solr. When adding a new sensor, it will be necessary to add a new schema defining the output fields appropriately. In addition, these fields are used internally by Metron and also required:</p> +<ul> + +<li><tt><field name="guid" type="string" indexed="true" stored="true" required="true" multiValued="false" /></tt></li> +<li><tt><field name="source.type" type="string" indexed="true" stored="true" /></tt></li> +<li><tt><field name="timestamp" type="timestamp" indexed="true" stored="true" /></tt></li> +<li><tt><field name="comments" type="string" indexed="true" stored="true" multiValued="true"/></tt></li> +<li><tt><field name="metaalerts" type="string" multiValued="true" indexed="true" stored="true"/></tt></li> +</ul> +<p>The unique key should be set to <tt>guid</tt> by including <tt><uniqueKey>guid</uniqueKey></tt> in the schema.</p> +<p>It is strongly suggested the <tt>fieldTypes</tt> match those in the built-in schemas.</p></div> +<div class="section"> <h3><a name="Indexing_Configuration_Examples"></a>Indexing Configuration Examples</h3> <p>For a given sensor, the following scenarios would be indicated by the following cases:</p> <div class="section"> @@ -294,7 +331,7 @@ limitations under the License. <p>The HBase column family to use for message updates.</p></div></div> <div class="section"> <h3><a name="The_MetaAlertDao"></a>The <tt>MetaAlertDao</tt></h3> -<p>The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions like searches, as if meta alerts were normal alerts. <tt>org.apache.metron.indexing.dao.MetaAlertDao</tt> extends <tt>IndexDao</tt> and enables several features:</p> +<p>The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions like searches, as if meta alerts were normal alerts. <tt>org.apache.metron.indexing.dao.metaalert.MetaAlertDao</tt> extends <tt>IndexDao</tt> and enables several features:</p> <ul> <li>the ability to get all meta alerts associated with an alert</li>
