http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/ansible/roles/opentaxii/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/ansible/roles/opentaxii/index.html b/site/current-book/metron-deployment/ansible/roles/opentaxii/index.html new file mode 100644 index 0000000..c3c8fdd --- /dev/null +++ b/site/current-book/metron-deployment/ansible/roles/opentaxii/index.html @@ -0,0 +1,301 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/opentaxii/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron РOpenTAXII</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">OpenTAXII</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-down"></span>Ansible</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/index.html" title="Roles"><span class="icon-chevron-down"></span>Roles</a> + <ul class="nav nav-list"> + <li class="active"><a href="#"><span class="none"></span>Opentaxii</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/pcap_replay/index.html" title="Pcap_replay"><span class="none"></span>Pcap_replay</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-stubs/index.html" title="Sensor-stubs"><span class="none"></span>Sensor-stubs</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-test-mode/index.html" title="Sensor-test-mode"><span class="none"></span>Sensor-test-mode</a></li> + </ul> +</li> + </ul> +</li> + <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>OpenTAXII</h1> +<p><a name="OpenTAXII"></a></p> +<p>Installs <a class="externalLink" href="https://github.com/EclecticIQ/OpenTAXII";>OpenTAXII</a> as a deamon that can be launched via a SysV service script. The complementary client implementation, <a class="externalLink" href="https://github.com/EclecticIQ/cabby";>Cabby</a> is also installed.</p> +<p>OpenTAXII is a robust Python implementation of TAXII Services that delivers a rich feature set and friendly pythonic API. <a class="externalLink" href="https://stixproject.github.io/";>TAXII</a> (Trusted Automated eXchange of Indicator Information) is a collection of specifications defining a set of services and message exchanges used for sharing cyber threat intelligence information between parties.</p> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<p>After deployment completes the OpenTAXII service is installed and running. A set of <a class="externalLink" href="http://hailataxii.com/";>Hail a TAXII</a> threat intel collections have been defined and configured. Use the <tt>status</tt> option to view the collections that have been defined.</p> + +<div> +<div> +<pre class="source">$ service opentaxii status +Checking opentaxii... Running +guest.phishtank_com 0 +guest.Abuse_ch 0 +guest.CyberCrime_Tracker 0 +guest.EmergingThreats_rules 0 +guest.Lehigh_edu 0 +guest.MalwareDomainList_Hostlist 0 +guest.blutmagie_de_torExits 0 +guest.dataForLast_7daysOnly 0 +guest.dshield_BlockList 0 +</pre></div></div> + +<p>Notice that each collections contain zero records. None of the data is automatically synced during deployment. To sync the data manually use the <tt>sync</tt> option as defined below. The following example does not provide a begin and end time so the data will be fetched for the current day only.</p> + +<div> +<div> +<pre class="source"># service opentaxii sync guest.blutmagie_de_torExits +2016-04-21 20:34:42,511 INFO: Starting new HTTP connection (1): localhost +2016-04-21 20:34:42,540 INFO: Response received for Inbox_Message from http://localhost:9000/services/inbox +2016-04-21 20:34:42,542 INFO: Sending Inbox_Message to http://localhost:9000/services/inbox +... +2016-04-21 20:34:42,719 INFO: Response received for Poll_Request from http://localhost:9000/services/poll +2016-04-21 20:34:42,719 INFO: Content blocks count: 1618, is partial: False +</pre></div></div> + +<p>The OpenTAXII service now contains 1,618 threat intel records indicating Tor Exit nodes.</p> + +<div> +<div> +<pre class="source">[root@source ~]# service opentaxii status +Checking opentaxii... Running +guest.phishtank_com 0 +guest.Abuse_ch 0 +guest.CyberCrime_Tracker 0 +guest.EmergingThreats_rules 0 +guest.Lehigh_edu 0 +guest.MalwareDomainList_Hostlist 0 +guest.blutmagie_de_torExits 1618 +guest.dataForLast_7daysOnly 0 +guest.dshield_BlockList 0 +</pre></div></div> +</div> +<div class="section"> +<h2><a name="Usage"></a>Usage</h2> +<p>A standard SysV script has been installed to manage OpenTAXII. The following functions are available.</p> +<p><tt>start</tt> <tt>stop</tt> <tt>restart</tt> the OpenTAXII service</p> +<p><tt>status</tt> of the OpenTAXII service. The command displays the collections that have been defined and the number of records in each.</p> + +<div> +<div> +<pre class="source">$ service opentaxii status +Checking opentaxii... Running +guest.phishtank_com 984 +guest.Abuse_ch 45 +guest.CyberCrime_Tracker 482 +guest.EmergingThreats_rules 0 +guest.Lehigh_edu 1030 +guest.MalwareDomainList_Hostlist 84 +guest.blutmagie_de_torExits 3236 +guest.dataForLast_7daysOnly 3377 +guest.dshield_BlockList 0 +</pre></div></div> + +<p><tt>setup</tt> Initializes the services and collections required to operate the OpenTAXII service. This will destroy all existing data. The user is prompted to continue before any data is destroyed.</p> + +<div> +<div> +<pre class="source"># service opentaxii setup +WARNING: force reset and destroy all opentaxii data? [Ny]: y +Stopping opentaxii ..Ok +2016-04-21T19:56:01.886157Z [opentaxii.server] info: api.persistence.loaded {timestamp=2016-04-21T19:56:01.886157Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.persistence.loaded, level=info} +2016-04-21T19:56:01.896503Z [opentaxii.server] info: api.auth.loaded {timestamp=2016-04-21T19:56:01.896503Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.auth.loaded, level=info} +2016-04-21T19:56:01.896655Z [opentaxii.server] info: taxiiserver.configured {timestamp=2016-04-21T19:56:01.896655Z, logger=opentaxii.server, event=taxiiserver.configured, level=info} +... +Ok +</pre></div></div> + +<p><tt>sync [collection] [begin-at] [end-at]</tt> Syncs the threat intel data available at <a class="externalLink" href="http://hailataxii.com/";>Hail a TAXII</a>. If no begin and end date is provided then data is synced over the current day only.</p> +<ul> + +<li><tt>collection</tt> Name of the collection to sync.</li> +<li><tt>begin-at</tt> Exclusive begin of time window; ISO8601</li> +<li><tt>end-at</tt> Inclusive end of time window; ISO8601</li> +</ul> + +<div> +<div> +<pre class="source">$ service opentaxii sync guest.phishtank_com ++ /usr/local/opentaxii/opentaxii-venv/bin/taxii-proxy --poll-path http://hailataxii.com/taxii-data --poll-collection guest.phishtank_com --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox --inbox-collection guest.phishtank_com --binding urn:stix.mitre.org:xml:1.1.1 --begin 2016-04-21 --end 2016-04-22 +2016-04-21 17:36:23,778 INFO: Sending Poll_Request to http://hailataxii.com/taxii-data +2016-04-21 17:36:23,784 INFO: Starting new HTTP connection (1): hailataxii.com +2016-04-21 17:36:24,175 INFO: Response received for Poll_Request from http://hailataxii.com/taxii-data +2016-04-21 17:36:24,274 INFO: Sending Inbox_Message to http://localhost:9000/services/guest.phishtank_com-inbox +... +2016-04-21 17:36:34,867 INFO: Response received for Poll_Request from http://localhost:9000/services/guest.phishtank_com-poll +2016-04-21 17:36:34,868 INFO: Content blocks count: 6993, is partial: False +</pre></div></div> + +<div class="section"> +<h3><a name="Troubleshooting"></a>Troubleshooting</h3> +<p>Should you need to explore the installation, here are instructions on doing so.</p> +<p>OpenTAXII is installed in a virtual environment. Before exploring the environment run the following commands to perform the necessary setup. The specific paths may change depending on your Ansible settings.</p> + +<div> +<div> +<pre class="source">export LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64 +export OPENTAXII_CONFIG=/usr/local/opentaxii/etc/opentaxii-conf.yml +cd /usr/local/opentaxii +. opentaxii-venv/bin/activate +</pre></div></div> + +<p>Discover available services.</p> + +<div> +<div> +<pre class="source">taxii-discovery --discovery http://localhost:9000/services/discovery +taxii-discovery --discovery http://hailataxii.com/taxii-data +</pre></div></div> + +<p>Explore available collections.</p> + +<div> +<div> +<pre class="source">taxii-collections --discovery http://localhost:9000/services/discovery +taxii-collections --discovery http://hailataxii.com/taxii-data +</pre></div></div> + +<p>Read data from a collection.</p> + +<div> +<div> +<pre class="source">taxii-poll --discovery http://localhost:9000/services/discovery -c guest.phishtank_com +taxii-poll --discovery http://hailataxii.com/taxii-data -c guest.phishtank_com --begin 2016-04-20 +</pre></div></div> + +<p>Manually load data into a collection.</p> + +<div> +<div> +<pre class="source">taxii-push \ + --discovery http://localhost:9000/services/discovery \ + --dest phishtank \ + --content-file data.xml \ + --username guest \ + --password guest +</pre></div></div> + +<p>Fetch data from a remote service and mirror it locally.</p> + +<div> +<div> +<pre class="source">taxii-proxy --poll-path http://hailataxii.com/taxii-data \ + --poll-collection guest.phishtank_com \ + --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox \ + --inbox-collection guest.phishtank_com \ + --binding urn:stix.mitre.org:xml:1.1.1 \ + --inbox-username guest \ + --inbox-password guest \ + --begin 2016-04-20 +</pre></div></div></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +̩ 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/ansible/roles/pcap_replay/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/ansible/roles/pcap_replay/index.html b/site/current-book/metron-deployment/ansible/roles/pcap_replay/index.html new file mode 100644 index 0000000..d97a97d --- /dev/null +++ b/site/current-book/metron-deployment/ansible/roles/pcap_replay/index.html @@ -0,0 +1,172 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/pcap_replay/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Pcap Replay</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Pcap Replay</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-down"></span>Ansible</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/index.html" title="Roles"><span class="icon-chevron-down"></span>Roles</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/opentaxii/index.html" title="Opentaxii"><span class="none"></span>Opentaxii</a></li> + <li class="active"><a href="#"><span class="none"></span>Pcap_replay</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-stubs/index.html" title="Sensor-stubs"><span class="none"></span>Sensor-stubs</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-test-mode/index.html" title="Sensor-test-mode"><span class="none"></span>Sensor-test-mode</a></li> + </ul> +</li> + </ul> +</li> + <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Pcap Replay</h1> +<p>This project enables packet capture data to be replayed through a network interface to simulate live network traffic. This can be used to support functional, performance, and load testing of Apache Metron.</p> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<p>To replay packet capture data, simply start the <tt>pcap-replay</tt> SysV service. To do this run the following command.</p> + +<div> +<div> +<pre class="source">service pcap-replay start +</pre></div></div> + +<p>All additional options accepted by <tt>tcpreplay</tt> can be passed to the service script to modify how the network data is replayed. For example, this makes it simple to control the amount and rate of data replayed during functional, performance and load testing.</p> +<p>Example: Replay data at a rate of 10 mbps.</p> + +<div> +<div> +<pre class="source">service pcap-replay start --mbps 10 +</pre></div></div> + +<p>Example: Replay data at a rate of 10 packets per second.</p> + +<div> +<div> +<pre class="source">service pcap-replay start --pps 10 +</pre></div></div> + +<p>All nodes on the same subnet with their network interface set to promiscuous mode will then be able to capture the network traffic being replayed. To validate, simply run something like the following.</p> + +<div> +<div> +<pre class="source">tcpdump -i eth1 +</pre></div></div> +</div> +<div class="section"> +<h2><a name="Data"></a>Data</h2> +<p>An example packet capture file has been installed at <tt>/opt/pcap-replay/example.pcap</tt>. By default, the network traffic contained within this file is continually replayed.</p> +<p>To replay your own packet capture data, simply add any number of files containing <tt>libpcap</tt> formatted packet capture data to <tt>/opt/pcap-replay</tt>. The files must end with the <tt>.pcap</tt> extension. To pick up newly installed files, simply restart the service.</p> + +<div> +<div> +<pre class="source">service pcap-replay restart +</pre></div></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html b/site/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html new file mode 100644 index 0000000..650fc89 --- /dev/null +++ b/site/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html @@ -0,0 +1,221 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-stubs/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Sensor Stubs</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Sensor Stubs</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-down"></span>Ansible</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/index.html" title="Roles"><span class="icon-chevron-down"></span>Roles</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/opentaxii/index.html" title="Opentaxii"><span class="none"></span>Opentaxii</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/pcap_replay/index.html" title="Pcap_replay"><span class="none"></span>Pcap_replay</a></li> + <li class="active"><a href="#"><span class="none"></span>Sensor-stubs</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-test-mode/index.html" title="Sensor-test-mode"><span class="none"></span>Sensor-test-mode</a></li> + </ul> +</li> + </ul> +</li> + <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Sensor Stubs</h1> +<p><a name="Sensor_Stubs"></a></p> +<p>A service has been created to simulate the behavior of a sensor by sending canned telemetry data to a Kafka topic. These “Sensor Stubs” consume fewer resources than the actual sensor that they replace.</p> +<div class="section"> +<div class="section"> +<h3><a name="a.28Q.29_How_do_the_sensor_stubs_work.3F"></a>(Q) How do the sensor stubs work?</h3> +<p>The stubs are installed with a set of canned data for each sensor type; Bro, Snort and YAF. A subset of this canned data is randomly selected and sent to the Kafka topic in batches. The timestamp of each message is updated to match current system time.</p></div> +<div class="section"> +<h3><a name="a.28Q.29_How_do_I_configure_the_message_rate.3F"></a>(Q) How do I configure the message rate?</h3> +<p>The number of telemetry messages sent in each batch, along with the time delay between batches is configurable. Before installation, these values can be configured by redefining <tt>sensor_stubs_delay</tt> and <tt>sensor_stubs_count</tt>. The values can also be configured by altering the deployed system service script at <tt>/etc/init.d/sensor-stubs</tt>.</p></div> +<div class="section"> +<h3><a name="a.28Q.29_How_do_I_install_the_sensor_stubs.3F"></a>(Q) How do I install the sensor stubs?</h3> +<p>Using the default playbooks, this role can be installed by using the Ansible tag <tt>sensor-stubs</tt>. This service is installed on the same hosts where the sensors would be; defined by the <tt>sensors</tt> host group.</p> +<p>The defaults for the “Full Dev” environment have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.</p></div> +<div class="section"> +<h3><a name="a.28Q.29_How_do_I_use_the_sensor_stubs.3F"></a>(Q) How do I use the sensor stubs?</h3> +<p>Start all sensor stubs. The output includes the PID for each running sensor stub.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs start +Starting sensor-stubs... + bro: Ok [26505] + yaf: Ok [26507] + snort: Ok [26509] +</pre></div></div> + +<p>Check the status of each sensor stub.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs status +Checking sensor-stubs... + bro: Running [26505] + yaf: Running [26507] + snort: Running [26509] +</pre></div></div> + +<p>Stop all sensor stubs.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs stop +Stopping sensor-stubs... +.. bro: Ok [26505] +.. yaf: Ok [26507] +.. snort: Ok [26509] +</pre></div></div> + +<p>Check the status. All sensor stubs should be stopped.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs status +Checking sensor-stubs... + bro: Not running + yaf: Not running + snort: Not running +</pre></div></div> + +<p>Start only the Bro sensor stub.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs start bro +Starting sensor-stubs... + bro: OK [11616] +</pre></div></div> + +<p>Stop the Bro sensor stub.</p> + +<div> +<div> +<pre class="source">$ service sensor-stubs stop bro +Stopping sensor-stubs... +.. bro: Ok [11616] +</pre></div></div> +</div> +<div class="section"> +<h3><a name="a.28Q.29_How_do_I_install_the_original_sensors.3F"></a>(Q) How do I install the original sensors?</h3> +<p>The default behavior can be changed by skipping the <tt>sensor-stubs</tt> flag and including the <tt>sensors</tt> flag. For example, to deploy “Full Dev” with the original sensors run the following command.</p> + +<div> +<div> +<pre class="source">cd metron-deployment/development/centos6 +vagrant --ansible-skip-tags="sensor-stubs,solr" up +</pre></div></div> +</div> +<div class="section"> +<h3><a name="a.28Q.29_Where_does_the_mock_data_come_from.3F"></a>(Q) Where does the mock data come from?</h3> +<p>The data produced by the sensor stubs was generated by running the sensors against the example pcap file that is distributed with Metron. This ensures that the data produced by the sensor stubs is similar to the data produced when using the actual sensors.</p></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html b/site/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html new file mode 100644 index 0000000..b989235 --- /dev/null +++ b/site/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html @@ -0,0 +1,156 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-test-mode/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Sensor Test Mode</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Sensor Test Mode</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-down"></span>Ansible</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/index.html" title="Roles"><span class="icon-chevron-down"></span>Roles</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/ansible/roles/opentaxii/index.html" title="Opentaxii"><span class="none"></span>Opentaxii</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/pcap_replay/index.html" title="Pcap_replay"><span class="none"></span>Pcap_replay</a></li> + <li><a href="../../../../metron-deployment/ansible/roles/sensor-stubs/index.html" title="Sensor-stubs"><span class="none"></span>Sensor-stubs</a></li> + <li class="active"><a href="#"><span class="none"></span>Sensor-test-mode</a></li> + </ul> +</li> + </ul> +</li> + <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Sensor Test Mode</h1> +<p>A role that configures each of the sensors to produce the maximum amount of telemetry data. This role is useful only for testing. It can be useful to support functional, performance, and load testing of Apache Metron.</p> +<p>The role does the following to maximize the amount of telemetry data produced by each Metron sensor.</p> +<ul> + +<li>Plays a packet capture file through a network interface to simulate live network traffic.</li> +<li>Configures <a class="externalLink" href="https://tools.netsa.cert.org/yaf/yaf.html";>YAF</a> with <tt>idle-timeout=0</tt>. This causes a flow record to be produced for every network packet received.</li> +<li>Configures <a class="externalLink" href="https://www.snort.org/";>Snort</a> to produce an alert for every network packet received.</li> +</ul> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<p>To enable the <tt>sensor-test-mode</tt> role apply the role to the <tt>sensors</tt> host group in your Ansible playbook.</p> + +<div> +<div> +<pre class="source">- hosts: sensors + roles: + - role: sensor-test-mode +</pre></div></div> + +<p>The role has also been added to the default <tt>metron_install.yml</tt> playbook so that it can be turned on/off with a property in both the local Virtualbox and the remote EC2 deployments.</p> + +<div> +<div> +<pre class="source">sensor_test_mode: True +</pre></div></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/development/centos6/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/development/centos6/index.html b/site/current-book/metron-deployment/development/centos6/index.html new file mode 100644 index 0000000..b036e37 --- /dev/null +++ b/site/current-book/metron-deployment/development/centos6/index.html @@ -0,0 +1,233 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/centos6/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Metron on CentOS 6</title> + <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../css/site.css" /> + <link rel="stylesheet" href="../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Metron on CentOS 6</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> + <li><a href="../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-down"></span>Development</a> + <ul class="nav nav-list"> + <li class="active"><a href="#"><span class="none"></span>Centos6</a></li> + <li><a href="../../../metron-deployment/development/fastcapa/index.html" title="Fastcapa"><span class="none"></span>Fastcapa</a></li> + <li><a href="../../../metron-deployment/development/ubuntu14/index.html" title="Ubuntu14"><span class="none"></span>Ubuntu14</a></li> + </ul> +</li> + <li><a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Metron on CentOS 6</h1> +<p>This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6.</p> +<p>Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services.</p> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<div class="section"> +<h3><a name="Prerequisites"></a>Prerequisites</h3> +<p>The computer used to deploy Apache Metron will need to have the following components installed.</p> +<ul> + +<li><a class="externalLink" href="https://github.com/ansible/ansible";>Ansible</a> (2.0.0.2, 2.2.2.0, or 2.5.0)</li> +<li><a class="externalLink" href="https://www.docker.com/community-edition";>Docker</a></li> +<li><a class="externalLink" href="https://www.vagrantup.com";>Vagrant</a> 2.0+</li> +<li><a class="externalLink" href="https://github.com/devopsgroup-io/vagrant-hostmanager";>Vagrant Hostmanager Plugin</a></li> +<li><a class="externalLink" href="https://virtualbox.org";>Virtualbox</a> 5.0+</li> +<li>Python 2.7</li> +<li>Maven 3.3.9</li> +<li>C++11 compliant compiler, like <a class="externalLink" href="https://gcc.gnu.org/projects/cxx-status.html#cxx11";>GCC</a></li> +</ul> +<p>Running the following script can help validate whether you have all the prerequisites installed and running correctly.</p> + +<div> +<div> +<pre class="source"> metron-deployment/scripts/platform-info.sh +</pre></div></div> + +<div class="section"> +<h4><a name="How_do_I_install_these_on_MacOS.3F"></a>How do I install these on MacOS?</h4> +<p>Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent <a class="externalLink" href="http://brew.sh/";>Homebrew</a> project.</p> +<ol style="list-style-type: decimal"> + +<li> + +<p>Install Homebrew by following the instructions at <a class="externalLink" href="http://brew.sh/";>Homebrew</a>.</p> +</li> +<li> + +<p>Run the following command in a terminal to install all of the required tools.</p> + +<div> +<div> +<pre class="source">brew cask install vagrant virtualbox docker +brew cask install caskroom/versions/java8 +brew install maven@3.3 git +pip install ansible==2.2.2.0 +vagrant plugin install vagrant-hostmanager +open /Applications/Docker.app +</pre></div></div> +</li> +</ol></div></div> +<div class="section"> +<h3><a name="Deploy_Metron"></a>Deploy Metron</h3> +<ol style="list-style-type: decimal"> + +<li> + +<p>Ensure that the Docker service is running.</p> +</li> +<li> + +<p>Deploy Metron</p> + +<div> +<div> +<pre class="source">cd metron-deployment/development/centos6 +vagrant up +</pre></div></div> + +<p>Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.</p> + +<div> +<div> +<pre class="source">vagrant provision +</pre></div></div> +</li> +</ol></div> +<div class="section"> +<h3><a name="Explore_Metron"></a>Explore Metron</h3> +<p>Navigate to the following resources to explore your newly minted Apache Metron environment.</p> +<ul> + +<li><a class="externalLink" href="http://node1:4201";>Metron Alerts</a></li> +<li><a class="externalLink" href="http://node1:8080";>Ambari</a></li> +</ul> +<p>Connecting to the host through SSH is as simple as running the following command.</p> + +<div> +<div> +<pre class="source">vagrant ssh +</pre></div></div> +</div> +<div class="section"> +<h3><a name="Working_with_Metron"></a>Working with Metron</h3> +<p>In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the <tt>sensor-stubs</tt> role on the Vagrant image. This will install and start the sensor stub components.</p> + +<div> +<div> +<pre class="source">vagrant --ansible-tags="sensor-stubs" provision +</pre></div></div> + +<p>Tags are listed in the playbooks, some frequently used tags:</p> +<ul> + +<li><tt>hdp-install</tt> - Install HDP</li> +<li><tt>hdp-deploy</tt> - Deploy and Start HDP Services (will start all Hadoop Services)</li> +<li><tt>sensors</tt> - Deploy and start the sensors.</li> +<li><tt>sensor-stubs</tt> - Deploy and start the sensor stubs.</li> +</ul></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/development/fastcapa/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/development/fastcapa/index.html b/site/current-book/metron-deployment/development/fastcapa/index.html new file mode 100644 index 0000000..d471576 --- /dev/null +++ b/site/current-book/metron-deployment/development/fastcapa/index.html @@ -0,0 +1,249 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/fastcapa/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Fastcapa Test Environment</title> + <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../css/site.css" /> + <link rel="stylesheet" href="../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Fastcapa Test Environment</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> + <li><a href="../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-down"></span>Development</a> + <ul class="nav nav-list"> + <li><a href="../../../metron-deployment/development/centos6/index.html" title="Centos6"><span class="none"></span>Centos6</a></li> + <li class="active"><a href="#"><span class="none"></span>Fastcapa</a></li> + <li><a href="../../../metron-deployment/development/ubuntu14/index.html" title="Ubuntu14"><span class="none"></span>Ubuntu14</a></li> + </ul> +</li> + <li><a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li> + <li><a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Fastcapa Test Environment</h1> +<p>Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly.</p> +<p>Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network.</p> +<ul> + +<li>The <tt>source</tt> node uses Metron’s <tt>pcap_replay</tt> functionality to transmit raw network packet data over a private network.</li> +<li>The <tt>sink</tt> node is running <tt>fastcapa</tt> and is capturing these network packets.</li> +<li>Fastcapa then transforms and bundles the packets into a message.</li> +<li>The message is sent to a Kafka broker running on the <tt>source</tt> node.</li> +</ul> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<p>The Fastcapa test environment can be executed on different operating systems. There is a sub-directory for each operating system that Fastcapa can be tested on.</p> +<p>To run, simply execute <tt>vagrant up</tt> within the appropriate directory. For example, to run the tests on CentOS 7.1 then execute the following commands.</p> + +<div> +<div> +<pre class="source">cd centos-7.1 +vagrant up +</pre></div></div> + +<p>Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly. If you see something like the following, then the tests have passed.</p> + +<div> +<div> +<pre class="source">$ vagrant up +==> source: Running provisioner: ansible... + source: Running ansible-playbook... +... +TASK [debug] ******************************************************************* +ok: [source] => { + "msg": "Successfully received packets sent from pcap-replay!" +} +... +TASK [debug] ******************************************************************* +ok: [source] => { + "msg": "Successfully received a Kafka message from fastcapa!" +} +</pre></div></div> + +<p>If the deployment process fails mid-course, running <tt>vagrant provision</tt> will continue the process from where it left off. This can sometimes occur when the VM reboots as part of the deployment process. The error might look like the following.</p> + +<div> +<div> +<pre class="source">TASK [fastcapa : Restart for modified kernel params] *************************** +fatal: [sink]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 127.0.0.1 closed.\r\n", "unreachable": true} + to retry, use: --limit @/Users/nallen/Development/metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.retry + +PLAY RECAP ********************************************************************* +sink : ok=11 changed=9 unreachable=1 failed=0 +source : ok=29 changed=25 unreachable=0 failed=0 + +Ansible failed to complete successfully. Any error output should be +visible above. Please fix these errors and try again. +</pre></div></div> +</div> +<div class="section"> +<h2><a name="Going_Deeper"></a>Going Deeper</h2> +<p>This section will outline in more detail the environment and how to interact with it.</p> +<div class="section"> +<h3><a name="source"></a><tt>source</tt></h3> +<p>To validate that the <tt>source</tt> node is functioning properly, run the following commands.</p> +<p>First, ensure that the <tt>pcap-replay</tt> service is running.</p> + +<div> +<div> +<pre class="source">vagrant ssh source +sudo service pcap-replay status +</pre></div></div> + +<p>Use <tt>tcpdump</tt> to ensure that the raw packet data is being sent over the private network. Enter ‘CTRL-C’ to kill the <tt>tcpdump</tt> process once you are able to see that packets are being sent.</p> + +<div> +<div> +<pre class="source">sudo yum -y install tcpdump +sudo tcpdump -i enp0s8 +</pre></div></div> +</div> +<div class="section"> +<h3><a name="sink"></a><tt>sink</tt></h3> +<p>Next validate that the <tt>sink</tt> is functioning properly. Run the following commands starting from the host operating system.</p> +<p>First, ensure that the <tt>fastcapa</tt> service is running.</p> + +<div> +<div> +<pre class="source">vagrant ssh sink +service fastcapa status +</pre></div></div> + +<p>Ensure that the raw network packet data is being received by Kafka.</p> + +<div> +<div> +<pre class="source">/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap +</pre></div></div> + +<p>Enter ‘CTRL-C’ to kill the <tt>kafka-console-consumer</tt> process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.</p></div></div> +<div class="section"> +<h2><a name="FAQ"></a>FAQ</h2> +<div class="section"> +<h3><a name="Error_Message:_Timed_out_while_waiting_for_the_machine_to_boot"></a>Error Message: Timed out while waiting for the machine to boot</h3> + +<div> +<div> +<pre class="source">Timed out while waiting for the machine to boot. This means that +Vagrant was unable to communicate with the guest machine within +the configured ("config.vm.boot_timeout" value) time period. +If you look above, you should be able to see the error(s) that +Vagrant had when attempting to connect to the machine. These errors +are usually good hints as to what may be wrong. +If you're using a custom box, make sure that networking is properly +working and you're able to connect to the machine. It is a common +problem that networking isn't setup properly in these boxes. +Verify that authentication configurations are also setup properly, +as well. +If the box appears to be booting properly, you may want to increase +the timeout ("config.vm.boot_timeout") value. +➜ centos-7.4 git:(master) ✗ vagrant status +Current machine states: +source running (virtualbox) +sink not created (virtualbox) +</pre></div></div> + +<p>If you are unable to launch any of the Fastcapa test environments, which results in a message like the one above, then you may need to upgrade your version of Virtualbox. Success has been reported with versions of VirtualBox 5.1.22+.</p></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html>
