Added: release/incubator/metron/0.3.1/book-site/metron-platform/metron-parsers/index.html ============================================================================== --- release/incubator/metron/0.3.1/book-site/metron-platform/metron-parsers/index.html (added) +++ release/incubator/metron/0.3.1/book-site/metron-platform/metron-parsers/index.html Thu Mar 16 15:07:53 2017 @@ -0,0 +1,691 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Parsers</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Parsers</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="none"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Parsers</h1> +<p><a name="Parsers"></a></p> +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>Parsers are pluggable components which are used to transform raw data (textual or raw bytes) into JSON messages suitable for downstream enrichment and indexing. </p> +<p>There are two general types types of parsers:</p> + +<ul> + +<li>A parser written in Java which conforms to the <tt>MessageParser</tt> interface. This kind of parser is optimized for speed and performance and is built for use with higher velocity topologies. These parsers are not easily modifiable and in order to make changes to them the entire topology need to be recompiled.</li> + +<li>A general purpose parser. This type of parser is primarily designed for lower-velocity topologies or for quickly standing up a parser for a new telemetry before a permanent Java parser can be written for it. As of the time of this writing, we have: + +<ul> + +<li>Grok parser: <tt>org.apache.metron.parsers.GrokParser</tt> with possible <tt>parserConfig</tt> entries of + +<ul> + +<li><tt>grokPath</tt> : The path in HDFS (or in the Jar) to the grok statement</li> + +<li><tt>patternLabel</tt> : The pattern label to use from the grok statement</li> + +<li><tt>timestampField</tt> : The field to use for timestamp</li> + +<li><tt>timeFields</tt> : A list of fields to be treated as time</li> + +<li><tt>dateFormat</tt> : The date format to use to parse the time fields</li> + +<li><tt>timezone</tt> : The timezone to use. <tt>UTC</tt> is default.</li> + </ul></li> + +<li>CSV Parser: <tt>org.apache.metron.parsers.csv.CSVParser</tt> with possible <tt>parserConfig</tt> entries of + +<ul> + +<li><tt>timestampFormat</tt> : The date format of the timestamp to use. If unspecified, the parser assumes the timestamp is ms since unix epoch.</li> + +<li><tt>columns</tt> : A map of column names you wish to extract from the CSV to their offsets (e.g. <tt>{ 'name' : 1, 'profession' : 3}</tt> would be a column map for extracting the 2nd and 4th columns from a CSV)</li> + +<li><tt>separator</tt> : The column separator, <tt>,</tt> by default. just</li> + </ul></li> + </ul></li> +</ul></div> +<div class="section"> +<h2><a name="Parser_Architecture"></a>Parser Architecture</h2> +<p><img src="../../images/parser_arch.png" alt="Architecture" /></p> +<p>Data flows through the parser bolt via kafka and into the <tt>enrichments</tt> topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an <tt>error</tt> queue. Invalid messages as determined by global validation functions are sent to an <tt>invalid</tt> queue. </p></div> +<div class="section"> +<h2><a name="Message_Format"></a>Message Format</h2> +<p>All Metron messages follow a specific format in order to ingest a message. If a message does not conform to this format it will be dropped and put onto an error queue for further examination. The message must be of a JSON format and must have a JSON tag message like so:</p> + +<div class="source"> +<div class="source"> +<pre>{"message" : message content} + +</pre></div></div> +<p>Where appropriate there is also a standardization around the 5-tuple JSON fields. This is done so the topology correlation engine further down stream can correlate messages from different topologies by these fields. We are currently working on expanding the message standardization beyond these fields, but this feature is not yet availabe. The standard field names are as follows:</p> + +<ul> + +<li>ip_src_addr: layer 3 source IP</li> + +<li>ip_dst_addr: layer 3 dest IP</li> + +<li>ip_src_port: layer 4 source port</li> + +<li>ip_dst_port: layer 4 dest port</li> + +<li>protocol: layer 4 protocol</li> + +<li>timestamp (epoch)</li> + +<li>original_string: A human friendly string representation of the message</li> +</ul> +<p>The timestamp and original_string fields are madatory. The remaining standard fields are optional. If any of the optional fields are not applicable then the field should be left out of the JSON.</p> +<p>So putting it all together a typical Metron message with all 5-tuple fields present would look like the following:</p> + +<div class="source"> +<div class="source"> +<pre>{ +"message": +{"ip_src_addr": xxxx, +"ip_dst_addr": xxxx, +"ip_src_port": xxxx, +"ip_dst_port": xxxx, +"protocol": xxxx, +"original_string": xxx, +"additional-field 1": xxx, +} + +} +</pre></div></div></div> +<div class="section"> +<h2><a name="Global_Configuration"></a>Global Configuration</h2> +<p>See the “<a href="../metron-common/index.html">Global Configuration</a>” section.</p></div> +<div class="section"> +<h2><a name="Parser_Configuration"></a>Parser Configuration</h2> +<p>The configuration for the various parser topologies is defined by JSON documents stored in zookeeper.</p> +<p>The document is structured in the following way</p> + +<ul> + +<li><tt>parserClassName</tt> : The fully qualified classname for the parser to be used.</li> + +<li><tt>filterClassName</tt> : The filter to use. This may be a fully qualified classname of a Class that implements the <tt>org.apache.metron.parsers.interfaces.MessageFilter<JSONObject></tt> interface. Message Filters are intended to allow the user to ignore a set of messages via custom logic. The existing implementations are: + +<ul> + +<li><tt>STELLAR</tt> : Allows you to apply a stellar statement which returns a boolean, which will pass every message for which the statement returns <tt>true</tt>. The Stellar statement that is to be applied is specified by the <tt>filter.query</tt> property in the <tt>parserConfig</tt>. Example Stellar Filter which includes messages which contain a the <tt>field1</tt> field:</li> + </ul></li> +</ul> + +<div class="source"> +<div class="source"> +<pre> { + "filterClassName" : "STELLAR" + ,"parserConfig" : { + "filter.query" : "exists(field1)" + } + } +</pre></div></div> + +<ul> + +<li><tt>sensorTopic</tt> : The kafka topic to send the parsed messages to.</li> + +<li><tt>parserConfig</tt> : A JSON Map representing the parser implementation specific configuration.</li> + +<li><tt>fieldTransformations</tt> : An array of complex objects representing the transformations to be done on the message generated from the parser before writing out to the kafka topic.</li> +</ul> +<p>The <tt>fieldTransformations</tt> is a complex object which defines a transformation which can be done to a message. This transformation can </p> + +<ul> + +<li>Modify existing fields to a message</li> + +<li>Add new fields given the values of existing fields of a message</li> + +<li>Remove existing fields of a message</li> +</ul> +<div class="section"> +<h3><a name="fieldTransformation_configuration"></a><tt>fieldTransformation</tt> configuration</h3> +<p>The format of a <tt>fieldTransformation</tt> is as follows:</p> + +<ul> + +<li><tt>input</tt> : An array of fields or a single field representing the input. This is optional; if unspecified, then the whole message is passed as input.</li> + +<li><tt>output</tt> : The outputs to produce from the transformation. If unspecified, it is assumed to be the same as inputs.</li> + +<li><tt>transformation</tt> : The fully qualified classname of the transformation to be used. This is either a class which implements <tt>FieldTransformation</tt> or a member of the <tt>FieldTransformations</tt> enum.</li> + +<li><tt>config</tt> : A String to Object map of transformation specific configuration.</li> +</ul> +<p>The currently implemented fieldTransformations are:</p> + +<ul> + +<li><tt>REMOVE</tt> : This transformation removes the specified input fields. If you want a conditional removal, you can pass a Metron Query Language statement to define the conditions under which you want to remove the fields.</li> +</ul> +<p>Consider the following simple configuration which will remove <tt>field1</tt> unconditionally:</p> + +<div class="source"> +<div class="source"> +<pre>{ +... + "fieldTransformations" : [ + { + "input" : "field1" + , "transformation" : "REMOVE" + } + ] +} +</pre></div></div> +<p>Consider the following simple sensor parser configuration which will remove <tt>field1</tt> whenever <tt>field2</tt> exists and whose corresponding equal to ‘foo’:</p> + +<div class="source"> +<div class="source"> +<pre>{ +... + "fieldTransformations" : [ + { + "input" : "field1" + , "transformation" : "REMOVE" + , "config" : { + "condition" : "exists(field2) and field2 == 'foo'" + } + } + ] +} +</pre></div></div> + +<ul> + +<li><tt>IP_PROTOCOL</tt> : This transformation maps IANA protocol numbers to consistent string representations.</li> +</ul> +<p>Consider the following sensor parser config to map the <tt>protocol</tt> field to a textual representation of the protocol:</p> + +<div class="source"> +<div class="source"> +<pre>{ +... + "fieldTransformations" : [ + { + "input" : "protocol" + , "transformation" : "IP_PROTOCOL" + } + ] +} +</pre></div></div> +<p>This transformation would transform <tt>{ "protocol" : 6, "source.type" : "bro", ... }</tt> into <tt>{ "protocol" : "TCP", "source.type" : "bro", ...}</tt></p> + +<ul> + +<li><tt>STELLAR</tt> : This transformation executes a set of transformations expressed as <a href="../metron-common/index.html">Stellar Language</a> statements.</li> +</ul> +<p>Consider the following sensor parser config to add three new fields to a message:</p> + +<ul> + +<li><tt>utc_timestamp</tt> : The unix epoch timestamp based on the <tt>timestamp</tt> field, a <tt>dc</tt> field which is the data center the message comes from and a <tt>dc2tz</tt> map mapping data centers to timezones</li> + +<li><tt>url_host</tt> : The host associated with the url in the <tt>url</tt> field</li> + +<li><tt>url_protocol</tt> : The protocol associated with the url in the <tt>url</tt> field</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>{ +... + "fieldTransformations" : [ + { + "transformation" : "STELLAR" + ,"output" : [ "utc_timestamp", "url_host", "url_protocol" ] + ,"config" : { + "utc_timestamp" : "TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd +HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )" + ,"url_host" : "URL_TO_HOST(url)" + ,"url_protocol" : "URL_TO_PROTOCOL(url)" + } + } + ] + ,"parserConfig" : { + "dc2tz" : { + "nyc" : "EST" + ,"la" : "PST" + ,"london" : "UTC" + } + } +} +</pre></div></div> +<p>Note that the <tt>dc2tz</tt> map is in the parser config, so it is accessible in the functions.</p></div> +<div class="section"> +<h3><a name="An_Example_Configuration_for_a_Sensor"></a>An Example Configuration for a Sensor</h3> +<p>Consider the following example configuration for the <tt>yaf</tt> sensor:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "parserClassName":"org.apache.metron.parsers.GrokParser", + "sensorTopic":"yaf", + "fieldTransformations" : [ + { + "input" : "protocol" + ,"transformation": "IP_PROTOCOL" + } + ], + "parserConfig": + { + "grokPath":"/patterns/yaf", + "patternLabel":"YAF_DELIMITED", + "timestampField":"start_time", + "timeFields": ["start_time", "end_time"], + "dateFormat":"yyyy-MM-dd HH:mm:ss.S" + } +} +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Parser_Adapters"></a>Parser Adapters</h2> +<p>Parser adapters are loaded dynamically in each Metron topology. They are defined in the Parser Config (defined above) JSON file in Zookeeper.</p> +<div class="section"> +<h3><a name="Java_Parser_Adapters"></a>Java Parser Adapters</h3> +<p>Java parser adapters are indended for higher-velocity topologies and are not easily changed or extended. As the adoption of Metron continues we plan on extending our library of Java adapters to process more log formats. As of this moment the Java adapters included with Metron are:</p> + +<ul> + +<li>org.apache.metron.parsers.ise.BasicIseParser : Parse ISE messages</li> + +<li>org.apache.metron.parsers.bro.BasicBroParser : Parse Bro messages</li> + +<li>org.apache.metron.parsers.sourcefire.BasicSourcefireParser : Parse Sourcefire messages</li> + +<li>org.apache.metron.parsers.lancope.BasicLancopeParser : Parse Lancope messages</li> +</ul></div> +<div class="section"> +<h3><a name="Grok_Parser_Adapters"></a>Grok Parser Adapters</h3> +<p>Grok parser adapters are designed primarly for someone who is not a Java coder for quickly standing up a parser adapter for lower velocity topologies. Grok relies on Regex for message parsing, which is much slower than purpose-built Java parsers, but is more extensible. Grok parsers are defined via a config file and the topplogy does not need to be recombiled in order to make changes to them. An example of a Grok perser is:</p> + +<ul> + +<li>org.apache.metron.parsers.GrokParser</li> +</ul> +<p>For more information on the Grok project please refer to the following link:</p> +<p><a class="externalLink" href="https://github.com/thekrakken/java-grok";>https://github.com/thekrakken/java-grok</a></p> +<p><a name="Starting_the_Parser_Topology"></a></p> +<h1>Starting the Parser Topology</h1> +<p>Starting a particular parser topology on a running Metron deployment is as easy as running the <tt>start_parser_topology.sh</tt> script located in <tt>$METRON_HOME/bin</tt>. This utility will allow you to configure and start the running topology assuming that the sensor specific parser configuration exists within zookeeper.</p> +<p>The usage for <tt>start_parser_topology.sh</tt> is as follows:</p> + +<div class="source"> +<div class="source"> +<pre>usage: start_parser_topology.sh + -e,--extra_topology_options <JSON_FILE> Extra options in the form + of a JSON file with a map + for content. + -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options + in the form of a JSON file + with a map for content. + Possible keys are: + retryDelayMaxMs,retryDelay + Multiplier,retryInitialDel + ayMs,stateUpdateIntervalMs + ,bufferSizeBytes,fetchMaxW + ait,fetchSizeBytes,maxOffs + etBehind,metricsTimeBucket + SizeInSecs,socketTimeoutMs + -ewnt,--error_writer_num_tasks <NUM_TASKS> Error Writer Num Tasks + -ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism + Hint + -h,--help This screen + -iwnt,--invalid_writer_num_tasks <NUM_TASKS> Invalid Writer Num Tasks + -iwp,--invalid_writer_p <PARALLELISM_HINT> Invalid Message Writer + Parallelism Hint + -k,--kafka <BROKER_URL> Kafka Broker URL + -mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds + -mtp,--max_task_parallelism <MAX_TASK> Max task parallelism + -na,--num_ackers <NUM_ACKERS> Number of Ackers + -nw,--num_workers <NUM_WORKERS> Number of Workers + -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks + -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint + -s,--sensor <SENSOR_TYPE> Sensor Type + -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks + -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint + -t,--test <TEST> Run in Test Mode + -z,--zk <ZK_QUORUM> Zookeeper Quroum URL + (zk1:2181,zk2:2181,... +</pre></div></div></div></div> +<div class="section"> +<h2><a name="The_--extra_kafka_spout_config_Option"></a>The <tt>--extra_kafka_spout_config</tt> Option</h2> +<p>These options are intended to configure the Storm Kafka Spout more completely. These options can be specified in a JSON file containing a map associating the kafka spout configuration parameter to a value. The range of values possible to configure are:</p> + +<ul> + +<li>retryDelayMaxMs</li> + +<li>retryDelayMultiplier</li> + +<li>retryInitialDelayMs</li> + +<li>stateUpdateIntervalMs</li> + +<li>bufferSizeBytes</li> + +<li>fetchMaxWait</li> + +<li>fetchSizeBytes</li> + +<li>maxOffsetBehind</li> + +<li>metricsTimeBucketSizeInSecs</li> + +<li>socketTimeoutMs</li> +</ul> +<p>These are described in some detail <a class="externalLink" href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/storm-kafka-api-ref.html";>here</a>.</p> +<p>For instance, creating a JSON file which will set the <tt>bufferSizeBytes</tt> to 2MB and <tt>retryDelayMaxMs</tt> to 2000 would look like</p> + +<div class="source"> +<div class="source"> +<pre>{ + "bufferSizeBytes" : 2000000, + "retryDelayMaxMs" : 2000 +} +</pre></div></div> +<p>This would be loaded by passing the file as argument to <tt>--extra_kafka_spout_config</tt></p></div> +<div class="section"> +<h2><a name="The_--extra_topology_options_Option"></a>The <tt>--extra_topology_options</tt> Option</h2> +<p>These options are intended to be Storm configuration options and will live in a JSON file which will be loaded into the Storm config. For instance, if you wanted to set a storm property on the config called <tt>topology.ticks.tuple.freq.secs</tt> to 1000 and <tt>storm.local.dir</tt> to <tt>/opt/my/path</tt> you could create a file called <tt>custom_config.json</tt> containing </p> + +<div class="source"> +<div class="source"> +<pre>{ + "topology.ticks.tuple.freq.secs" : 1000, + "storm.local.dir" : "/opt/my/path" +} +</pre></div></div> +<p>and pass <tt>--extra_topology_options custom_config.json</tt> to <tt>start_parser_topology.sh</tt>.</p> +<p><a name="Notes_on_Performance_Tuning"></a></p> +<h1>Notes on Performance Tuning</h1> +<p>Default installed Metron is untuned for production deployment. There are a few knobs to tune to get the most out of your system.</p></div> +<div class="section"> +<h2><a name="Kafka_Queue"></a>Kafka Queue</h2> +<p>The kafka queue associated with your parser is a collection point for all of the data sent to your parser. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect from your parser topology.</p></div> +<div class="section"> +<h2><a name="Parser_Topology"></a>Parser Topology</h2> +<p>The enrichment topology as started by the <tt>$METRON_HOME/bin/start_parser_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the arguments sent to this utility.</p> + +<ul> + +<li>Topology Wide + +<ul> + +<li><tt>--num_workers</tt> : The number of workers for the topology</li> + +<li><tt>--num_ackers</tt> : The number of ackers for the topology</li> + </ul></li> + +<li>The Kafka Spout + +<ul> + +<li><tt>--spout_num_tasks</tt> : The number of tasks for the spout</li> + +<li><tt>--spout_p</tt> : The parallelism hint for the spout</li> + +<li>Ensure that the spout has enough parallelism so that it can dedicate a worker per partition in your kafka topic.</li> + </ul></li> + +<li>The Parser Bolt + +<ul> + +<li><tt>--parser_num_tasks</tt> : The number of tasks for the parser bolt</li> + +<li><tt>--parser_p</tt> : The parallelism hint for the spout</li> + +<li>This is bolt that gets the most processing, so ensure that it is configured with sufficient parallelism to match your throughput expectations.</li> + </ul></li> + +<li>The Error Message Writer Bolt + +<ul> + +<li><tt>--error_writer_num_tasks</tt> : The number of tasks for the error writer bolt</li> + +<li><tt>--error_writer_p</tt> : The parallelism hint for the error writer bolt</li> + </ul></li> + +<li>The Invalid Message Writer Bolt + +<ul> + +<li><tt>--invalid_writer_num_tasks</tt> : The number of tasks for the error writer bolt</li> + +<li><tt>--invalid_writer_p</tt> : The parallelism hint for the error writer bolt</li> + </ul></li> +</ul> +<p>Finally, if workers and executors are new to you, the following might be of use to you:</p> + +<ul> + +<li><a class="externalLink" href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/";>Understanding the Parallelism of a Storm Topology</a></li> +</ul></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: release/incubator/metron/0.3.1/book-site/metron-platform/metron-pcap-backend/index.html ============================================================================== --- release/incubator/metron/0.3.1/book-site/metron-platform/metron-pcap-backend/index.html (added) +++ release/incubator/metron/0.3.1/book-site/metron-platform/metron-pcap-backend/index.html Thu Mar 16 15:07:53 2017 @@ -0,0 +1,376 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Metron PCAP Backend</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Metron PCAP Backend</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="none"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Pcap-backend</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Metron PCAP Backend</h1> +<p><a name="Metron_PCAP_Backend"></a></p> +<p>The purpose of the Metron PCAP backend is to create a storm topology capable of ingesting rapidly raw packet capture data directly into HDFS from Kafka.</p> +<div class="section"> +<h2><a name="The_Sensors_Feeding_Kafka"></a>The Sensors Feeding Kafka</h2> +<p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p> + +<ul> + +<li>The pycapa <a href="../../metron-sensors/pycapa/index.html">tool</a> aimed at low-volume packet capture</li> + +<li>The <a class="externalLink" href="http://dpdk.org/";>DPDK</a> based <a href="../../metron-sensors/fastcapa/index.html">tool</a> aimed at high-volume packet capture</li> +</ul> +<p>Both of these sensors feed kafka raw packet data directly into Kafka. The format of the record structure that this component expects is the following:</p> + +<ul> + +<li>A key which is the byte representation of a 64-bit <tt>unsigned long</tt> representing a time-unit since the unix epoch</li> + +<li>A value which is the raw packet data without header (either global pcap header or packet header)</li> +</ul></div> +<div class="section"> +<h2><a name="The_PCAP_Topology"></a>The PCAP Topology</h2> +<p>The structure of the topology is extremely simple. In fact, it is a spout-only topology. The <tt>Storm Kafka</tt> spout is used but extended to allow a callback to be used rather than having a separate bolt. </p> +<p>The following happens as part of this spout for each packet:</p> + +<ul> + +<li>A custom <tt>Scheme</tt> is used which attaches the appropriate headers to the packet (both global and packet headers) using the timestamp in the key and the raw packet data in the value.</li> + +<li>A callback is called which appends the packet data to a sequence file in HDFS.</li> +</ul></div> +<div class="section"> +<h2><a name="The_Files_on_HDFS"></a>The Files on HDFS</h2> +<p>The sequence files on HDFS fit the following pattern: <tt>$BASE_PATH/pcap_$TOPIC_$TS_$PARTITION_$UUID</tt></p> +<p>where</p> + +<ul> + +<li><tt>BASE_PATH</tt> is the base path to where pcap data is stored in HDFS</li> + +<li><tt>TOPIC</tt> is the kafka topic</li> + +<li><tt>TS</tt> is the timestamp, in nanoseconds since the unix epoch</li> + +<li><tt>PARTITION</tt> is the kafka partition</li> + +<li><tt>UUID</tt> the UUID for the storm worker</li> +</ul> +<p>These files contain a set of packet data with headers on them in sequence files.</p></div> +<div class="section"> +<h2><a name="Configuration"></a>Configuration</h2> +<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/etc/env/pcap.properties</tt> and the possible options are as follows:</p> + +<ul> + +<li><tt>spout.kafka.topic.pcap</tt> : The kafka topic to listen to</li> + +<li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li> + +<li><tt>kafka.pcap.start</tt> : One of <tt>START</tt>, <tt>END</tt>, <tt>WHERE_I_LEFT_OFF</tt> representing where to start listening on the queue.</li> + +<li><tt>kafka.pcap.numPackets</tt> : The number of packets to keep in one file.</li> + +<li><tt>kafka.pcap.maxTimeMS</tt> : The number of packets to keep in one file in terms of duration (in milliseconds). For instance, you may only want to keep an hour’s worth of packets in a given file.</li> + +<li><tt>kafka.pcap.ts_scheme</tt> : One of <tt>FROM_KEY</tt> or <tt>FROM_VALUE</tt>. You really only want <tt>FROM_KEY</tt> as that fits the current tooling. <tt>FROM_VALUE</tt> assumes that fully headerized packets are coming in on the value, which is legacy.</li> + +<li><tt>kafka.pcap.out</tt> : The directory in HDFS to store the packet capture data</li> + +<li><tt>kafka.pcap.ts_granularity</tt> : The granularity of timing used in the timestamps. One of <tt>MILLISECONDS</tt>, <tt>MICROSECONDS</tt>, or <tt>NANOSECONDS</tt> representing milliseconds, microseconds or nanoseconds since the unix epoch (respectively).</li> +</ul></div> +<div class="section"> +<h2><a name="Starting_the_Topology"></a>Starting the Topology</h2> +<p>To assist in starting the topology, a utility script which takes no arguments has been created to make this very simple. Simply, execute <tt>$METRON_HOME/bin/start_pcap_topology.sh</tt>.</p></div> +<div class="section"> +<h2><a name="Utilities"></a>Utilities</h2> +<div class="section"> +<h3><a name="Inspector_Utility"></a>Inspector Utility</h3> +<p>In order to ensure that data can be read back out, a utility, <tt>$METRON_HOME/bin/pcap_inspector.sh</tt> has been created to read portions of the sequence files.</p> + +<div class="source"> +<div class="source"> +<pre>usage: PcapInspector + -h,--help Generate Help screen + -i,--input <SEQ_FILE> Input sequence file on HDFS + -n,--num_packets <N> Number of packets to dump +</pre></div></div></div> +<div class="section"> +<h3><a name="Query_Filter_Utility"></a>Query Filter Utility</h3> +<p>This tool exposes the two methods for filtering PCAP data via a command line tool:</p> + +<ul> + +<li>fixed</li> + +<li>query (Metron Stellar)</li> +</ul> +<p>The tool is executed via </p> + +<div class="source"> +<div class="source"> +<pre>${metron_home}/bin/pcap_query.sh [fixed|query] +</pre></div></div> +<div class="section"> +<h4><a name="Usage"></a>Usage</h4> + +<div class="source"> +<div class="source"> +<pre>usage: Fixed filter options + -bop,--base_output_path <arg> Query result output path. Default is + '/tmp' + -bp,--base_path <arg> Base PCAP data path. Default is + '/apps/metron/pcap' + -da,--ip_dst_addr <arg> Destination IP address + -df,--date_format <arg> Date format to use for parsing start_time + and end_time. Default is to use time in + millis since the epoch. + -dp,--ip_dst_port <arg> Destination port + -et,--end_time <arg> Packet end time range. Default is current + system time. + -nr,--num_reducers <arg> The number of reducers to use. Default + is 10. + -h,--help Display help + -ir,--include_reverse Indicates if filter should check swapped + src/dest addresses and IPs + -p,--protocol <arg> IP Protocol + -sa,--ip_src_addr <arg> Source IP address + -sp,--ip_src_port <arg> Source port + -st,--start_time <arg> (required) Packet start time range. +</pre></div></div> + +<div class="source"> +<div class="source"> +<pre>usage: Query filter options + -bop,--base_output_path <arg> Query result output path. Default is + '/tmp' + -bp,--base_path <arg> Base PCAP data path. Default is + '/apps/metron/pcap' + -df,--date_format <arg> Date format to use for parsing start_time + and end_time. Default is to use time in + millis since the epoch. + -et,--end_time <arg> Packet end time range. Default is current + system time. + -nr,--num_reducers <arg> The number of reducers to use. Default + is 10. + -h,--help Display help + -q,--query <arg> Query string to use as a filter + -st,--start_time <arg> (required) Packet start time range. +</pre></div></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/incubator/metron/0.3.1/book-site/metron-sensors/fastcapa/index.html ============================================================================== --- release/incubator/metron/0.3.1/book-site/metron-sensors/fastcapa/index.html (added) +++ release/incubator/metron/0.3.1/book-site/metron-sensors/fastcapa/index.html Thu Mar 16 15:07:53 2017 @@ -0,0 +1,195 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Fastcapa</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Fastcapa</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-down"></i> + Sensors</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Fastcapa</a> + </li> + + <li> + + <a href="../../metron-sensors/pycapa/index.html" title="Pycapa"> + <i class="none"></i> + Pycapa</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Fastcapa</h1> +<p>Fastcapa is an Apache Metron sensor that performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. </p> +<p>The sensor will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data into a Hadoop-y cluster.</p> +<p>Fastcapa leverages the Data Plane Development Kit (<a class="externalLink" href="http://dpdk.org/";>DPDK</a>). DPDK is a set of libraries and drivers to perform fast packet processing in Linux user space. </p> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<p>The quickest way to get up and running is to use a Virtualbox environment on your local machine. The necessary files and instructions to do this are located at <a href="../../metron-deployment/vagrant/fastcapa-test-platform/index.html"><tt>metron-deployment/vagrant/fastcapa-vagrant</tt></a>. </p></div> +<div class="section"> +<h2><a name="Installation"></a>Installation</h2> +<p>The process of installing Fastcapa has a fair number of steps and involves building DPDK, loading specific kernel modules, enabling huge page memory, and binding compatible network interface cards.</p> +<p>The best documentation is code that actually does this for you. An Ansible role that performs the entire installation procedure can be found at <a href="../../metron-deployment/roles/fastcapa/index.html"><tt>metron-deployment/roles/fastcapa</tt></a>. Use this to install Fastcapa or as a guide for manual installation.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/incubator/metron/0.3.1/book-site/metron-sensors/index.html ============================================================================== --- release/incubator/metron/0.3.1/book-site/metron-sensors/index.html (added) +++ release/incubator/metron/0.3.1/book-site/metron-sensors/index.html Thu Mar 16 15:07:53 2017 @@ -0,0 +1,193 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Metron Sensors</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Metron Sensors</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Sensors</a> + <ul class="nav nav-list"> + + <li> + + <a href="../metron-sensors/fastcapa/index.html" title="Fastcapa"> + <i class="none"></i> + Fastcapa</a> + </li> + + <li> + + <a href="../metron-sensors/pycapa/index.html" title="Pycapa"> + <i class="none"></i> + Pycapa</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Metron Sensors</h1> +<p><a name="Metron_Sensors"></a></p> + +<ul> + +<li>Fast CAPA</li> + +<li>Py CAPA</li> +</ul> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/incubator/metron/0.3.1/book-site/metron-sensors/pycapa/index.html ============================================================================== --- release/incubator/metron/0.3.1/book-site/metron-sensors/pycapa/index.html (added) +++ release/incubator/metron/0.3.1/book-site/metron-sensors/pycapa/index.html Thu Mar 16 15:07:53 2017 @@ -0,0 +1,262 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Pycapa</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Pycapa</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-down"></i> + Sensors</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"> + <i class="none"></i> + Fastcapa</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Pycapa</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Pycapa</h1> +<p><a name="Pycapa"></a></p> +<div class="section"> +<h2><a name="Overview"></a>Overview</h2> +<p>Pycapa performs network packet capture, both off-the-wire and from Kafka, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/incubator-metron";>Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic.</p></div> +<div class="section"> +<h2><a name="Installation"></a>Installation</h2> + +<div class="source"> +<div class="source"> +<pre>pip install -r requirements.txt +python setup.py install +</pre></div></div></div> +<div class="section"> +<h2><a name="Usage"></a>Usage</h2> + +<div class="source"> +<div class="source"> +<pre>$ pycapa --help +usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t TOPIC] [-n PACKET_COUNT] + [-d DEBUG] [-i INTERFACE] + +optional arguments: + -h, --help show this help message and exit + -p, --producer sniff packets and send to kafka + -c, --consumer read packets from kafka + -k KAFKA_BROKERS, --kafka KAFKA_BROKERS + kafka broker(s) + -t TOPIC, --topic TOPIC + kafka topic + -n PACKET_COUNT, --number PACKET_COUNT + number of packets to consume + -d DEBUG, --debug DEBUG + debug every X packets + -i INTERFACE, --interface INTERFACE + interface to listen on +</pre></div></div> +<p>Pycapa has two primary runtime modes.</p> +<div class="section"> +<h3><a name="Producer_Mode"></a>Producer Mode</h3> +<p>Pycapa can be configured to capture packets from a network interface and then forward those packets to a Kafka topic. The following example will capture packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>pycapa --producer --kafka localhost:9092 --topic pcap -i eth0 +</pre></div></div> +<p>To output debug messages every 100 captured packets, run the following.</p> + +<div class="source"> +<div class="source"> +<pre>pycapa --producer --kafka localhost:9092 --topic pcap -i eth0 --debug 100 +</pre></div></div></div> +<div class="section"> +<h3><a name="Consumer_Mode"></a>Consumer Mode</h3> +<p>Pycapa can be configured to consume packets from a Kafka topic and then write those packets to a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a>. To read 100 packets from a kafka topic and then write those to a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a>, run the following command. The file <tt>out.pcap</tt> can then be opened with a tool such as Wireshark for further validation.</p> + +<div class="source"> +<div class="source"> +<pre>pycapa --consumer --kafka localhost:9092 --topic pcap --n 100 > out.pcap +</pre></div></div> +<p>To consume packets from Kafka continuously and print debug messages every 10 packets, run the following command. </p> + +<div class="source"> +<div class="source"> +<pre>pycapa --consumer --kafka localhost:9092 --topic pcap --debug 10 +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Dependencies"></a>Dependencies</h2> + +<ul> + +<li><a class="externalLink" href="https://github.com/dpkp/kafka-python";>kafka-python</a></li> + +<li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy";>pcapy</a></li> +</ul></div> +<div class="section"> +<h2><a name="Implementation"></a>Implementation</h2> +<p>When run in Producer Mode, Pycapa embeds the raw network packet data in the Kafka message. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch. This value is in network byte order.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
