http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-platform/metron-parsers/parser-testing.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-parsers/parser-testing.html b/site/current-book/metron-platform/metron-parsers/parser-testing.html index 0edfd53..ff22013 100644 --- a/site/current-book/metron-platform/metron-parsers/parser-testing.html +++ b/site/current-book/metron-platform/metron-parsers/parser-testing.html @@ -1,356 +1,180 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/parser-testing.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Parser Contribution and Testing</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Parser Contribution and Testing</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Parser Contribution and Testing</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-down"></i> - Platform</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"> - <i class="none"></i> - Performance-tuning-guide</a> - </li> - - <li> - - <a href="../../metron-platform/metron-api/index.html" title="Api"> - <i class="none"></i> - Api</a> - </li> - - <li> - - <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> - Common</a> - </li> - - <li> - - <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> - <i class="none"></i> - Data-management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"> - <i class="none"></i> - Elasticsearch</a> - </li> - - <li> - - <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> - <i class="none"></i> - Enrichment</a> - </li> - - <li> - - <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> - <i class="none"></i> - Indexing</a> - </li> - - <li> - - <a href="../../metron-platform/metron-management/index.html" title="Management"> - <i class="none"></i> - Management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> - <i class="icon-chevron-down"></i> - Parsers</a> - <ul class="nav nav-list"> - - <li class="active"> - - <a href="#"><i class="none"></i>parser-testing</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> - <i class="none"></i> - Pcap-backend</a> - </li> - - <li> - - <a href="../../metron-platform/metron-writer/index.html" title="Writer"> - <i class="none"></i> - Writer</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> + <ul class="nav nav-list"> + <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> + <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> + <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> + <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> + <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> + <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> + <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> + <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-down"></span>Parsers</a> + <ul class="nav nav-list"> + <li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li> + <li class="active"><a href="#"><span class="none"></span>parser-testing</a></li> + </ul> +</li> + <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> + </ul> +</li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Parser Contribution and Testing</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Parser Contribution and Testing</h1> <p><a name="Parser_Contribution_and_Testing"></a></p> -<p>So you want to contribute a parser to Apache Metron. First off, on behalf of the community, thank you very much! Now that you have implemented a parser by writing a java class which implements <tt>org.apache.metron.parsers.interfaces.MessageParser</tt> what are the testing expectations for a new parser?</p> +<p>So you want to contribute a parser to Apache Metron. First off, on behalf of the community, thank you very much! Now that you have implemented a parser by writing a java class which implements <tt>org.apache.metron.parsers.interfaces.MessageParser</tt> what are the testing expectations for a new parser?</p> <p>It is expected that a new parser have two tests:</p> - <ul> - + <li>A JUnit test directly testing your parser class.</li> - <li>An Integration test validating that your parser class can parse messages inside the <tt>ParserBolt</tt>.</li> </ul> <div class="section"> <h2><a name="The_JUnit_Test"></a>The JUnit Test</h2> -<p>The JUnit Test should be focused on testing your Parser directly. You should feel free to use mocks or stubs or whatever else you need to completely test that unit of functionality.</p></div> +<p>The JUnit Test should be focused on testing your Parser directly. You should feel free to use mocks or stubs or whatever else you need to completely test that unit of functionality.</p></div> <div class="section"> <h2><a name="The_Integration_Test"></a>The Integration Test</h2> -<p>Integration tests are more structured. The intent is that the parser that you have implemented can be driven successfully from <tt>org.apache.metron.parsers.bolt.ParserBolt</tt>.</p> +<p>Integration tests are more structured. The intent is that the parser that you have implemented can be driven successfully from <tt>org.apache.metron.parsers.bolt.ParserBolt</tt>.</p> <p>The procedure for creating a new test is as follows:</p> - <ul> - + <li>Create an integration test that extends <tt>org.apache.metron.parsers.integration.ParserIntegrationTest</tt> - <ul> - + <li>Override <tt>getSensorType()</tt> to return the sensor type to be used in the test (referred to as <tt>${sensor_type}</tt> at times)</li> - <li>Override <tt>getValidations()</tt> to indicate how you want the output of the parser to be validated (more on validations later)</li> - <li>Optionally <tt>readSensorConfig(String sensorType)</tt> to read the sensor config - <ul> - -<li>By default, we will pull this from <tt>metron-parsers/src/main/config/zookeeper/parsers/${sensor_type}</tt>. Override if you want to provide your own</li> - </ul></li> - + +<li>By default, we will pull this from <tt>metron-parsers/src/main/config/zookeeper/parsers/${sensor_type}</tt>. Override if you want to provide your own</li> +</ul> +</li> <li>Optionally <tt>readGlobalConfig()</tt> to return the global config - <ul> - -<li>By default, we will pull this from <tt>metron-integration-test/src/main/config/zookeeper/global.json)</tt>. Override if you want to provide your own</li> - </ul></li> - </ul></li> - + +<li>By default, we will pull this from <tt>metron-integration-test/src/main/config/zookeeper/global.json)</tt>. Override if you want to provide your own</li> +</ul> +</li> +</ul> +</li> <li>Place sample input data in <tt>metron-integration-test/src/main/sample/data/${sensor_type}/raw</tt> - <ul> - + <li>It should be one line per input record.</li> - </ul></li> - +</ul> +</li> <li>Place expected output based on sample data in <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> - <ul> - + <li>Line <tt>k</tt> in the expected data should match with line <tt>k</tt></li> - </ul></li> </ul> -<p>The way these tests function is by creating a <tt>ParserBolt</tt> instance with your specified global configuration and sensor configuration. It will then send your specified sample input data in line-by-line. It will then perform some basic sanity validation:</p> - +</li> +</ul> +<p>The way these tests function is by creating a <tt>ParserBolt</tt> instance with your specified global configuration and sensor configuration. It will then send your specified sample input data in line-by-line. It will then perform some basic sanity validation:</p> <ul> - + <li>Ensure no errors were logged</li> - <li>Execute your specified validation methods</li> </ul> <div class="section"> <h3><a name="Validations"></a>Validations</h3> -<p>Validations are functions which indicate how one should validate the parsed messages. The basic one which is sufficient for most cases is <tt>org.apache.metron.parsers.integration.validation.SampleDataValidation</tt>. This will read the expected results from <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> and validate that the actual parsed message conforms (excluding timestamp).</p> +<p>Validations are functions which indicate how one should validate the parsed messages. The basic one which is sufficient for most cases is <tt>org.apache.metron.parsers.integration.validation.SampleDataValidation</tt>. This will read the expected results from <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> and validate that the actual parsed message conforms (excluding timestamp).</p> <p>If you have special validations required, you may implement your own and return an instance of that in the <tt>getValidations()</tt> method of your Integration Test.</p></div> <div class="section"> <h3><a name="Sample_Integration_Test"></a>Sample Integration Test</h3> <p>A sample integration test for the <tt>snort</tt> parser is as follows:</p> -<div class="source"> -<div class="source"> -<pre>public class SnortIntegrationTest extends ParserIntegrationTest { +<div> +<div> +<pre class="source">public class SnortIntegrationTest extends ParserIntegrationTest { @Override String getSensorType() { return "snort"; @@ -364,23 +188,17 @@ } } </pre></div></div></div></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-platform/metron-pcap-backend/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-pcap-backend/index.html b/site/current-book/metron-platform/metron-pcap-backend/index.html index 01961eb..5206fd0 100644 --- a/site/current-book/metron-platform/metron-pcap-backend/index.html +++ b/site/current-book/metron-platform/metron-pcap-backend/index.html @@ -1,408 +1,217 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-pcap-backend/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron PCAP Backend</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Metron PCAP Backend</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Metron PCAP Backend</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-down"></i> - Platform</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"> - <i class="none"></i> - Performance-tuning-guide</a> - </li> - - <li> - - <a href="../../metron-platform/metron-api/index.html" title="Api"> - <i class="none"></i> - Api</a> - </li> - - <li> - - <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> - Common</a> - </li> - - <li> - - <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> - <i class="none"></i> - Data-management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"> - <i class="none"></i> - Elasticsearch</a> - </li> - - <li> - - <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> - <i class="none"></i> - Enrichment</a> - </li> - - <li> - - <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> - <i class="none"></i> - Indexing</a> - </li> - - <li> - - <a href="../../metron-platform/metron-management/index.html" title="Management"> - <i class="none"></i> - Management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> - <i class="icon-chevron-right"></i> - Parsers</a> - </li> - - <li class="active"> - - <a href="#"><i class="none"></i>Pcap-backend</a> - </li> - - <li> - - <a href="../../metron-platform/metron-writer/index.html" title="Writer"> - <i class="none"></i> - Writer</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> + <ul class="nav nav-list"> + <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> + <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> + <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> + <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> + <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> + <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> + <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> + <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> + <li class="active"><a href="#"><span class="none"></span>Pcap-backend</a></li> + <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> + </ul> +</li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Metron PCAP Backend</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Metron PCAP Backend</h1> <p><a name="Metron_PCAP_Backend"></a></p> <p>The purpose of the Metron PCAP backend is to create a storm topology capable of rapidly ingesting raw packet capture data directly into HDFS from Kafka.</p> - <ul> - + <li><a href="#the-sensors-feeding-kafka">Sensors</a></li> - <li><a href="#the-pcap-topology">PCAP Topology</a></li> - <li><a href="#the-files-on-hdfs">HDFS Files</a></li> - <li><a href="#Configuration">Configuration</a></li> - <li><a href="#Starting_the_Topology">Starting the Topology</a></li> - <li><a href="#Utilities">Utilities</a> - <ul> - + <li><a href="#Inspector_Utility">Inspector Utility</a></li> - <li><a href="#Query_Filter_Utility">Query Filter Utility</a></li> - </ul></li> - +</ul> +</li> <li><a href="#Performance_Tuning">Performance Tuning</a></li> </ul> <div class="section"> <h2><a name="The_Sensors_Feeding_Kafka"></a>The Sensors Feeding Kafka</h2> -<p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p> - +<p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p> <ul> - + <li>The pycapa <a href="../../metron-sensors/pycapa/index.html">tool</a> aimed at low-volume packet capture</li> - <li>The <a class="externalLink" href="http://dpdk.org/";>DPDK</a> based <a href="../../metron-sensors/fastcapa/index.html">tool</a> aimed at high-volume packet capture</li> </ul> <p>Both of these sensors feed kafka raw packet data directly into Kafka. The format of the record structure that this component expects is the following:</p> - <ul> - + <li>A key which is the byte representation of a 64-bit <tt>unsigned long</tt> representing a time-unit since the unix epoch</li> - <li>A value which is the raw packet data without header (either global pcap header or packet header)</li> </ul></div> <div class="section"> <h2><a name="The_PCAP_Topology"></a>The PCAP Topology</h2> -<p>The structure of the topology is extremely simple. In fact, it is a spout-only topology. The <tt>Storm Kafka</tt> spout is used but extended to allow a callback to be used rather than having a separate bolt. </p> +<p>The structure of the topology is extremely simple. In fact, it is a spout-only topology. The <tt>Storm Kafka</tt> spout is used but extended to allow a callback to be used rather than having a separate bolt.</p> <p>The following happens as part of this spout for each packet:</p> - <ul> - + <li>A custom <tt>Scheme</tt> is used which attaches the appropriate headers to the packet (both global and packet headers) using the timestamp in the key and the raw packet data in the value.</li> - <li>A callback is called which appends the packet data to a sequence file in HDFS.</li> </ul></div> <div class="section"> <h2><a name="The_Files_on_HDFS"></a>The Files on HDFS</h2> <p>The sequence files on HDFS fit the following pattern: <tt>$BASE_PATH/pcap_$TOPIC_$TS_$PARTITION_$UUID</tt></p> <p>where</p> - <ul> - + <li><tt>BASE_PATH</tt> is the base path to where pcap data is stored in HDFS</li> - <li><tt>TOPIC</tt> is the kafka topic</li> - <li><tt>TS</tt> is the timestamp, in nanoseconds since the unix epoch</li> - <li><tt>PARTITION</tt> is the kafka partition</li> - <li><tt>UUID</tt> the UUID for the storm worker</li> </ul> <p>These files contain a set of packet data with headers on them in sequence files.</p></div> <div class="section"> <h2><a name="Configuration"></a>Configuration</h2> <p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/pcap.properties</tt> and the possible options are as follows:</p> - <ul> - + <li><tt>spout.kafka.topic.pcap</tt> : The kafka topic to listen to</li> - -<li><tt>storm.auto.credentials</tt> : The kerberos ticket renewal. If running on a kerberized cluster, this should be <tt>['org.apache.storm.security.auth.kerberos.AutoTGT']</tt></li> - -<li><tt>kafka.security.protocol</tt> : The security protocol to use for kafka. This should be <tt>PLAINTEXT</tt> for a non-kerberized cluster and probably <tt>SASL_PLAINTEXT</tt> for a kerberized cluster.</li> - -<li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li> - +<li><tt>storm.auto.credentials</tt> : The kerberos ticket renewal. If running on a kerberized cluster, this should be <tt>['org.apache.storm.security.auth.kerberos.AutoTGT']</tt></li> +<li><tt>kafka.security.protocol</tt> : The security protocol to use for kafka. This should be <tt>PLAINTEXT</tt> for a non-kerberized cluster and probably <tt>SASL_PLAINTEXT</tt> for a kerberized cluster.</li> +<li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li> <li><tt>kafka.pcap.start</tt> : One of <tt>EARLIEST</tt>, <tt>LATEST</tt>, <tt>UNCOMMITTED_EARLIEST</tt>, <tt>UNCOMMITTED_LATEST</tt> representing where to start listening on the queue.</li> - <li><tt>kafka.pcap.numPackets</tt> : The number of packets to keep in one file.</li> - -<li><tt>kafka.pcap.maxTimeMS</tt> : The number of packets to keep in one file in terms of duration (in milliseconds). For instance, you may only want to keep an hour’s worth of packets in a given file.</li> - -<li><tt>kafka.pcap.ts_scheme</tt> : One of <tt>FROM_KEY</tt> or <tt>FROM_VALUE</tt>. You really only want <tt>FROM_KEY</tt> as that fits the current tooling. <tt>FROM_VALUE</tt> assumes that fully headerized packets are coming in on the value, which is legacy.</li> - +<li><tt>kafka.pcap.maxTimeMS</tt> : The number of packets to keep in one file in terms of duration (in milliseconds). For instance, you may only want to keep an hour’s worth of packets in a given file.</li> +<li><tt>kafka.pcap.ts_scheme</tt> : One of <tt>FROM_KEY</tt> or <tt>FROM_VALUE</tt>. You really only want <tt>FROM_KEY</tt> as that fits the current tooling. <tt>FROM_VALUE</tt> assumes that fully headerized packets are coming in on the value, which is legacy.</li> <li><tt>kafka.pcap.out</tt> : The directory in HDFS to store the packet capture data</li> - -<li><tt>kafka.pcap.ts_granularity</tt> : The granularity of timing used in the timestamps. One of <tt>MILLISECONDS</tt>, <tt>MICROSECONDS</tt>, or <tt>NANOSECONDS</tt> representing milliseconds, microseconds or nanoseconds since the unix epoch (respectively).</li> +<li><tt>kafka.pcap.ts_granularity</tt> : The granularity of timing used in the timestamps. One of <tt>MILLISECONDS</tt>, <tt>MICROSECONDS</tt>, or <tt>NANOSECONDS</tt> representing milliseconds, microseconds or nanoseconds since the unix epoch (respectively).</li> </ul></div> <div class="section"> <h2><a name="Starting_the_Topology"></a>Starting the Topology</h2> -<p>To assist in starting the topology, a utility script which takes no arguments has been created to make this very simple. Simply, execute <tt>$METRON_HOME/bin/start_pcap_topology.sh</tt>.</p></div> +<p>To assist in starting the topology, a utility script which takes no arguments has been created to make this very simple. Simply, execute <tt>$METRON_HOME/bin/start_pcap_topology.sh</tt>.</p></div> <div class="section"> <h2><a name="Utilities"></a>Utilities</h2> <div class="section"> <h3><a name="Inspector_Utility"></a>Inspector Utility</h3> <p>In order to ensure that data can be read back out, a utility, <tt>$METRON_HOME/bin/pcap_inspector.sh</tt> has been created to read portions of the sequence files.</p> -<div class="source"> -<div class="source"> -<pre>usage: PcapInspector +<div> +<div> +<pre class="source">usage: PcapInspector -h,--help Generate Help screen -i,--input <SEQ_FILE> Input sequence file on HDFS -n,--num_packets <N> Number of packets to dump -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h3><a name="Query_Filter_Utility"></a>Query Filter Utility</h3> <p>This tool exposes the two methods for filtering PCAP data via a command line tool:</p> - <ul> - + <li>fixed</li> - <li>query (via Stellar)</li> </ul> -<p>The tool is executed via </p> +<p>The tool is executed via</p> -<div class="source"> -<div class="source"> -<pre>${metron_home}/bin/pcap_query.sh [fixed|query] +<div> +<div> +<pre class="source">${metron_home}/bin/pcap_query.sh [fixed|query] </pre></div></div> + <div class="section"> <h4><a name="Usage"></a>Usage</h4> -<div class="source"> -<div class="source"> -<pre>usage: Fixed filter options +<div> +<div> +<pre class="source">usage: Fixed filter options -bop,--base_output_path <arg> Query result output path. Default is '/tmp' -bp,--base_path <arg> Base PCAP data path. Default is @@ -426,9 +235,9 @@ -st,--start_time <arg> (required) Packet start time range. </pre></div></div> -<div class="source"> -<div class="source"> -<pre>usage: Query filter options +<div> +<div> +<pre class="source">usage: Query filter options -bop,--base_output_path <arg> Query result output path. Default is '/tmp' -bp,--base_path <arg> Base PCAP data path. Default is @@ -444,28 +253,22 @@ -q,--query <arg> Query string to use as a filter -st,--start_time <arg> (required) Packet start time range. </pre></div></div> -<p>The Query filter’s <tt>--query</tt> argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:</p> +<p>The Query filter’s <tt>--query</tt> argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:</p> <ul> - + <li><tt>packet</tt> : The packet data (a <tt>byte[]</tt>)</li> - <li><tt>ip_src_addr</tt> : The source address for the packet (a <tt>String</tt>)</li> - <li><tt>ip_src_port</tt> : The source port for the packet (an <tt>Integer</tt>)</li> - <li><tt>ip_dst_addr</tt> : The destination address for the packet (a <tt>String</tt>)</li> - <li><tt>ip_dst_port</tt> : The destination port for the packet (an <tt>Integer</tt>)</li> </ul></div> <div class="section"> <h4><a name="Binary_Regex"></a>Binary Regex</h4> -<p>Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:</p> - +<p>Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:</p> <ul> - + <li>The <tt>-pf</tt> or <tt>--packet_filter</tt> options for the fixed query filter</li> - <li>The <tt>BYTEARRAY_MATCHER(pattern, data)</tt> Stellar function. The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via the<tt>packet</tt> variable in Stellar.</li> </ul> <p>The format of this regular expression is described <a class="externalLink" href="https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md";>here</a>.</p></div></div></div> @@ -475,9 +278,9 @@ <div class="section"> <h3><a name="pcap.properties_file"></a>pcap.properties file</h3> -<div class="source"> -<div class="source"> -<pre>spout.kafka.topic.pcap=pcap +<div> +<div> +<pre class="source">spout.kafka.topic.pcap=pcap storm.topology.workers=16 kafka.spout.parallelism=128 kafka.pcap.numPackets=1000000000 @@ -485,14 +288,15 @@ kafka.pcap.maxTimeMS=0 hdfs.replication=1 hdfs.sync.every=10000 </pre></div></div> + <p>You’ll notice that the number of kakfa partitions equals the spout parallelism, and this is no coincidence. The ordering guarantees for a partition in Kafka enforces that you may have no more consumers than 1 per topic. Any additional parallelism will leave you with dormant threads consuming resources but performing no additional work. For our cluster with 4 Storm Supervisors, we found 16 workers to provide optimal throughput as well. We were largely IO bound rather than CPU bound with the incoming PCAP data.</p></div> <div class="section"> <h3><a name="remote.yaml"></a>remote.yaml</h3> <p>In the flux file, we introduced the following configuration:</p> -<div class="source"> -<div class="source"> -<pre>name: "pcap" +<div> +<div> +<pre class="source">name: "pcap" config: topology.workers: ${storm.topology.workers} topology.worker.childopts: ${topology.worker.childopts} @@ -615,6 +419,7 @@ spouts: - ref: "writerConfig" </pre></div></div> + <div class="section"> <h4><a name="Flux_Changes_Introduced"></a>Flux Changes Introduced</h4> <div class="section"> @@ -623,9 +428,9 @@ spouts: <div class="section"> <h5><a name="Kafka_configuration"></a>Kafka configuration</h5> -<div class="source"> -<div class="source"> -<pre>poll.timeout.ms +<div> +<div> +<pre class="source">poll.timeout.ms offset.commit.period.ms session.timeout.ms max.uncommitted.offsets @@ -633,44 +438,40 @@ max.poll.interval.ms max.poll.records receive.buffer.bytes max.partition.fetch.bytes -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h5><a name="Writer_Configuration"></a>Writer Configuration</h5> <p>This is a combination of settings for the HDFSWriter (see pcap.properties values above) as well as HDFS.</p> <p><b>HDFS config</b></p> <p>Component config HashMap with the following properties:</p> -<div class="source"> -<div class="source"> -<pre>io.file.buffer.size +<div> +<div> +<pre class="source">io.file.buffer.size dfs.blocksize </pre></div></div> + <p><b>Writer config</b></p> <p>References the HDFS props component specified above.</p> -<div class="source"> -<div class="source"> -<pre> - name: "withHDFSConfig" +<div> +<div> +<pre class="source"> - name: "withHDFSConfig" args: - ref: "hdfsProps" </pre></div></div></div></div></div></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-platform/metron-writer/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-writer/index.html b/site/current-book/metron-platform/metron-writer/index.html index b9c2125..cb1a26f 100644 --- a/site/current-book/metron-platform/metron-writer/index.html +++ b/site/current-book/metron-platform/metron-writer/index.html @@ -1,271 +1,93 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-writer/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Writer</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Writer</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Writer</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-down"></i> - Platform</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"> - <i class="none"></i> - Performance-tuning-guide</a> - </li> - - <li> - - <a href="../../metron-platform/metron-api/index.html" title="Api"> - <i class="none"></i> - Api</a> - </li> - - <li> - - <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> - Common</a> - </li> - - <li> - - <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> - <i class="none"></i> - Data-management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"> - <i class="none"></i> - Elasticsearch</a> - </li> - - <li> - - <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> - <i class="none"></i> - Enrichment</a> - </li> - - <li> - - <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> - <i class="none"></i> - Indexing</a> - </li> - - <li> - - <a href="../../metron-platform/metron-management/index.html" title="Management"> - <i class="none"></i> - Management</a> - </li> - - <li> - - <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> - <i class="icon-chevron-right"></i> - Parsers</a> - </li> - - <li> - - <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> - <i class="none"></i> - Pcap-backend</a> - </li> - - <li class="active"> - - <a href="#"><i class="none"></i>Writer</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> + <ul class="nav nav-list"> + <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> + <li><a href="../../metron-platform/metron-api/index.html" title="Api"><span class="none"></span>Api</a></li> + <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> + <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> + <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> + <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> + <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> + <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> + <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> + <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> + <li class="active"><a href="#"><span class="none"></span>Writer</a></li> + </ul> +</li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <!-- Licensed to the Apache Software Foundation (ASF) under one or more + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 @@ -278,19 +100,21 @@ Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. --><h1>Writer</h1> +limitations under the License. + --> +<h1>Writer</h1> <p><a name="Writer"></a></p> <div class="section"> <h2><a name="Introduction"></a>Introduction</h2> -<p>The writer module provides some utilties for writing to outside components from within Storm. This includes managing bulk writing. An implemention is included for writing to HDFS in this module. Other writers can be found in their own modules.</p></div> +<p>The writer module provides some utilties for writing to outside components from within Storm. This includes managing bulk writing. An implemention is included for writing to HDFS in this module. Other writers can be found in their own modules.</p></div> <div class="section"> <h2><a name="HDFS_Writer"></a>HDFS Writer</h2> -<p>The HDFS writer included here expands on what Storm has in several ways. There’s customization in syncing to HDFS, rotation policy, etc. In addition, the writer allows for users to define output paths based on the fields in the provided JSON message. This can be defined using Stellar.</p> +<p>The HDFS writer included here expands on what Storm has in several ways. There’s customization in syncing to HDFS, rotation policy, etc. In addition, the writer allows for users to define output paths based on the fields in the provided JSON message. This can be defined using Stellar.</p> <p>To manage the output path, a base path argument is provided by the Flux file, with the FileNameFormat as follows</p> -<div class="source"> -<div class="source"> -<pre> - id: "fileNameFormat" +<div> +<div> +<pre class="source"> - id: "fileNameFormat" className: "org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat" configMethods: - name: "withPrefix" @@ -303,33 +127,36 @@ limitations under the License. --><h1>Writer</h1> args: - "/apps/metron/" </pre></div></div> -<p>This means that all output will land in <tt>/apps/metron/</tt>. With no further adjustment, it will be <tt>/apps/metron/<sensor>/</tt>. However, by modifying the sensor’s JSON config, it is possible to provide additional pathing based on the the message itself.</p> + +<p>This means that all output will land in <tt>/apps/metron/</tt>. With no further adjustment, it will be <tt>/apps/metron/<sensor>/</tt>. However, by modifying the sensor’s JSON config, it is possible to provide additional pathing based on the the message itself.</p> <p>E.g.</p> -<div class="source"> -<div class="source"> -<pre>{ +<div> +<div> +<pre class="source">{ "index": "bro", "batchSize": 5, "outputPathFunction": "FORMAT('uid-%s', uid)" } </pre></div></div> + <p>will land data in <tt>/apps/metron/uid-<uid>/</tt>.</p> <p>For example, if the data contains uid’s 1, 3, and 5, there will be 3 output folders in HDFS:</p> -<div class="source"> -<div class="source"> -<pre>/apps/metron/uid-1/ +<div> +<div> +<pre class="source">/apps/metron/uid-1/ /apps/metron/uid-3/ /apps/metron/uid-5/ </pre></div></div> + <p>The Stellar function must return a String, but is not limited to FORMAT functions. Other functions, such as <tt>TO_LOWER</tt>, <tt>TO_UPPER</tt>, etc. are all available for use. Typically, it’s preferable to do nontrivial transformations as part of enrichment and simply reference the output here.</p> <p>If no Stellar function is provided, it will default to putting the sensor in a folder, as above.</p> -<p>A caveat is that the writer will only allow a certain number of files to be created at once. HdfsWriter has a function <tt>withMaxOpenFiles</tt> allowing this to be set. The default is 500. This can be set in Flux:</p> +<p>A caveat is that the writer will only allow a certain number of files to be created at once. HdfsWriter has a function <tt>withMaxOpenFiles</tt> allowing this to be set. The default is 500. This can be set in Flux:</p> -<div class="source"> -<div class="source"> -<pre> - id: "hdfsWriter" +<div> +<div> +<pre class="source"> - id: "hdfsWriter" className: "org.apache.metron.writer.hdfs.HdfsWriter" configMethods: - name: "withFileNameFormat" @@ -341,23 +168,17 @@ limitations under the License. --><h1>Writer</h1> - name: "withMaxOpenFiles" args: 500 </pre></div></div></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
