http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-analytics/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-analytics/index.html b/site/current-book/metron-analytics/index.html index fc4137a..02a9064 100644 --- a/site/current-book/metron-analytics/index.html +++ b/site/current-book/metron-analytics/index.html @@ -1,250 +1,121 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Analytics</title> - <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../css/site.css" /> <link rel="stylesheet" href="../css/print.css" media="print" /> - - - <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Metron Analytics</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Metron Analytics</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li class="active"> - - <a href="#"><i class="icon-chevron-down"></i>Analytics</a> - <ul class="nav nav-list"> - - <li> - - <a href="../metron-analytics/metron-maas-service/index.html" title="Maas-service"> - <i class="none"></i> - Maas-service</a> - </li> - - <li> - - <a href="../metron-analytics/metron-profiler/index.html" title="Profiler"> - <i class="none"></i> - Profiler</a> - </li> - - <li> - - <a href="../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"> - <i class="none"></i> - Profiler-client</a> - </li> - - <li> - - <a href="../metron-analytics/metron-statistics/index.html" title="Statistics"> - <i class="icon-chevron-right"></i> - Statistics</a> - </li> - </ul> - </li> - - <li> - - <a href="../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li class="active"><a href="#"><span class="icon-chevron-down"></span>Analytics</a> + <ul class="nav nav-list"> + <li><a href="../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li> + <li><a href="../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li> + <li><a href="../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li> + <li><a href="../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li> + </ul> +</li> + <li><a href="../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Metron Analytics</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Metron Analytics</h1> <p><a name="Metron_Analytics"></a></p> <p>Metron analytics consists of:</p> - <ul> - + <li>Model-as-a-Service (MAAS) access to Machine Learning services</li> - <li>Profiler and Profiler Client</li> - <li>Statistics</li> </ul> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-analytics/metron-maas-service/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-analytics/metron-maas-service/index.html b/site/current-book/metron-analytics/metron-maas-service/index.html index 0298a29..b47dbea 100644 --- a/site/current-book/metron-analytics/metron-maas-service/index.html +++ b/site/current-book/metron-analytics/metron-maas-service/index.html @@ -1,280 +1,146 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-maas-service/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Model Management Infrastructure</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Model Management Infrastructure</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Model Management Infrastructure</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-down"></i> - Analytics</a> - <ul class="nav nav-list"> - - <li class="active"> - - <a href="#"><i class="none"></i>Maas-service</a> - </li> - - <li> - - <a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"> - <i class="none"></i> - Profiler</a> - </li> - - <li> - - <a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"> - <i class="none"></i> - Profiler-client</a> - </li> - - <li> - - <a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"> - <i class="icon-chevron-right"></i> - Statistics</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a> + <ul class="nav nav-list"> + <li class="active"><a href="#"><span class="none"></span>Maas-service</a></li> + <li><a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li> + <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li> + <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li> + </ul> +</li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Model Management Infrastructure</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Model Management Infrastructure</h1> <p><a name="Model_Management_Infrastructure"></a></p> <div class="section"> <h2><a name="Introduction"></a>Introduction</h2> -<p>One of the main features envisioned and requested is the ability to augment the threat intelligence and enrichment processes with insights derived from machine learning or statistical models. The challenges with this sort of infrastructure are</p> - +<p>One of the main features envisioned and requested is the ability to augment the threat intelligence and enrichment processes with insights derived from machine learning or statistical models. The challenges with this sort of infrastructure are</p> <ul> - + <li>Applying the model may be sufficiently computationally/resource intensive that we need to support scaling via load balancing, which will require service discovery and management.</li> - <li>Models require out of band and frequent training to react to growing threats and new patterns that emerge.</li> - -<li>Models should be language/environment agnostic as much as possible. These should include small-data and big-data libraries and languages.</li> +<li>Models should be language/environment agnostic as much as possible. These should include small-data and big-data libraries and languages.</li> </ul> -<p>To support a high throughput environment that is manageable, it is evident that </p> - +<p>To support a high throughput environment that is manageable, it is evident that</p> <ul> - + <li>Multiple versions of models will need to be exposed</li> - <li>Deployment should happen using Yarn to manage resources</li> - <li>Clients should have new model endpoints pushed to them</li> </ul></div> <div class="section"> <h2><a name="Architecture"></a>Architecture</h2> <p><img src="../../images/maas_arch.png" alt="Architecture" /></p> <p>To support these requirements, the following components have been created:</p> - <ul> - + <li>A Yarn application which will listen for model deployment requests and upon execution, register their endpoints in zookeeper: - <ul> - + <li>Operation type: ADD, REMOVE, LIST</li> - <li>Model Name</li> - <li>Model Version</li> - <li>Memory requirements (in megabytes)</li> - <li>Number of instances</li> - </ul></li> - +</ul> +</li> <li>A command line deployment client which will localize the model payload onto HDFS and submit a model request</li> - <li>A Java client which will interact with zookeeper and receive updates about model state changes (new deployments, removals, etc.)</li> - <li>A series of Stellar functions for interacting with models deployed via the Model as a Service infrastructure.</li> </ul></div> <div class="section"> <h2><a name="maas_service.sh"></a><tt>maas_service.sh</tt></h2> -<p>The <tt>maas_service.sh</tt> script starts the Yarn application which will listen for requests. Right now the queue for the requests is a distributed queue stored in <a class="externalLink" href="http://curator.apache.org/curator-recipes/distributed-queue.html";>zookeeper</a> for convenience.</p> +<p>The <tt>maas_service.sh</tt> script starts the Yarn application which will listen for requests. Right now the queue for the requests is a distributed queue stored in <a class="externalLink" href="http://curator.apache.org/curator-recipes/distributed-queue.html";>zookeeper</a> for convenience.</p> -<div class="source"> -<div class="source"> -<pre>./maas_service.sh +<div> +<div> +<pre class="source">./maas_service.sh usage: MaaSClient -c,--create Flag to indicate whether to create the domain specified with -domain. @@ -311,23 +177,21 @@ usage: MaaSClient given domain -zq,--zk_quorum <arg> Zookeeper Quorum -zr,--zk_root <arg> Zookeeper Root -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h2><a name="maas_deploy.sh"></a><tt>maas_deploy.sh</tt></h2> -<p>The <tt>maas_deploy.sh</tt> script allows users to deploy models and their collateral from their local disk to the cluster. It is assumed that the </p> - +<p>The <tt>maas_deploy.sh</tt> script allows users to deploy models and their collateral from their local disk to the cluster. It is assumed that the</p> <ul> - + <li>Collateral has exactly one <tt>.sh</tt> script capable of starting the endpoint</li> - <li>The model service executable will expose itself as a URL endpoint (e.g. as a REST interface, but not necessarily)</li> - <li>The model service executable will write out to local disk a JSON blob indicating the endpoint (see <a class="externalLink" href="https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a#file-dga-py-L21";>here</a> for an example mock service using Python and Flask).</li> </ul> -<div class="source"> -<div class="source"> -<pre>./maas_deploy.sh +<div> +<div> +<pre class="source">./maas_deploy.sh usage: ModelSubmission -h,--help This screen -hmp,--hdfs_model_path <arg> Model Path (HDFS) @@ -340,92 +204,82 @@ usage: ModelSubmission -v,--version <arg> Model version -zq,--zk_quorum <arg> Zookeeper Quorum -zr,--zk_root <arg> Zookeeper Root -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h2><a name="Kerberos_Support"></a>Kerberos Support</h2> -<p>Model as a service will run on a kerberized cluster (see <a href="../../metron-deployment/vagrant/Kerberos-setup.html">here</a> for instructions for vagrant) with a caveat. The user who submits the service will be the user who executes the models on the cluster. That is to say that user impersonation of models deployed is not done at the moment.</p></div> +<p>Model as a service will run on a kerberized cluster (see <a href="../../metron-deployment/vagrant/Kerberos-setup.html">here</a> for instructions for vagrant) with a caveat. The user who submits the service will be the user who executes the models on the cluster. That is to say that user impersonation of models deployed is not done at the moment.</p></div> <div class="section"> <h2><a name="Stellar_Integration"></a>Stellar Integration</h2> <p>Two Stellar functions have been added to provide the ability to call out to models deployed via Model as a Service. One aimed at recovering a load balanced endpoint of a deployed model given the name and, optionally, the version. The second is aimed at calling that endpoint assuming that it is exposed as a REST endpoint.</p> - <ul> - -<li><tt>MAAS_MODEL_APPLY(endpoint, function?, model_args)</tt> : Returns the output of a model deployed via model which is deployed at endpoint. <tt>endpoint</tt> is a map containing <tt>name</tt>, <tt>version</tt>, <tt>url</tt> for the REST endpoint, <tt>function</tt> is the endpoint path and is optional, and <tt>model_args</tt> is a dictionary of arguments for the model (these become request params).</li> - -<li><tt>MAAS_GET_ENDPOINT(model_name, model_version?)</tt> : Inspects zookeeper and returns a map containing the <tt>name</tt>, <tt>version</tt> and <tt>url</tt> for the model referred to by <tt>model_name</tt> and <tt>model_version</tt>. If <tt>model_version</tt> is not specified, the most current model associated with <tt>model_name</tt> is returned. In the instance where more than one model is deployed, a random one is selected with uniform probability.</li> + +<li><tt>MAAS_MODEL_APPLY(endpoint, function?, model_args)</tt> : Returns the output of a model deployed via model which is deployed at endpoint. <tt>endpoint</tt> is a map containing <tt>name</tt>, <tt>version</tt>, <tt>url</tt> for the REST endpoint, <tt>function</tt> is the endpoint path and is optional, and <tt>model_args</tt> is a dictionary of arguments for the model (these become request params).</li> +<li><tt>MAAS_GET_ENDPOINT(model_name, model_version?)</tt> : Inspects zookeeper and returns a map containing the <tt>name</tt>, <tt>version</tt> and <tt>url</tt> for the model referred to by <tt>model_name</tt> and <tt>model_version</tt>. If <tt>model_version</tt> is not specified, the most current model associated with <tt>model_name</tt> is returned. In the instance where more than one model is deployed, a random one is selected with uniform probability.</li> </ul> <p><a name="Example"></a></p> <h1>Example</h1> -<p>Let’s augment the <tt>squid</tt> proxy sensor to use a model that will determine if the destination host is a domain generating algorithm. For the purposes of demonstration, this algorithm is super simple and is implemented using Python with a REST interface exposed via the Flask python library.</p></div> +<p>Let’s augment the <tt>squid</tt> proxy sensor to use a model that will determine if the destination host is a domain generating algorithm. For the purposes of demonstration, this algorithm is super simple and is implemented using Python with a REST interface exposed via the Flask python library.</p></div> <div class="section"> <h2><a name="Install_Prerequisites_and_Mock_DGA_Service"></a>Install Prerequisites and Mock DGA Service</h2> <p>Now let’s install some prerequisites:</p> - <ul> - + <li>Flask via <tt>yum install python-flask</tt></li> - <li>Jinja2 via <tt>yum install python-jinja2</tt></li> - <li>Squid client via <tt>yum install squid</tt></li> - <li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></li> </ul> <p>Start Squid via <tt>service squid start</tt></p> <p>Now that we have flask and jinja, we can create a mock DGA service to deploy with MaaS:</p> - <ul> - + <li>Download the files in <a class="externalLink" href="https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a";>this</a> gist into the <tt>$HOME/mock_dga</tt> directory</li> - <li>Make <tt>rest.sh</tt> executable via <tt>chmod +x $HOME/mock_dga/rest.sh</tt></li> </ul> -<p>This service will treat <tt>yahoo.com</tt> and <tt>amazon.com</tt> as legit and everything else as malicious. The contract is that the REST service exposes an endpoint <tt>/apply</tt> and returns back JSON maps with a single key <tt>is_malicious</tt> which can be <tt>malicious</tt> or <tt>legit</tt>.</p></div> +<p>This service will treat <tt>yahoo.com</tt> and <tt>amazon.com</tt> as legit and everything else as malicious. The contract is that the REST service exposes an endpoint <tt>/apply</tt> and returns back JSON maps with a single key <tt>is_malicious</tt> which can be <tt>malicious</tt> or <tt>legit</tt>.</p></div> <div class="section"> <h2><a name="Deploy_Mock_DGA_Service_via_MaaS"></a>Deploy Mock DGA Service via MaaS</h2> -<p>The following presumes that you are a logged in as a user who has a home directory in HDFS under <tt>/user/$USER</tt>. If you do not, please create one and ensure the permissions are set appropriate:</p> +<p>The following presumes that you are a logged in as a user who has a home directory in HDFS under <tt>/user/$USER</tt>. If you do not, please create one and ensure the permissions are set appropriate:</p> -<div class="source"> -<div class="source"> -<pre>su - hdfs -c "hadoop fs -mkdir /user/$USER" +<div> +<div> +<pre class="source">su - hdfs -c "hadoop fs -mkdir /user/$USER" su - hdfs -c "hadoop fs -chown $USER:$USER /user/$USER" </pre></div></div> + <p>Or, in the common case for the <tt>metron</tt> user:</p> -<div class="source"> -<div class="source"> -<pre>su - hdfs -c "hadoop fs -mkdir /user/metron" +<div> +<div> +<pre class="source">su - hdfs -c "hadoop fs -mkdir /user/metron" su - hdfs -c "hadoop fs -chown metron:metron /user/metron" </pre></div></div> -<p>Now let’s start MaaS and deploy the Mock DGA Service:</p> +<p>Now let’s start MaaS and deploy the Mock DGA Service:</p> <ul> - + <li>Start MaaS via <tt>$METRON_HOME/bin/maas_service.sh -zq node1:2181</tt></li> - <li>Start one instance of the mock DGA model with 512M of memory via <tt>$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp $HOME/mock_dga -hmp /user/$USER/models -mo ADD -m 512 -n dga -v 1.0 -ni 1</tt></li> - <li>As a sanity check: - <ul> - -<li>Ensure that the model is running via <tt>$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -mo LIST</tt>. You should see <tt>Model dga @ 1.0</tt> be displayed and under that a url such as (but not exactly) <tt>http://node1:36161</tt></li> - + +<li>Ensure that the model is running via <tt>$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -mo LIST</tt>. You should see <tt>Model dga @ 1.0</tt> be displayed and under that a url such as (but not exactly) <tt>http://node1:36161</tt></li> <li>Try to hit the model via curl: <tt>curl 'http://localhost:36161/apply?host=caseystella.com'</tt> and ensure that it returns a JSON map indicating the domain is malicious.</li> - </ul></li> +</ul> +</li> </ul></div> <div class="section"> <h2><a name="Adjust_Configurations_for_Squid_to_Call_Model"></a>Adjust Configurations for Squid to Call Model</h2> <p>Now that we have a deployed model, let’s adjust the configurations for the Squid topology to annotate the messages with the output of the model.</p> - <ul> - + <li>Edit the squid parser configuration at <tt>$METRON_HOME/config/zookeeper/parsers/squid.json</tt> in your favorite text editor and add a new FieldTransformation to indicate a threat alert based on the model (note the addition of <tt>is_malicious</tt> and <tt>is_alert</tt>):</li> </ul> -<div class="source"> -<div class="source"> -<pre>{ +<div> +<div> +<pre class="source">{ "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "parserConfig": { @@ -449,13 +303,13 @@ su - hdfs -c "hadoop fs -chown metron:metron /user/metron" </pre></div></div> <ul> - + <li>Edit the squid enrichment configuration at <tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will not exist, so create a new one) to make the threat triage adjust the level of risk based on the model output:</li> </ul> -<div class="source"> -<div class="source"> -<pre>{ +<div> +<div> +<pre class="source">{ "enrichment" : { "fieldMap": {} }, @@ -475,56 +329,43 @@ su - hdfs -c "hadoop fs -chown metron:metron /user/metron" </pre></div></div> <ul> - + <li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z node1:2181</tt></li> - <li>Make the Squid topic in kafka via <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1</tt></li> </ul></div> <div class="section"> <h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send Data</h2> <p>Now we need to start the topologies and send some data:</p> - <ul> - + <li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid</tt></li> - <li>Generate some data via the squid client: - <ul> - + <li>Generate a legit example: <tt>squidclient http://yahoo.com</tt></li> - <li>Generate a malicious example: <tt>squidclient http://cnn.com</tt></li> - </ul></li> - +</ul> +</li> <li>Send the data to kafka via <tt>cat /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid</tt></li> - <li>Browse the data in elasticsearch via the ES Head plugin @ <a class="externalLink" href="http://node1:9200/_plugin/head/";>http://node1:9200/_plugin/head/</a> and verify that in the squid index you have two documents - <ul> - + <li>One from <tt>yahoo.com</tt> which does not have <tt>is_alert</tt> set and does have <tt>is_malicious</tt> set to <tt>legit</tt></li> - <li>One from <tt>cnn.com</tt> which does have <tt>is_alert</tt> set to <tt>true</tt>, <tt>is_malicious</tt> set to <tt>malicious</tt> and <tt>threat:triage:level</tt> set to 100</li> - </ul></li> +</ul> +</li> </ul></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
