http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-platform/metron-writer/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-writer/index.html b/site/current-book/metron-platform/metron-writer/index.html index ad36a4f..9464a4d 100644 --- a/site/current-book/metron-platform/metron-writer/index.html +++ b/site/current-book/metron-platform/metron-writer/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-06-27 + | Generated by Apache Maven Doxia at 2017-09-15 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Writer</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -96,7 +96,14 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -106,9 +113,9 @@ <li> - <a href="../../metron-docker/index.html" title="Docker"> + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> <i class="none"></i> - Docker</a> + Alerts</a> </li> <li> @@ -124,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -134,17 +141,24 @@ <li> + <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"> + <i class="none"></i> + Performance-tuning-guide</a> + </li> + + <li> + <a href="../../metron-platform/metron-api/index.html" title="Api"> <i class="none"></i> Api</a> </li> - + <li> <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="icon-chevron-right"></i> + <i class="none"></i> Common</a> - </li> + </li> <li> @@ -173,13 +187,13 @@ <i class="none"></i> Management</a> </li> - + <li> <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> - <i class="none"></i> + <i class="icon-chevron-right"></i> Parsers</a> - </li> + </li> <li> @@ -201,6 +215,20 @@ <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> </ul> </li> </ul>
http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-sensors/bro-plugin-kafka/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/bro-plugin-kafka/index.html b/site/current-book/metron-sensors/bro-plugin-kafka/index.html index 6a9654a..3456bc7 100644 --- a/site/current-book/metron-sensors/bro-plugin-kafka/index.html +++ b/site/current-book/metron-sensors/bro-plugin-kafka/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-06-27 + | Generated by Apache Maven Doxia at 2017-09-15 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Logging Bro Output to Kafka</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -96,7 +96,14 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -106,9 +113,9 @@ <li> - <a href="../../metron-docker/index.html" title="Docker"> + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> <i class="none"></i> - Docker</a> + Alerts</a> </li> <li> @@ -124,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -159,6 +166,20 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> </ul> </li> </ul> http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-sensors/fastcapa/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/fastcapa/index.html b/site/current-book/metron-sensors/fastcapa/index.html index 68d74d0..38bae8a 100644 --- a/site/current-book/metron-sensors/fastcapa/index.html +++ b/site/current-book/metron-sensors/fastcapa/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-06-27 + | Generated by Apache Maven Doxia at 2017-09-15 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Fastcapa</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -96,7 +96,14 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -106,9 +113,9 @@ <li> - <a href="../../metron-docker/index.html" title="Docker"> + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> <i class="none"></i> - Docker</a> + Alerts</a> </li> <li> @@ -124,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -159,6 +166,20 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> </ul> </li> </ul> @@ -405,7 +426,7 @@ Network devices using kernel driver <pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz cd librdkafka-0.9.4/ ./configure --prefix=$RDK_PREFIX -make +make make install </pre></div></div></li> @@ -546,27 +567,38 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <tr class="a"> -<td>Burst Size </td> +<td>Receive Burst Size </td> -<td>-b BURST_SIZE </td> +<td>-b RX_BURST_SIZE </td> -<td>Maximum number of packets to receive at one time. </td> +<td>The max number of packets processed by a receive worker. </td> <td>32 </td> </tr> <tr class="b"> +<td>Transmit Burst Size </td> + +<td>-w TX_BURST_SIZE </td> + +<td>The max number of packets processed by a transmit worker. </td> + +<td>256 </td> + </tr> + +<tr class="a"> + <td>Receive Descriptors </td> -<td>-r NB_RX_DESC </td> +<td>-d NB_RX_DESC </td> <td>The number of descriptors for each receive queue (the size of the receive queue.) Limited by the ethernet device in use. </td> <td>1024 </td> </tr> -<tr class="a"> +<tr class="b"> <td>Transmission Ring Size </td> @@ -577,7 +609,7 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <td>2048 </td> </tr> -<tr class="b"> +<tr class="a"> <td>Number Receive Queues </td> @@ -588,7 +620,7 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <td>2 </td> </tr> -<tr class="a"> +<tr class="b"> <td>Kafka Topic </td> @@ -599,7 +631,7 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <td>pcap </td> </tr> -<tr class="b"> +<tr class="a"> <td>Configuration File </td> @@ -610,7 +642,7 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <td> </td> </tr> -<tr class="a"> +<tr class="b"> <td>Stats </td> @@ -627,6 +659,18 @@ $DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0" <div class="source"> <div class="source"> <pre>fastcapa -- -h + +fastcapa [EAL options] -- [APP options] + -p PORT_MASK bitmask of ports to bind [0x01] + -b RX_BURST_SIZE burst size of receive worker [32] + -w TX_BURST_SIZE burst size of transmit worker [256] + -d NB_RX_DESC num of descriptors for receive ring [1024] + -x TX_RING_SIZE size of tx rings (must be a power of 2) [2048] + -q NB_RX_QUEUE num of receive queues for each device [1] + -t KAFKA_TOPIC name of the kafka topic [pcap] + -c KAFKA_CONF file containing configs for kafka client + -s KAFKA_STATS append kafka client stats to a file + -h print this help message </pre></div></div></div> <div class="section"> <h4><a name="Fastcapa-Kafka_Configuration_File"></a>Fastcapa-Kafka Configuration File</h4> @@ -844,7 +888,7 @@ metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092 <pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz cd librdkafka-0.9.4/ ./configure --prefix=$RDK_PREFIX --enable-sasl -make +make make install </pre></div></div></li> @@ -983,7 +1027,7 @@ PMD: rte_enic_pmd: vNIC resources used: wq 1 rq 4 cq 3 intr 0 PANIC in rte_eal_init(): Cannot get hugepage information </pre></div></div> -<p>Solution: This can occur if any process that has been allocated THPs crashes and fails to return the resources. </p> +<p>Solution: This can occur if any process that has been allocated THPs crashes and fails to return the resources.</p> <ul> http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-sensors/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/index.html b/site/current-book/metron-sensors/index.html index e750460..f3ab985 100644 --- a/site/current-book/metron-sensors/index.html +++ b/site/current-book/metron-sensors/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-06-27 + | Generated by Apache Maven Doxia at 2017-09-15 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Sensors</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../index.html" title="Metron"> @@ -96,7 +96,14 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + + <li> + + <a href="../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + <li> <a href="../metron-deployment/index.html" title="Deployment"> @@ -106,9 +113,9 @@ <li> - <a href="../metron-docker/index.html" title="Docker"> + <a href="../metron-interface/metron-alerts/index.html" title="Alerts"> <i class="none"></i> - Docker</a> + Alerts</a> </li> <li> @@ -124,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../metron-platform/index.html" title="Platform"> @@ -159,6 +166,20 @@ </li> </ul> </li> + + <li> + + <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> </ul> </li> </ul> http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-sensors/pycapa/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/pycapa/index.html b/site/current-book/metron-sensors/pycapa/index.html index 4bdd35c..52d8f5a 100644 --- a/site/current-book/metron-sensors/pycapa/index.html +++ b/site/current-book/metron-sensors/pycapa/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-06-27 + | Generated by Apache Maven Doxia at 2017-09-15 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Pycapa</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -96,7 +96,14 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -106,9 +113,9 @@ <li> - <a href="../../metron-docker/index.html" title="Docker"> + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> <i class="none"></i> - Docker</a> + Alerts</a> </li> <li> @@ -124,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -159,6 +166,20 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> </ul> </li> </ul> http://git-wip-us.apache.org/repos/asf/metron/blob/87ff7b73/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html new file mode 100644 index 0000000..d5633cc --- /dev/null +++ b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html @@ -0,0 +1,354 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-15 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Custom Stellar Functions</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Custom Stellar Functions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-down"></i> + Stellar-common</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>3rdPartyStellar</a> + </li> + </ul> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Custom Stellar Functions</h1> +<p><a name="Custom_Stellar_Functions"></a></p> +<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch. </p> +<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p> +<div class="section"> +<h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2> +<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> +<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.1 for the purpose of this demonstration</p> +<div class="section"> +<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3> +<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> +<p>First, we should depend on <tt>metron-common</tt> and we can do that by adjusting the <tt>pom.xml</tt> just created:</p> + +<div class="source"> +<div class="source"> +<pre><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <modelVersion>4.0.0</modelVersion> + + <groupId>com.mycompany.stellar</groupId> + <artifactId>tempus</artifactId> + <version>1.0-SNAPSHOT</version> + <packaging>jar</packaging> + + <name>Stellar Time Functions</name> + <url>http://mycompany.com</url> + + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + </properties> + + <dependencies> + <dependency> + <groupId>org.apache.metron</groupId> + <artifactId>metron-common</artifactId> + <version>0.4.1</version> + <!-- NOTE: We will want to depend on the deployed common on the classpath. --> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <version>3.8.1</version> + <scope>test</scope> + </dependency> + </dependencies> +</project> +</pre></div></div> +<p>Let’s add our implementation in <tt>src/main/java/com/mycompany/stellar/TimeFunctions.java</tt> with the following content:</p> + +<div class="source"> +<div class="source"> +<pre>package com.notmetron.stellar; + +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.ParseException; +import org.apache.metron.stellar.dsl.Stellar; +import org.apache.metron.stellar.dsl.StellarFunction; + +import java.util.List; + +public class TimeFunction { + @Stellar( name="NOW", + description = "Right now!", + params = {}, + returns="Timestamp" + ) + public static class Now implements StellarFunction { + + public Object apply(List<Object> list, Context context) throws ParseException { + return System.currentTimeMillis(); + } + + public void initialize(Context context) { } + + public boolean isInitialized() { + return true; + } + } +} +</pre></div></div> +<p>Now we can build the project via <tt>mvn package</tt> which will create a <tt>target/tempus-1.0-SNAPSHOT.jar</tt> file.</p></div></div> +<div class="section"> +<h2><a name="Install_the_Function"></a>Install the Function</h2> +<p>Now that we have a jar with our custom function, we must make Metron aware of it.</p> +<div class="section"> +<h3><a name="Deploy_the_Jar"></a>Deploy the Jar</h3> +<p>First you need to place the jar in HDFS, if we have it on an access node, one way to do that is:</p> + +<ul> + +<li><tt>hadoop fs -put tempus-1.0-SNAPSHOT.jar /apps/metron/stellar</tt> This presumes that:</li> + +<li>you’ve standardized on <tt>/apps/metron/stellar</tt> as the location for custom jars</li> + +<li>you are running the command from an access node with the <tt>hadoop</tt> command installed</li> + +<li>you are running from a user that has write access to <tt>/apps/metron/stellar</tt></li> +</ul></div> +<div class="section"> +<h3><a name="Set_Global_Config"></a>Set Global Config</h3> +<p>You may not need this if your Metron administrator already has this setup.</p> +<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS. </p> +<p>This property looks like, the following for a vagrant install</p> + +<div class="source"> +<div class="source"> +<pre>{ + "es.clustername": "metron", + "es.ip": "node1", + "es.port": "9300", + "es.date.format": "yyyy.MM.dd.HH", + "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar", +} +</pre></div></div> +<p>The <tt>stellar.function.paths</tt> property takes a comma separated list of URIs or URIs with regex expressions at the end. Also, note path is prefaced by the HDFS default name, which, if you do not know, can be found by executing, <tt>hdfs getconf -confKey fs.default.name</tt>, such as</p> + +<div class="source"> +<div class="source"> +<pre>[root@node1 ~]# hdfs getconf -confKey fs.default.name +hdfs://node1:8020 +</pre></div></div></div> +<div class="section"> +<h3><a name="Use_the_Function"></a>Use the Function</h3> +<p>Now that we have deployed the function, if we want to use it, any running topologies that use Stellar will need to be restarted.</p> +<p>Beyond that, let’s take a look at it in the REPL:</p> + +<div class="source"> +<div class="source"> +<pre>Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, stellar.function.paths=hdfs://node1:8020/apps/metron/stellar/.*.jar, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES} +[Stellar]>>> # Get the help for NOW +[Stellar]>>> ?NOW +Functions loaded, you may refer to functions now... +NOW +Description: Right now! + +Returns: Timestamp +[Stellar]>>> # Try to run the NOW function, which we added: +[Stellar]>>> NOW() +1488400515655 +[Stellar]>>> # Looks like I got a timestamp, success! +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
