[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174654407
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do th

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174652376
  
--- Diff: metron-platform/Performance-tuning-guide.md ---
@@ -422,10 +422,12 @@ modifying the options outlined above, increasing the 
poll timeout, or both.
 
 ## Reference
 
+* [Enrichment 
Performance](metron-platform/metron-enrichment/Performance.md)
--- End diff --

`s/metron-platform/./`


---

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174653285
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do th

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174653386
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do th

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174653589
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do this for all 1 million entries in the data set. Doing this for around 10,000 
records is sufficient.
+
+   ```python
--- End diff --

This code block is broken in the rendered doc.


---

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174653490
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do th

[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r174652892
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Unified Enrichment 
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to 
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance 
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do th

[GitHub] metron-bro-plugin-kafka issue #6: METRON-1469: Kafka Plugin for Bro - Config...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
I'm +1 on this.  @nickwallen anything outstanding on your end?


---

[GitHub] metron pull request #959: METRON-1485 Upgrade vagrant for dev environments

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/959

METRON-1485 Upgrade vagrant for dev environments

## Contributor Comments
It looks like we are going to be forced into upgrading vagrant based on 
some HashiCorp deprecation activities.  See:

https://www.hashicorp.com/blog/terraform-enterprise-saas-has-a-new-address

I'm still digging around for the vagrant commit that fixes this in newer 
versions - I just upgraded to the latest and that fixed it.

Currently, if you don't have centos6 or ubuntu14 locally, you get see 
something like the following:

$ vagrant up
Bringing machine 'node1' up with 'virtualbox' provider...
==> node1: Box 'centos/6' could not be found. Attempting to find and 
install...
node1: Box Provider: virtualbox
node1: Box Version: >= 0
The box 'centos/6' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/centos/6;]
Error: The requested URL returned error: 404 Not Found

## Testing
Testing is underway, just need to upgrade vagrant and spin up the dev 
environment to test.  Review for errors and such.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #959


commit 4ca17d5780729a61615e7ee6bc86e6ddf86c339e
Author: Nick Allen <nick@...>
Date:   2017-11-27T20:29:38Z

METRON-1320 Cannot perform a bare-metal installation

commit b64606997307e233a4ef9741c2ad16e622eac952
Author: Nick Allen <nick@...>
Date:   2017-11-27T20:55:12Z

Cleaning up after the C++ file(s) that we create

commit 551e3084c60346a64591019940

[GitHub] metron issue #948: METRON-1468: Add support for apache/metron-bro-plugin-kaf...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/948
  
I agree, updated the script and readme.


---

[GitHub] metron-bro-plugin-kafka issue #7: METRON-1324: Increment metron-bro-plugin-k...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/7
  
# Testing
## Build and install manually
Some guideline commands to test:

```
mkdir tmp
cd tmp
git clone https://github.com/bro/bro
cd bro
git checkout v2.5.3
git submodule update --recursive --init
./configure && make
cd ..
git clone https://github.com/apache/metron-bro-plugin-kafka
cd metron-bro-plugin-kafka
./configure --bro-dist=../bro && make && sudo make install
bro -N Apache::Kafka
```

## Use `bro-pkg`

```
bro-pkg install metron-bro-plugin-kafka
bro -N Apache::Kafka
```


---

[GitHub] metron pull request #948: METRON-1468: Add support for apache/metron-bro-plu...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/948

METRON-1468: Add support for apache/metron-bro-plugin-kafka to 
prepare-commit

## Contributor Comments
This updates the prepare-commit script to work with 
`apache/metron-bro-plugin-kafka`.

To test, run the `prepare-commit` script and specify `metron`, `bro`, or 
blank when it asks for which repo.  It should use metron by default, but 
properly accounts for the bro repo if you specify it.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh
  ```

- [N/A] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [N/A] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1468

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/948.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #948


commit 4ca17d5780729a61615e7ee6bc86e6ddf86c339e
Author: Nick Allen <nick@...>
Date:   2017-11-27T20:29:38Z

METRON-1320 Cannot perform a bare-metal installation

commit b64606997307e233a4ef9741c2ad16e622eac952
Author: Nick Allen <nick@...>
Date:   2017-11-27T20:55:12Z

Cleaning up after the C++ file(s) that we create

commit 551e3084c60346a6459101994043afa84869ab61
Author: Jon Zeolla <zeolla@...>
Date:   2017-11-29T02:03:54Z

Merge branch 'METRON-1320' of https://github.com/nickwallen/metron into 
METRON-1320

commit 1040681a59907fdedf8784bf19fc025cc9125b1c
Author: Jon Zeolla <zeolla@...>
Date:   2017-12-05T12:56:40Z

Merge branch 'master' of https://github.com/apache/metron

commit e916d3dcbeb744a426559d7e7e47695ebd37748b
Author: Jon Zeolla <zeolla@...>
Date:   2017-12-06T14:00:51Z

Merge branch 'master' of https://github.com/apache/metron

commit a7b5bdfd5652035beb38aed52fd4e1e12e680439
Author: Jon Zeolla <zeolla@...>
Date:   2017-12-07T20:14:46Z

Merge branch 'master' of https://github.com/apache/metron

commit f9af29979385877b90f7940561382c722404eff8
Author: Jon Zeolla <zeolla@...>
Date:   2017-12-08T14:22:24Z

Merge branch 'master' of https://github.com/apache/metron

commit 5c6dc76d9fbdb58274171446bc75

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
It's a part of the `apache/metron` project (of which this is considered a 
component) and uses the open apache JIRA that I linked above.  In order to 
accept PRs we need to have a JIRA.  You should be able to register and submit 
something rather simply, but I also wouldn't mind handling this if you'd 
prefer, just let me know.


---

[GitHub] metron-bro-plugin-kafka pull request #7: METRON-1324: Increment metron-bro-p...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron-bro-plugin-kafka/pull/7

METRON-1324:  Increment metron-bro-plugin-kafka version

We have some changes staged to upgrade the plugin, so we should increment 
the version.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1324

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron-bro-plugin-kafka/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit 41a4ee6c478f59676c5a1a3ce7d5f1e0e1a39b85
Author: Jon Zeolla <zeolla@...>
Date:   2018-03-05T17:45:05Z

METRON-1324:  Increment metron-bro-plugin-kafka version




---

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
This is really coming together.  Is there a 
[JIRA](https://issues.apache.org/jira/browse/METRON-1325?filter=-5=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20order%20by%20priority%20DESC%2Cupdated%20DESC)
 for this?  I poked around for a bit and couldn't find one.  


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
The PR title here doesn't match the 
[JIRA](https://issues.apache.org/jira/projects/METRON/issues/METRON-1450?filter=allopenissues)
 but as long as we account for that during merging, I'm a +1 on this.  Thanks 
@MohanDV


---

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
Okay great, thanks.  I will add to my todo list this week to get this and 
#5 tested and in.  Then we can look at merging #2 and maybe the bool option to 
send all logs.


---

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
Would you mind removing the send all logs by default portion of this?  I 
would love to get this reviewed and in but I feel like that approach could be 
problematic and is better addressed via #2.  That said, please feel free to 
disagree.

Another approach I plan to take (and have a branch somewhere for, just no 
PR yet) is adding a bool to turn "all" logs (except for reporter) on by default.


---

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
Thanks @dcode I'll need to review this more specifically but I don't recall 
using any of your contributions for my other work.  I modeled it after other 
plugins in the old bro-plugins repo.

We have also discussed the default on vs not before and I think we settled 
on default off.  However, even if you wanted to do something default on you 
should exclude reporter.log otherwise you can get in an infinite loop with 
errors.


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
I don't see a PR for METRON-1458, did you mean you opened a 
[JIRA](https://issues.apache.org/jira/projects/METRON/issues/METRON-1458)?  In 
the JIRA for 
[METRON-1450](https://issues.apache.org/jira/projects/METRON/issues/METRON-1450)
 the summary is:
```
Add documentation for splitting the indexing topology into 
random_access_indexing and batch_indexing
```

Based on that description, this would be the ticket to address the indexing 
README.  If you think it's worth splitting this into two JIRAs that's fine with 
me, but I already had 
[METRON-1454](https://issues.apache.org/jira/projects/METRON/issues/METRON-1454)
 open to remediate the main indexing README, but I closed it because the 
description of 
[METRON-1450](https://issues.apache.org/jira/projects/METRON/issues/METRON-1450)
 mirrored what I was looking to accomplish.  Let me know where you'd like to go 
from here, but regardless we should make sure that the PR title here matches 
the JIRA summary.


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
I'm specifically referring to 
[this](https://github.com/MohanDV/metron/blob/4cb99e7186c408159cb41b0e5e8761fdcbfe74d2/metron-platform/metron-indexing/README.md#indexing-architecture)
 which is no longer accurate.


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
Yes


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
Would you also mind updating the main README?


---

[GitHub] metron pull request #930: METRON-1318 updated MacOS instructions and explain...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/930#discussion_r166752581
  
--- Diff: metron-deployment/amazon-ec2/README.md ---
@@ -41,11 +50,13 @@ Any platform that supports these tools is suitable, but 
the following instructio
   /usr/bin/ruby -e "$(curl -fsSL 
https://raw.githubusercontent.com/Homebrew/install/master/install)"
   ```
 
-2. With Homebrew installed, run the following command in a terminal to 
install all of the required tools.
+2. With Homebrew installed, run the following command in a terminal to 
install all of the required tools and dependencies.
 
   ```
-  brew cask install java
-  brew install maven git
+  brew update
--- End diff --

Apologies, you're right - I misread it as `upgrade`.


---

[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/812
  
Thanks @mmiklavc this totally fell off my plate


---

[GitHub] metron pull request #929: METRON-1448: Update SolrWriter to conform to new c...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/929#discussion_r166680167
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
 ---
@@ -33,17 +39,19 @@
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.util.List;
-import java.util.Map;
+import java.lang.invoke.MethodHandles;
+import java.util.*;
--- End diff --

Style police checking in.  No \* please.


---

[GitHub] metron issue #865: METRON-1212 The bundle System and Maven Plugin (Feature B...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/865
  
Ping


---

[GitHub] metron pull request #918: METRON-1436: Manually Install Solr Cloud in Full D...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/918#discussion_r164925885
  
--- Diff: metron-platform/metron-solr/src/main/scripts/install_solr.sh ---
@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This is provided for development purposes
+
+# Full dev env setup script for Solr Cloud 6.6.2
+# - Stops ES and Kibana
+# - Downloads Solr
+# - Installs Solr
+# - Starts Solr Cloud
+
+# Note: for production mode, see 
https://lucene.apache.org/solr/guide/6_6/taking-solr-to-production.html
+
+service kibana stop
+service elasticsearch stop
+
+SOLR_VERSION=6.6.2
+SOLR_USER=solr
+SOLR_SERVICE=$SOLR_USER
+SOLR_VAR_DIR="/var/$SOLR_SERVICE"
+
+# create user if not exists
+solr_uid="`id -u "$SOLR_USER"`"
+if [ $? -ne 0 ]; then
+  echo "Creating new user: $SOLR_USER"
+  adduser --system -U -m --home-dir "$SOLR_VAR_DIR" "$SOLR_USER"
+fi
+cd $SOLR_VAR_DIR
+wget 
http://archive.apache.org/dist/lucene/solr/${SOLR_VERSION}/solr-${SOLR_VERSION}.tgz
+tar zxvf solr-${SOLR_VERSION}.tgz
+chown -R solr:solr solr-${SOLR_VERSION}
--- End diff --

`chown -R $SOLR_USER:$SOLR_USER solr-${SOLR_VERSION}`


---

[GitHub] metron pull request #918: METRON-1436: Manually Install Solr Cloud in Full D...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/918#discussion_r164925950
  
--- Diff: metron-platform/metron-solr/src/main/scripts/install_solr.sh ---
@@ -0,0 +1,62 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This is provided for development purposes
+
+# Full dev env setup script for Solr Cloud 6.6.2
+# - Stops ES and Kibana
+# - Downloads Solr
+# - Installs Solr
+# - Starts Solr Cloud
+
+# Note: for production mode, see 
https://lucene.apache.org/solr/guide/6_6/taking-solr-to-production.html
+
+service kibana stop
+service elasticsearch stop
+
+SOLR_VERSION=6.6.2
+SOLR_USER=solr
+SOLR_SERVICE=$SOLR_USER
+SOLR_VAR_DIR="/var/$SOLR_SERVICE"
+
+# create user if not exists
+solr_uid="`id -u "$SOLR_USER"`"
+if [ $? -ne 0 ]; then
+  echo "Creating new user: $SOLR_USER"
+  adduser --system -U -m --home-dir "$SOLR_VAR_DIR" "$SOLR_USER"
+fi
+cd $SOLR_VAR_DIR
+wget 
http://archive.apache.org/dist/lucene/solr/${SOLR_VERSION}/solr-${SOLR_VERSION}.tgz
+tar zxvf solr-${SOLR_VERSION}.tgz
+chown -R solr:solr solr-${SOLR_VERSION}
+cd solr-${SOLR_VERSION}
+su solr -c "bin/solr -e cloud -noprompt"
--- End diff --

`su $SOLR_USER -c "bin/solr -e cloud -noprompt"`


---

[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/690
  
Perfect, ty.


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r164805799
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1413,6 +1413,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.2-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│   └── stellar
+└── lib
+└── stellar-common-0.4.2-uber.jar
--- End diff --

Optional:  Update to `0.4.3`.  Only mentioning because I saw you updated 
this in the past from `0.4.1` to `0.4.2`.


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r164805617
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│   └── stellar
+└── lib
+└── stellar-common-0.4.1-uber.jar
+```
+
+To run the Stellar Shell run the following from the directory you unpacked 
to:
+
+```bash
+bin/stellar
+```
+
+```bash
+-> % bin/stellar
+Stellar, Go!
+Please note that functions are loading lazily in the background and will 
be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, 
CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, 
DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, 
IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, 
MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, 
SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, 
TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, 
TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, 
URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST
+[Stellar]>>>
+```
+
+By default the shell will have the base Stellar Language commands 
available.  Any jars in the lib directory
+that contain Stellar functions will also be loaded, and their commands 
will be available to shell, as long
+as their dependencies are satisfied.
--- End diff --

I still think that it should be a little bit more explicit that some 
Stellar functions won't be available by default (without needing to explicitly 
outline which).  Something like:
`By default the shell is limited to only the base Stellar common language 
commands.`


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r164805812
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1413,6 +1413,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.2-stand-alone.tar.gz
--- End diff --

Optional:  Update to `0.4.3`.  Only mentioning because I saw you updated 
this in the past from `0.4.1` to `0.4.2`.


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/903
  
+1 to that latest round of naming


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/903
  
@lvets trusty is 14.04.  As far as I'm aware the only newer LTS is 16.04, 
with a new one expected in April.  https://wiki.ubuntu.com/Releases


---

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/902
  
I'm game to merge the `prepare-commit` and `checkout-pr` logic into one 
script; I'm even fine with pushing that to be a follow-on.

Regarding the HTML-encoded JIRA description issue, I'm also fine with this 
being a follow-on.  I will be submitting a PR to support both of the Metron 
repos in this script after it's merged, and can take a stab at the JIRA 
description issue if it's still a problem.


---

[GitHub] metron pull request #902: METRON-1413 Add Metron Commit Tool

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/902#discussion_r162688528
  
--- Diff: metron-dev-support/committer-utils/README.md ---
@@ -0,0 +1,96 @@
+# Committer Tools
+
+This project contains tools to assist Apache Metron project committers.
+
+## Prepare Commit
+
+This script automates the process of merging a pull request into 
`apache/master`.  The script will prompt for the pull request number.  Most of 
the remaining information is automatically extracted from Github or the Apache 
JIRA.
+
+When prompted the `[value in brackets]` is used by default.  To accept the 
default, simply press `enter`.  If you would like to change the default, type 
it in and hit `enter` when done.
+
+In the following example, I enter the pull request number (`80`) when 
prompted.   Using the pull request number, the script can extract most of the 
remaining required information.
--- End diff --

`s/80/897/`


---

[GitHub] metron-bro-plugin-kafka issue #5: METRON-1407: Metron-Bro-Kafka plugin unabl...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/5
  
There we go, yeah that was significantly more delayed than normal.  

@z0r0 Would you mind sending an email to the dev list to get you added as a 
contributor on Jira so we can assign this to you?  Thanks


---

[GitHub] metron-bro-plugin-kafka issue #5: METRON-1407: Metron-Bro-Kafka plugin unabl...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/5
  
Great catch, thanks @z0r0 I will check this out soon.  Not sure why this 
isn't auto linking to https://issues.apache.org/jira/browse/METRON-1407 
though...

I can't assign this JIRA to you yet, I need to get you added as a 
contributor.  I've reached out to @ottobackwards via IRC to see if he can help, 
otherwise someone can shoot an email to the dev@ mailing list for you to be 
added.


---

[GitHub] metron issue #896: METRON-1396: Fix .gitignore files to not ignore themselve...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/896
  
+1 by inspection


---

[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/883
  
Did you mean to tag @nickwallen in that last comment?  I took a quick look 
and it all looks good to me but didn't do any verification that you got all of 
the md files (depending on Travis), and didn't spin up the site-book.


---

[GitHub] metron issue #881: METRON-1071: Create CONTRIBUTING.md

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/881
  
Yup, looks good


---

[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/881#discussion_r158477376
  
--- Diff: CONTRIBUTING.md ---
@@ -0,0 +1,27 @@
+#  How To Contribute
+As an open source project, Metron welcomes contributions of all forms. 
There are several great ways to contribute!
+
+* [Contributing a Code Change](#contributing-a-code-change)
+* Reviewing pull requests on our GitHub page. Check out current open [Pull 
Requests](https://github.com/apache/metron/pulls)
+* Improve our documentation. Our docs are self contained in the project in 
README files. Doc changes is the same process as a code change. See 
[Contributing a Code Change](#contributing-a-code-change)
--- End diff --

`s/is/are/`?


---

[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/881#discussion_r158477818
  
--- Diff: CONTRIBUTING.md ---
@@ -0,0 +1,27 @@
+#  How To Contribute
+As an open source project, Metron welcomes contributions of all forms. 
There are several great ways to contribute!
+
+* [Contributing a Code Change](#contributing-a-code-change)
+* Reviewing pull requests on our GitHub page. Check out current open [Pull 
Requests](https://github.com/apache/metron/pulls)
+* Improve our documentation. Our docs are self contained in the project in 
README files. Doc changes is the same process as a code change. See 
[Contributing a Code Change](#contributing-a-code-change)
+* Contributing to or starting discussions on the mailing lists. Both the 
user and dev lists are great places to give and receive help, or provide 
feedback.
--- End diff --

We should point people to the lists.  I'm mobile so not sure what the 
anchor is on here http://metron.apache.org/community


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157717084
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1352,6 +1352,61 @@ IS_EMAIL
 [Stellar]>>> 
 ```
 
+ %timing
+
+Prints out timing information from the last stellar statement executed. 
The information has the total
+execution time, and also a break down of execution time by stellar 
function.
+
+%timing supports filter on the timing output.  If you pass one or more 
strings to %timing, only
+execution times those each of those tags will show.  Any functions that do 
not have ALL the tags passed will not display
--- End diff --

Should this say 'times with each of those'?


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157385625
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

That makes sense, just looking to limit the effect here.


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157376617
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -346,6 +348,18 @@ private void handleMagicFunctions(String[] expression) 
{
 writeLine(functions);
   }
 
+  /**
+   * Handle a magice %timing. Returns the results of the last timing 
operation
--- End diff --

`s/magice/magic/`


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157376611
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

Do you have any metrics regarding the overhead of this?  Instead of doing 
this for all stellar commands, would it make sense to implement a TIME() 
steller function that could wrap all other commands, similar to how it 
functions on *nix boxes?


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157303681
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -20,7 +20,9 @@ Apache Metron Bundles Maven Plugin helps to build Bundles 
Archives to support th
 
 - [Requirements](#requirements)
 - [Building](#building)
-- [Getting Stared](#getting_started)
+- [Getting Stared](#getting-started)
--- End diff --

`s/Stared/Started/`


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196243
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
+- [Getting Help](#getting-help)
+- [License](#license)
+
+## Requirements
+* JDK 1.7 or higher
+* Apache Maven 3.1.0 or higher
+
+## Building 
+
+Building the bundles-maven-plugin module should be rare since it will be 
released infrequently compared to
+the main 'metron' code tree.
+
+- Build with `mvn clean install`
+- Presuming you need to make use of changes to the bundles-maven-plugin 
module, you should next
+  go to the [metron](../metron) directory and follow its instructions. 
+
+## Getting Started
+
+While it is most likely
+that a maven archetype is being utilized to create bundles, as part of a 
toolkit etc, you may want to create on manually, or may need to create a 
project for use in an archetype.
+
+The plugin is utilized by setting the packaging of a maven module to 
'bundle'.
+
+```xml
+bundle
+```
+
+This means that when you package this module, any of it's non-provided 
dependencies will be packaged into the produced bundle ( and all of their 
non-provided dependencies as well).
+Since a library may not always be distributed as part of a bundle with all 
it's dependencies, the bundle module
+shall be a separate module from the actual classes and dependencies to be 
bundled.
+
+A very simple example layout for a project that utilizes bundles would be:
+
+```bash
+├── README.md
+├── pom.xml
+├── testapp
+│   ├── pom.xml
+│   ├── src
+│   │   ├── main
+│   │   │   └── java
+│   │   │   └── org
+│   │   │   └── apache
+│   │   │   └── test
+│   │   │   └── App.java
+│   │   └── test
+│   │   └── java
+│   │   └── org
+│   │   └── apache
+│   │   └── test
+│   │   └── AppTest.java
+└── testappbundle
+├── pom.xml
+```
+Where testappbundle is the bundle module that creates a bundle of testapp, 
and contains the following pom.xml:
+```xml
+
+http://maven.apache.org/POM/4.0.0;
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+test.bundles.plugin
+org.apache.test
+1.0-SNAPSHOT
+  
+  4.0.0
+
+  test.app.bundle
+
+  
+  bundle
+  
+  
+  
+
+  org.apache.test
+  test.app
+  1.0-SNAPSHOT
+
+  
+
+  
+  
+
+  
+
+  org.apache.metron
+  bundles-maven-plugin
+  0.4.2
+  true
+  
+  
+
+  
+
+
+  
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+true
+  
+
+  
+
+```
+When the module is packaged, it packages all of it's  non-provided 
dependencies into the bundles /bundled-dependencies directory.
+Thus, to create a bundle of a module's jar and that jar's non-provided 
dependencies, you add that module to your
+bundle modules dependencies.  You can unzip and examine the bundle in the 
target directory, and verify 
+it's contents, which should be similar to :
+
+```bash
+-> % tree .
+.
+└── META-INF
+├── MANIFEST.MF
+├── bundled-dependencies
+│   ├── log4j-1.2.17.jar
+│   ├── metron-common-0.4.1.jar
+│   ├── slf4j-api-1.7.7.jar
+│   ├── slf4j-log4j12-1.7.7.jar
+│   └── test.app-1.0-SNAPSHOT.jar
+└── maven
+└── org.apache.test
+└── test.app.bundle
+├── pom.properties
+└── pom.xml
+```
+
+This reflects the testapp project, which has these dependencies :
+
+```xml
+
+
+  org.apache.metron
+  metron-common
+  0.4.1
+
+

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196009
  
--- Diff: 
metron-bundles/bundles-lib/src/main/java/org/apache/metron/bundles/VfsBundleClassLoaderResource.java
 ---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
--- End diff --

I think there's an extraneous space before http in this comment block.


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196275
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
+- [Getting Help](#getting-help)
+- [License](#license)
+
+## Requirements
+* JDK 1.7 or higher
+* Apache Maven 3.1.0 or higher
+
+## Building 
+
+Building the bundles-maven-plugin module should be rare since it will be 
released infrequently compared to
+the main 'metron' code tree.
+
+- Build with `mvn clean install`
+- Presuming you need to make use of changes to the bundles-maven-plugin 
module, you should next
+  go to the [metron](../metron) directory and follow its instructions. 
+
+## Getting Started
+
+While it is most likely
+that a maven archetype is being utilized to create bundles, as part of a 
toolkit etc, you may want to create on manually, or may need to create a 
project for use in an archetype.
+
+The plugin is utilized by setting the packaging of a maven module to 
'bundle'.
+
+```xml
+bundle
+```
+
+This means that when you package this module, any of it's non-provided 
dependencies will be packaged into the produced bundle ( and all of their 
non-provided dependencies as well).
+Since a library may not always be distributed as part of a bundle with all 
it's dependencies, the bundle module
+shall be a separate module from the actual classes and dependencies to be 
bundled.
+
+A very simple example layout for a project that utilizes bundles would be:
+
+```bash
+├── README.md
+├── pom.xml
+├── testapp
+│   ├── pom.xml
+│   ├── src
+│   │   ├── main
+│   │   │   └── java
+│   │   │   └── org
+│   │   │   └── apache
+│   │   │   └── test
+│   │   │   └── App.java
+│   │   └── test
+│   │   └── java
+│   │   └── org
+│   │   └── apache
+│   │   └── test
+│   │   └── AppTest.java
+└── testappbundle
+├── pom.xml
+```
+Where testappbundle is the bundle module that creates a bundle of testapp, 
and contains the following pom.xml:
+```xml
+
+http://maven.apache.org/POM/4.0.0;
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+test.bundles.plugin
+org.apache.test
+1.0-SNAPSHOT
+  
+  4.0.0
+
+  test.app.bundle
+
+  
+  bundle
+  
+  
+  
+
+  org.apache.test
+  test.app
+  1.0-SNAPSHOT
+
+  
+
+  
+  
+
+  
+
+  org.apache.metron
+  bundles-maven-plugin
+  0.4.2
+  true
+  
+  
+
+  
+
+
+  
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+true
+  
+
+  
+
+```
+When the module is packaged, it packages all of it's  non-provided 
dependencies into the bundles /bundled-dependencies directory.
+Thus, to create a bundle of a module's jar and that jar's non-provided 
dependencies, you add that module to your
+bundle modules dependencies.  You can unzip and examine the bundle in the 
target directory, and verify 
+it's contents, which should be similar to :
+
+```bash
+-> % tree .
+.
+└── META-INF
+├── MANIFEST.MF
+├── bundled-dependencies
+│   ├── log4j-1.2.17.jar
+│   ├── metron-common-0.4.1.jar
+│   ├── slf4j-api-1.7.7.jar
+│   ├── slf4j-log4j12-1.7.7.jar
+│   └── test.app-1.0-SNAPSHOT.jar
+└── maven
+└── org.apache.test
+└── test.app.bundle
+├── pom.properties
+└── pom.xml
+```
+
+This reflects the testapp project, which has these dependencies :
+
+```xml
+
+
+  org.apache.metron
+  metron-common
+  0.4.1
+
+

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157195764
  
--- Diff: metron-bundles/bundles-lib/README.md ---
@@ -0,0 +1,213 @@
+# Apache Metron Bundles
+
+Apache Metron Bundles and this documentation are a derivative of the 
[Apache Nifi](http://www.nifi.apache.org) 
[NARs](http://nifi.apache.org/developer-guide.html).
--- End diff --

Change this to
Apache Metron Bundles and this documentation are a derivative of the 
\[Apache Nifi]\(https://nifi.apache.org) 
\[NARs\](https://nifi.apache.org/developer-guide.html).


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196527
  
--- Diff: metron-bundles/bundles-lib/pom.xml ---
@@ -0,0 +1,185 @@
+
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.metron
+metron-bundles
+0.4.2
+
+
+bundles-lib
+jar
+
+
+
+The Apache Software License, Version 2.0
+http://www.apache.org/licenses/LICENSE-2.0.txt
--- End diff --

`s/http/https/`


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196342
  
--- Diff: bundles-maven-plugin/pom.xml ---
@@ -0,0 +1,328 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache
+apache
+17
+
+
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+maven-plugin
+Apache Metron Bundles Maven Plugin
+http://metron.apache.org
--- End diff --

`s/http/https/`


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196122
  
--- Diff: bundles-maven-plugin/NOTICE ---
@@ -0,0 +1,8 @@
+Apache NiFi
+Copyright 2014-2017 The Apache Software Foundation
+
+Apache Metron
+Copyright 2015-2017 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
--- End diff --

`s/http/https`?


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196378
  
--- Diff: bundles-maven-plugin/pom.xml ---
@@ -0,0 +1,328 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache
+apache
+17
+
+
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+maven-plugin
+Apache Metron Bundles Maven Plugin
+http://metron.apache.org
+
+Apache Metron Project
+http://metron.apache.org/
--- End diff --

`s/http/https/`


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157197191
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
+- [Getting Help](#getting-help)
+- [License](#license)
+
+## Requirements
+* JDK 1.7 or higher
+* Apache Maven 3.1.0 or higher
+
+## Building 
+
+Building the bundles-maven-plugin module should be rare since it will be 
released infrequently compared to
+the main 'metron' code tree.
+
+- Build with `mvn clean install`
+- Presuming you need to make use of changes to the bundles-maven-plugin 
module, you should next
+  go to the [metron](../metron) directory and follow its instructions. 
--- End diff --

Broken link.  Not positive offhand how to fix in a way that also works in 
the site-book, I need to take another look later.  Just noting for now.


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157196478
  
--- Diff: bundles-maven-plugin/pom.xml ---
@@ -0,0 +1,328 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache
+apache
+17
+
+
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+maven-plugin
+Apache Metron Bundles Maven Plugin
+http://metron.apache.org
+
+Apache Metron Project
+http://metron.apache.org/
+
+
+
+Apache License, Version 2.0
+http://www.apache.org/licenses/LICENSE-2.0
--- End diff --

`s/http/https/`


---

[GitHub] metron pull request #865: METRON-1212 The bundle System and Maven Plugin (Fe...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157194648
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
--- End diff --

Change to

\[Getting Started\](#getting-started)

then insert below

\[Quickstart\](#quickstart)
\[Settings and configuration\](#settings-and-configuration)


---

[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/812
  
Sorry, I'm not at a computer right now but if anybody wants to push this 
before our release candidate is cut feel free 


---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
Merged master, ran tests successfully, built site-books and did some 
clicking around.  Going to merge.


---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/849
  
Instead of trying to explaining my position, I threw together a quick PR to 
illustrate my thoughts.  Feel free to take another approach.  I tested it with 
`g++` configured, unconfigured, and not in my PATH on macOS, as well as on a 
CentOS 7 with and without gcc installed, and an Ubuntu 14.04 box with an old 
version of gcc installed (not C++11 compliant).  All worked as expected.


---

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/847
  
Thanks Otto.  Leaving this one open for now, until things get sorted on the 
mailing list.


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/848
  
Ahh sorry, missed that was in the docs as well.  Ok, looks good


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/848
  
+1, this is awesome work thanks Otto.  Tested via
```
cd metron-deployment/packaging/docker/ansible-docker/
docker build -t ansible-docker:2.0.0.2 .
docker run -it -v /Users/jzeolla/metron-pr848:/root/metron 
ansible-docker:2.0.0.2 bash
cd /root/metron
mvn clean package -DskipTests # Success
mvn clean install && cd metron-deployment && mvn package -P build-rpms # 
Success
exit
cd ../../../vagrant/full-dev-platform/
vagrant --ansible-skip-tags="build,quick_dev" up # Success
```


---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
@cestella I know you were out recently, just wanted to bring this one to 
the top of your inbox.  Would like to have this in the upcoming release, but 
also want to get your input.


---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/848
  
Tested via
```
cd /Users/jzeolla/metron-pr848
docker run -it -v /Users/jzeolla/metron-pr848:/root/metron 
ansible-docker:2.0.0.2 bash
cd /root/metron
mvn clean package -DskipTests # Failure
```

Ran into
```
[ERROR] npm ERR! Linux 4.9.49-moby
[ERROR] npm ERR! argv 
"/root/metron/metron-interface/metron-config/node/node" 
"/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
 "run" "build"
[ERROR] npm ERR! node v6.2.0
[ERROR] npm ERR! npm  v3.8.9
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! metron-management-ui@0.4.2 build: 
`./node_modules/angular-cli/bin/ng build -prod`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR!
[ERROR] npm ERR! Failed at the metron-management-ui@0.4.2 build script 
'./node_modules/angular-cli/bin/ng build -prod'.
[ERROR] npm ERR! Make sure you have the latest version of node.js and npm 
installed.
[ERROR] npm ERR! If you do, this is most likely a problem with the 
metron-management-ui package,
[ERROR] npm ERR! not with npm itself.
[ERROR] npm ERR! Tell the author that this fails on your system:
[ERROR] npm ERR! ./node_modules/angular-cli/bin/ng build -prod
[ERROR] npm ERR! You can get information on how to open an issue for this 
project with:
[ERROR] npm ERR! npm bugs metron-management-ui
[ERROR] npm ERR! Or if that isn't available, you can get their info via:
[ERROR] npm ERR! npm owner ls metron-management-ui
[ERROR] npm ERR! There is likely additional logging output above.
[ERROR]
[ERROR] npm ERR! Please include the following file with any support request:
[ERROR] npm ERR! 
/root/metron/metron-interface/metron-config/npm-debug.log
[INFO] 

[INFO] Reactor Summary:
[INFO]
[INFO] Metron . SUCCESS [ 
18.059 s]
[INFO] metron-stellar . SUCCESS [  
7.965 s]
[INFO] stellar-common . SUCCESS [01:27 
min]
[INFO] metron-analytics ... SUCCESS [  
0.012 s]
[INFO] metron-maas-common . SUCCESS [ 
10.648 s]
[INFO] metron-platform  SUCCESS [  
0.031 s]
[INFO] metron-zookeeper ... SUCCESS [  
1.542 s]
[INFO] metron-test-utilities .. SUCCESS [ 
42.995 s]
[INFO] metron-integration-test  SUCCESS [ 
40.759 s]
[INFO] metron-maas-service  SUCCESS [ 
10.431 s]
[INFO] metron-common .. SUCCESS [ 
46.535 s]
[INFO] metron-statistics .. SUCCESS [ 
22.078 s]
[INFO] metron-writer .. SUCCESS [ 
33.132 s]
[INFO] metron-storm-kafka-override  SUCCESS [  
4.151 s]
[INFO] metron-storm-kafka . SUCCESS [  
2.083 s]
[INFO] metron-hbase ... SUCCESS [ 
16.480 s]
[INFO] metron-profiler-common . SUCCESS [  
5.086 s]
[INFO] metron-profiler-client . SUCCESS [ 
26.173 s]
[INFO] metron-profiler  SUCCESS [ 
56.438 s]
[INFO] metron-hbase-client  SUCCESS [ 
16.283 s]
[INFO] metron-enrichment .. SUCCESS [01:01 
min]
[INFO] metron-indexing  SUCCESS [ 
22.402 s]
[INFO] metron-solr  SUCCESS [ 
56.301 s]
[INFO] metron-pcap  SUCCESS [  
5.602 s]
[INFO] metron-parsers . SUCCESS [01:09 
min]
[INFO] metron-pcap-backend  SUCCESS [ 
35.876 s]
[INFO] metron-data-management . SUCCESS [01:46 
min]
[INFO] metron-api . SUCCESS [ 
57.116 s]
[INFO] metron-management .. SUCCESS [ 
14.077 s]
[INFO] elasticsearch-shaded ... SUCCESS [ 
11.380 s]
[INFO] metron-elasticsearch ... SUCCESS [01:02 
min]
[INFO] metron-deployment .. SUCCESS [  
0.006 s]
[INFO] Metron Ambari Management Pack .. SUCCESS [  
4.145 s]
[INFO] metron-contrib . SUCCESS [  
0.015 s]
[INFO] metron-docker .. SUCCESS 

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153327669
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/README.md ---
@@ -1,17 +1,34 @@
 # Overview
-The Metron ansible-docker container is provided in an effort reduce the 
installation burden of deploying Metron in a live envirionment.
-It is provisioned with software required to sucessfully run the deployment 
scripts.
+The Metron ansible-docker container is provided in an effort reduce the 
installation burden of building Metron.
+It may also be used to deploy Metron in a live environment.
+It is provisioned with software required to sucessfully build metron run 
the deployment scripts.
--- End diff --

`s/metron run/metron and run/`


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153326885
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile ---
@@ -14,13 +14,18 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-FROM centos:centos6
+FROM centos:centos6.9
 MAINTAINER Apache Metron
 
 RUN yum install -y tar
 RUN yum install -y wget
+# base development tools required
 RUN yum groupinstall -y "Development tools"
+# newer cpp 11 support required for building node modules
+RUN wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O 
/etc/yum.repos.d/devtools-2.repo
--- End diff --

Please upgrade to devtoolset-4.  
```
yum -y install centos-release-scl
yum -y install devtoolset-4-gcc devtoolset-4-gcc-c++ devtoolset-4-binutils
```
or similar.


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153326917
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile ---
@@ -33,18 +38,28 @@ RUN tar xvf setuptools-11.3.tar.gz
 WORKDIR /usr/src/setuptools-11.3
 RUN python2.7 setup.py install
 RUN easy_install-2.7 pip
+# install ansible and set the configuration var
 RUN pip2.7 install ansible==2.0.0.2
 RUN pip2.7 install boto
 COPY ansible.cfg /root/
 ENV ANSIBLE_CONFIG /root/ansible.cfg
+# java
 RUN yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
 RUN yum install -y which
 RUN yum install -y nss
 WORKDIR /usr/src
+# setup maven
 RUN wget 
http://apache.cs.utah.edu/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
 RUN tar xzvf apache-maven-3.3.9-bin.tar.gz
 RUN mv apache-maven-3.3.9 /opt/maven
 RUN ln -s /opt/maven/bin/mvn /usr/bin/mvn
-RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint 
&& yum clean all
+# install rpm tools required to build rpms
+RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint 
make && yum clean all
+# create a .bashrc for root, enabling the cpp 11 toolset
+RUN touch /root/.bashrc \
+ && cat '/opt/rh/devtoolset-2/enable' >> /root/.bashrc
--- End diff --

Same as above


---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152850661
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

Ok I'm convinced, I guess my posture on this is more aggressive than most.  
I will adjust


---

[GitHub] metron pull request #847: METRON-1313: Update metron-deployment to use bro-p...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/847#discussion_r152715716
  
--- Diff: metron-deployment/roles/bro/tasks/metron-bro-plugin-kafka.yml ---
@@ -15,36 +15,34 @@
 #  limitations under the License.
 #
 ---
-- name: Clone metron-bro-plugin-kafka into /tmp
-  git:
-repo: https://github.com/apache/metron-bro-plugin-kafka
-version: master
-dest: /tmp/metron-bro-plugin-kafka
+- name: Install bro-pkg
+  environment:
+LD_LIBRARY_PATH: "{{ python27_lib }}"
+  command: "{{ python27_bin }}/pip install bro-pkg"
+
+- name: Configure bro-pkg
+  environment:
+PATH: "{{ git29_bin }}:{{ bro_bin }}:{{ ansible_env.PATH }}"
+LD_LIBRARY_PATH: "{{ python27_lib }}"
+  command: "{{ python27_bin }}/bro-pkg autoconfig"
 
-- name: Compile and install the plugin
-  shell: "{{ item }}"
+- name: Install the metron-bro-plugin-kafka package
   environment:
+PATH: "{{ git29_bin }}:{{ bro_bin }}:{{ ansible_env.PATH }}"
+LD_LIBRARY_PATH: "{{ httpd24_lib }}:{{ python27_lib }}"
 CXX: /opt/rh/devtoolset-4/root/usr/bin/g++
 CC: /opt/rh/devtoolset-4/root/usr/bin/gcc
-  args:
-chdir: "/tmp/metron-bro-plugin-kafka"
-creates: "{{ bro_home }}/lib/bro/plugins/BRO_KAFKA"
-  with_items:
-- rm -rf build/
-- "./configure --bro-dist=/tmp/bro-{{ bro_version }} --install-root={{ 
bro_home }}/lib/bro/plugins/ --with-librdkafka={{ librdkafka_home }}"
    -- make
-- make install
+  command: "{{ python27_bin }}/bro-pkg install 
https://github.com/jonzeolla/metron-bro-plugin-kafka --force --version {{ 
metron_bro_plugin_kafka_version }}"
--- End diff --

See my Contributor Comments for why this is 
`https://github.com/jonzeolla/metron-bro-plugin-kafka` instead of 
`apache/metron-bro-plugin-kafka`.  The reason why it won't need to be a full 
URL when it moves to `apache/` is because of 
[this](https://github.com/bro/packages/blob/master/apache/bro-pkg.index).


---

[GitHub] metron pull request #847: METRON-1313: Update metron-deployment to use bro-p...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/847

METRON-1313: Update metron-deployment to use bro-pkg to install the kafka 
plugin

## Contributor Comments
This PR requires #844.

For testing, this PR currently references my fork of 
metron-bro-plugin-kafka, as it is leveraging a [git 
tag](http://bro-package-manager.readthedocs.io/en/stable/package.html?highlight=tag#package-versioning)
 that I didn't want to push without a review.  The merge of this will need to 
be coordinated with apache/metron-bro-plugin-kafka#4, and a corollary 0.1 tag 
against HEAD.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [X] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1313

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/847.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #847






---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152674806
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

I was talking to a few people in the bro community about this and I'm 
hearing that people mostly prefer an unset send_logs to send nothing.  What are 
your thoughts on that?  I would prefer to send all by default, but it's not a 
huge deal to go either way for me.


---

[GitHub] metron issue #846: METRON-1328: Enhance platform-info.sh script to check if ...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/846
  
+1 tested on my mac with and without the daemon running.


---

[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/844
  
Okay, so I spun up master, pushed my template via `curl`, and then ran the 
above commands to confirm backward compatibility with the template on bro 
2.4.x.  The only change to my steps that I had to make was to remove the new 
`RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG` logs from `logs_to_send`, 
but that was expected and I encountered no other issues.


---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152085762
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

Actually, wait, sorry.  If `|Kafka::logs_to_send| > 0` is removed, this 
doesn't send when `logs_to_send` is unset.  Re-adding this.


---

[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/844
  
[METRON-1322](https://issues.apache.org/jira/browse/METRON-1322) for your 
PCAP feature request.

Also, I totally agree with your documentation notes.  Cleaning this up has 
been on my to-do list for a while.


---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152075056
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

Yeah, that's valid, I have removed the check and simplify.

Yeah, I would prefer a default 'send everything' policy when someone loads 
the package, as long as it's otherwise configured.  That said, it will require 
a bit of Metron testing to make sure that it can handle that.  We don't 
currently handle some of the less interesting logs that are on by default, like 
packet filter or loaded scripts.


---

[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/844
  
Thanks @ottobackwards 

While I feel like the ES template documentation is good enough for now, I 
really want to investigate something cleaner, probably via 777 but also 
potentially by splitting up indexes as discussed in 
[METRON-1010](https://issues.apache.org/jira/browse/METRON-1010?filter=-2).


---

[GitHub] metron pull request #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/844#discussion_r152067166
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
 ---
@@ -1133,6 +1133,233 @@ public void testKnownDevicesBroMessage() throws 
ParseException {
 }
 
/**
--- End diff --

Fixed with my latest commit - thanks.


---

[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/844
  
There is no requirement to upgrade bro with this change. All old fields and 
logs are still supported, this simply adds support for the new fields in 
existing logs or new logs altogether to be supported.  You may notice that I 
removed capture password, as it is an internal field and never exposed to logs. 
It should not have been in there in the first place.


---

[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/844
  
I'm going to see if I can find some time today to fix the tests, but this 
is ready for review otherwise.  Full-dev worked as expected for me.


---

[GitHub] metron pull request #844: METRON-1088: Upgrade bro to 2.5.2

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/844

METRON-1088:  Upgrade bro to 2.5.2

## Contributor Comments
This upgrades bro to 2.5.2, and natively provides support for many of the 
fields from a default bro 2.5.2 install.  Specifically, this attempts to 
provide initial support the default-on fields of the following logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)
 - 
[RFB](https://www.bro.org/sphinx-git/scripts/base/protocols/rfb/main.bro.html#type-RFB::Info)
 - 
[Stats](https://www.bro.org/sphinx/scripts/policy/misc/stats.bro.html#type-Stats::Info)
 - 
[CaptureLoss](https://www.bro.org/sphinx/scripts/policy/misc/capture-loss.bro.html#type-CaptureLoss::Info)
 - 
[SIP](https://www.bro.org/sphinx/scripts/base/protocols/sip/main.bro.html#type-SIP::Info)

Still outstanding:
 - [ ] Unit/integration tests.

# Testing
1.  Create a working directory and pull in this PR (selfishly using my 
branch of `checkout-pr` from 
[metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin)
 to test some recent updates)
```
# mkdir ~/tmp
# git clone https://github.com/jonzeolla/metron-commit-stuff 
~/tmp/metron-commit-stuff
# cd ~/tmp/metron-commit-stuff
# git checkout support-bro-plugin
# cd
# ~/tmp/metron-commit-stuff/checkout-pr 844
Please select a repository:
  1) metron
  2) metron-bro-plugin-kafka
Selection [metron]: metron

# cd metron-pr844
# sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
# cd metron-deployment/vagrant/full-dev-platform/
# vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark screen
```
1.  Configure kafka in local.bro so all of the currently supported bro logs 
are being sent.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, 
Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, 
SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG, 
Stats::LOG, CaptureLoss::LOG, SIP::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
screen
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Ctrl+A c to make a new screen window
# Use Ctrl+A n to cycle through screen sessions
# In your new 

[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/827
  
I didn't get myself intimately familiar with this PR, but I wanted to 
mention that assuming two fields with the same name but different types between 
indexes are not the same may not always hold.  For instance, in bro there are a 
couple of field names that overlap between bro logs, and so the type in the bro 
index needs to be one that can commonly work (usually string), but if there was 
no overlap it would be a different type.  
[Example](https://github.com/JonZeolla/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template#L131-L134).
  Not even positive this is relevant (especially given 
`getCommonColumnMetadata` may just get removed), but in case it is I wanted to 
mention it.


---

[GitHub] metron-bro-plugin-kafka pull request #3: METRON-813: Migrate metron-bro-plug...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron-bro-plugin-kafka/pull/3

METRON-813: Migrate metron-bro-plugin-kafka to be a bro package

This should turn this repo into a bro package containing a bro plugin.

# Testing
The below testing plan assumes CentOS/RHEL, but can be tweaked to work on 
most mainstream linux distros.


1.  Install [Kafka 
0.10.0.1](https://kafka.apache.org/0101/documentation.html#quickstart), 
[Zookeeper 
3.4.6](https://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html) (The same 
versions from [HDP 
2.5.5](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_release-notes/content/ch01s01.html)),
 and any package dependancies for testing.
```
# cd
# yum -y install java screen
# wget 
https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz{,.sha1}
# if [[ "$(sha1sum zookeeper-3.4.6.tar.gz)" == "$(cat 
zookeeper-3.4.6.tar.gz.sha1)" ]]; then tar -xvf zookeeper-3.4.6.tar.gz; else 
echo "sha1 sums do not match"; fi
# cd zookeeper-3.4.6
# cp conf/zoo_sample.cfg conf/zoo.cfg
# bin/zkServer.sh start
# cd
# wget 
https://mirrors.sonic.net/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz
# wget 
https://dist.apache.org/repos/dist/release/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz.md5
# # Compare MD5s using md5sum
# tar -xvf kafka_2.10-0.10.0.1.tgz
# cd kafka_2.10-0.10.0.1
# bin/kafka-server-start.sh config/server.properties &
# bin/kafka-topics.sh --create --zookeeper localhost:2181 
--replication-factor 1 --partitions 1 --topic bro
```
1.  [Install bro 2.5.2](https://www.bro.org/sphinx/install/install.html) 
and 
[bro-pkg](http://bro-package-manager.readthedocs.io/en/stable/quickstart.html#installation).
  Make sure you are running at least bro 2.5 and bro-pkg 1.2.0, and configure 
bro-pkg properly.
```
# export PATH=$PATH:/usr/local/bro/bin
# bro --version
bro version 2.5.2
# bro-pkg --version
bro-pkg 1.2.2
# bro-pkg autoconfig
```
1.  Create a working directory and pull in this PR (selfishly using my 
branch of `checkout-pr` from 
[metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin)
 to test some recent updates)
```
    # git clone https://github.com/jonzeolla/metron-commit-stuff 
~/metron-commit-stuff
# cd ~/metron-commit-stuff
# git checkout support-bro-plugin
# cd
# ~/metron-commit-stuff/checkout-pr 3
Please select a repository:
  1) metron
  2) metron-bro-plugin-kafka
Selection [metron]: bro

```
1.  Install the package, and all of its dependancies, from the PR branch.  
Ensure it passes its unit tests.
```
# # Install librdkafka by following ONLY instruction 1 
[here](https://github.com/apache/metron-bro-plugin-kafka#installation)
# cd ~/metron-bro-plugin-kafka-pr3/
# bro-pkg install .
```
1.  Configure the plugin.
```
cat << EOF >> /usr/local/bro/share/bro/site/local.bro

# Activate metron-bro-plugin-kafka
@load metron-bro-plugin-kafka-pr3/Bro/Kafka

# Configure metron-bro-plugin-kafka
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG);
EOF 
```
1.  Run bro manually while monitoring kafka to confirm things are working.
```
# mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic
# wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
~/brotmp/example-traffic/exercise-traffic.pcap
# wget 
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
 -O ~/brotmp/nitroba/nitroba.pcap
# export PATH=$PATH:~/kafka_2.11-0.10.1.0/bin
# screen
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
# # Ctrl+A c to make a new screen window
# cd ~/brotmp/example-traffic
# bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro 
-C
# # Use Ctrl+A n to cycle through screen sessions for validation.  To 
run another test, on your second window, do
# cd ~/brotmp/nitroba
# bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-813

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron-bro-plugin-kafka/pull/3.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3


commit f21e51f4f91452d66b644b1c041e9a3ae3b39bd7
Author: Jon Zeolla &

[GitHub] metron pull request #837: METRON-1309: Change metron-deployment to pull the ...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/837#discussion_r150056693
  
--- Diff: 
metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md ---
@@ -638,15 +638,15 @@ Edit crontab with `# crontab -e` and add:
 
 bro-kafka:
 ```
-# cp -r /root/metron/metron-sensors/bro-plugin-kafka /tmp
-# cd /tmp/bro-plugin-kafka
+# git clone https://github.com/apache/metron-bro-plugin-kafka 
/tmp/metron-bro-plugin-kafka
+# cd /tmp/metron-bro-plugin-kafka
 # rm -rf build/
--- End diff --

Not a bad idea - I was just migrating the instructions to use the new 
location without changing anything else.


---

[GitHub] metron issue #837: METRON-1309: Change metron-deployment to pull the plugin ...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/837
  
This is ready for review.


---

[GitHub] metron pull request #837: METRON-1309: Change metron-deployment to pull the ...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/837

METRON-1309: Change metron-deployment to pull the plugin from 
apache/metron-bro-plugin-kafka

## Contributor Comments
Based on the conversation on the [mailing 
list](https://lists.apache.org/thread.html/dd63e5bcf8a288dfaf3cdd551128e3df59580367080bda5f344e8538@%3Cdev.metron.apache.org%3E),
 this removes the bro kafka plugin from this repo and pulls it from our new 
apache/metron-bro-plugin-kafka repo.

# Testing
## Please hold on testing, this needs to be validated
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-1309
git clone https://github.com/apache/metron ~/metron-1309/metron
cd ~/metron-1309/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-1309
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-1309/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Spin up full-dev
```
cd metron-deployment/vagrant/full-dev-platform/
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-1309/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Run bro against some public pcaps.
```
mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic
wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
~/brotmp/example-traffic/exercise-traffic.pcap
wget 
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
 -O ~/brotmp/nitroba/nitroba.pcap
cd ~/brotmp/example-traffic
bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/nitroba
bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
```
1.  Verify that logs are properly getting to the bro kafka topic, via the 
terminal you set up two steps ago.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository su

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r149767436
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│   └── stellar
+└── lib
+└── stellar-common-0.4.1-uber.jar
+```
+
+To run the Stellar Shell run the following from the directory you unpacked 
to:
+
+```bash
+bin/stellar
+```
+
+```bash
+-> % bin/stellar
+Stellar, Go!
+Please note that functions are loading lazily in the background and will 
be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, 
CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, 
DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, 
IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, 
MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, 
SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, 
TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, 
TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, 
URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST
+[Stellar]>>>
+```
+
+By default the shell will have the base Stellar Language commands 
available.  Any jars in the lib directory
+that contain Stellar functions will also be loaded, and their commands 
will be available to shell, as long
+as their dependencies are satisfied.
--- End diff --

What about if you just change it from:
have the base Stellar Language commands
to
have only the base Stellar Language commands
?

I'm not looking to explicitly mention *what* functions would not exist 
here, and why, but simply make it more obvious that things will be missing here 
by default.


---

[GitHub] metron issue #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
I just took a stab at an update, @cestella can you take a look when you get 
a second


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r149686111
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│   └── stellar
+└── lib
+└── stellar-common-0.4.1-uber.jar
--- End diff --

`s/0\.4\.1/0.4.2/`


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r149686086
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz
--- End diff --

`s/0\.4\.1/0.4.2/`


---

[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/690#discussion_r149759730
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the 
background and will be unav
 %functions
 ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, 
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, 
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, 
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY,
  SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR 
 ```
+## Stellar Shell Stand Alone
+
+The Stellar Shell is also packaged as a stand alone application.  It can 
be unpacked on any supported
+operating system.
+
+```bash

+metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz
+```
+
+When unpacked, the following structure will be created:
+
+```bash
+.
+├── bin
+│   └── stellar
+└── lib
+└── stellar-common-0.4.1-uber.jar
+```
+
+To run the Stellar Shell run the following from the directory you unpacked 
to:
+
+```bash
+bin/stellar
+```
+
+```bash
+-> % bin/stellar
+Stellar, Go!
+Please note that functions are loading lazily in the background and will 
be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, 
CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, 
DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, 
DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, 
IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, 
MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, 
SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, 
TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, 
TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, 
URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST
+[Stellar]>>>
+```
+
+By default the shell will have the base Stellar Language commands 
available.  Any jars in the lib directory
+that contain Stellar functions will also be loaded, and their commands 
will be available to shell, as long
+as their dependencies are satisfied.
--- End diff --

Maybe we can be a little bit more explicit that, by default, not all 
stellar functions will work in the standalone shell.  For instance, clarifying 
that only functions available listed in response to %functions will be included 
may be enough?  I quickly noted that BIN, HLLP_*, STATS_*, OUTLIER_MAD_*, etc. 
weren't in there as profiler/management commands aren't included.  Maybe we say 
that as well?


---

[GitHub] metron issue #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
I think I'm more of a fan of being explicit than providing a key, given 
this only happens a few times through the whole doc.  Also, like I mentioned, 
the \*s are being suppressed, so they're not actually in either form of the 
documentation 
([here](https://metron.apache.org/current-book/metron-stellar/stellar-common/index.html)
 and 
[here](https://github.com/apache/metron/blob/master/metron-stellar/stellar-common/README.md))
 which we've seen bite us numerous times in the past in our docs.


---

[GitHub] metron pull request #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/836#discussion_r149706457
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -748,7 +728,7 @@ In the core language functions, we support basic 
functional programming primitiv
 ### `MULTISET_INIT`
   * Description: Creates an empty multiset, which is a map associating 
objects to their instance counts.
   * Input:
-* input? - An initialization of the multiset
+* input?? - An initialization of the multiset
--- End diff --

Gotcha, okay.  I think I would prefer something like (optional), what do 
you think?


---

[GitHub] metron pull request #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/836#discussion_r149701086
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -257,7 +258,7 @@ In the core language functions, we support basic 
functional programming primitiv
   * Description: Adds an element to the bloom filter passed in
   * Input:
 * bloom - The bloom filter
-* value* - The values to add
+* value\*? - The values to add
--- End diff --

Why does this have an \*?


---

[GitHub] metron pull request #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/836#discussion_r149701286
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -1034,15 +1035,15 @@ In the core language functions, we support basic 
functional programming primitiv
   See [python](https://docs.python.org/3/library/functions.html#zip)
   and 
[wikipedia](https://en.wikipedia.org/wiki/Convolution_(computer_science)) for 
more context.
   * Input:
-* list* - Lists to zip.
+* list\*? - Lists to zip.
   * Returns: The zip of the lists.  The returned list is the min size of 
all the lists. e.g. `ZIP( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4] ]`
 
 ### `ZIP_LONGEST`
   * Description: Zips lists into a single list where the ith element is an 
list containing the ith items from the constituent lists.
   See 
[python](https://docs.python.org/3/library/itertools.html#itertools.zip_longest)
   and 
[wikipedia](https://en.wikipedia.org/wiki/Convolution_(computer_science)) for 
more context.
   * Input:
-* list* - Lists to zip.
+* list\*? - Lists to zip.
--- End diff --

Why does this have an \*?


---

[GitHub] metron pull request #836: DO NOT MERGE METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/836#discussion_r149701209
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -849,7 +829,7 @@ In the core language functions, we support basic 
functional programming primitiv
 ### `SET_INIT`
   * Description: Creates an new set
   * Input:
-* input? - An initialization of the set
+* input?? - An initialization of the set
--- End diff --

Why does this have a \??


---