Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/961#discussion_r174772238
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,527 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of
the Enrichment topology. The guide also provides detailed steps on how to
execute those benchmarks along with advice for tuning the Unified Enrichment
topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+The following section describes a set of enrichments that will be used to
benchmark the performance of the Enrichment topology.
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.
Given a valid IP address the enrichment will append detailed location
information for that IP. The location information is sourced from an external
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java).
+
+#### Configuration
+
+Adding the following Stellar expression to the Enrichment topology
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the telemetry message will
contain a set of fields with location information for the given IP address.
+```
+{
+ "ip_dst_addr":"151.101.129.140",
+ ...
+ "geo.city":"San Francisco",
+ "geo.country":"US",
+ "geo.dmaCode":"807",
+ "geo.latitude":"37.7697",
+ "geo.location_point":"37.7697,-122.3933",
+ "geo.locID":"5391959",
+ "geo.longitude":"-122.3933",
+ "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that
retrieves data from an external HBase table. This type of enrichment is useful
for enriching telemetry from an Asset Database or other source of relatively
static data.
+
+#### Configuration
+
+Adding the following Stellar expression to the Enrichment topology
configuration will define an Hbase enrichment. This looks up the 'ip_dst_addr'
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP
elements that were retrieved from the HBase table.
+```
+{
+ "ip_dst_addr":"151.101.2.166",
+ ...
+ "top1m.host":"earther.com",
+ "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar
expression. In this benchmark, the enrichment is purely a computational task
that has no dependence on an external system like a database.
+
+#### Configuration
+
+Adding the following Stellar expression to the Enrichment topology
configuration will define a basic Stellar enrichment. The following returns
true if the IP is in the given subnet and false otherwise.
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a
boolean value indicating whether the IP was within the given subnet.
+```
+{
+ "ip_dst_addr":"151.101.2.166",
+ ...
+ "local":false
+}
+```
+
+## Benchmark Execution
+
+This section describes the steps necessary to execute the performance
benchmarks for the Enrichment topology.
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.
+
+ A script like the following can be used for this. There is no need to
do this for all 1 million entries in the data set. Doing this for around 10,000
records is sufficient.
+
+ ```python
+ import dns.resolver
+ import csv
+
+ resolver = dns.resolver.Resolver()
+ resolver.nameservers = ['8.8.8.8', '8.8.4.4']
+
+ with open('top-1m.csv', 'r') as infile:
+ with open('top-1m-with-ip.csv', 'w') as outfile:
+
+ reader = csv.reader(infile, delimiter=',')
+ writer = csv.writer(outfile, delimiter=',')
+ for row in reader:
+
+ host = row[1]
+ try:
+ response = resolver.query(host, "A")
+ for record in response:
+ ip = record
+ writer.writerow([host, ip])
+ print "host={}, ip={}".format(host, ip)
+
+ except:
+ pass
+ ```
+
+3. The resulting data set contains an IP to hostname mapping.
+ ```bash
+ $ head top-1m-with-ip.csv
+ google.com,172.217.9.46
+ youtube.com,172.217.4.78
+ facebook.com,157.240.18.35
+ baidu.com,220.181.57.216
+ baidu.com,111.13.101.208
+ baidu.com,123.125.114.144
+ wikipedia.org,208.80.154.224
+ yahoo.com,98.139.180.180
+ yahoo.com,206.190.39.42
+ reddit.com,151.101.1.140
+ ```
+
+### Load HBase with Enrichment Data
+
+1. Create an HBase table for this data.
+
+ Ensure that the table is evenly distributed across the HBase nodes.
This can be done by pre-splitting the table or splitting the data after loading
it.
+
+ ```
+ create 'top-1m', 't', {SPLITS => ['2','4','6','8','a','c','e']}
+ ```
+
+1. Create a configuration file called `extractor.json`. This defines how
the data will be loaded into the HBase table.
+
+ ```bash
+ > cat extractor.json
+ {
+ "config": {
+ "columns": {
+ "host" : 0,
+ "ip": 1
+ },
+ "indicator_column": "ip",
+ "type": "top-1m",
+ "separator": ","
+ },
+ "extractor": "CSV"
+ }
+ ```
+
+1. Use the `flatfile_loader.sh` to load the data into the HBase table.
+ ```
+ $METRON_HOME/bin/flatfile_loader.sh \
+ -e extractor.json \
+ -t top-1m \
+ -c t \
+ -i top-1m-with-ip.csv
+ ```
+
+### Configure the Enrichments
+
+1. Define the Enrichments using the REPL.
+
+ ```
+ > $METRON_HOME/bin/stellar -z $ZOOKEEPER
+ Stellar, Go!
+
+ [Stellar]>>> conf
+ {
+ "enrichment": {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "geo" : "GEO_GET(ip_dst_addr)",
+ "top1m" : "ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m',
't')",
+ "local" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')"
+ }
+ }
+ },
+ "fieldToTypeMap": {
+ }
+ },
+ "threatIntel": {
+ }
+ }
+ [Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "asa")
+ ```
--- End diff --
Everything should be addressed @JonZeolla . Thanks
---
