This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new e1d1901 METRON-2100 Update developer documentation for full dev
management UI parser aggregation feature gap (mmiklavc) closes
apache/metron#1398
e1d1901 is described below
commit e1d1901395d7b135deb48c3c4af5a94ed35fbaa4
Author: mmiklavc <[email protected]>
AuthorDate: Wed May 8 10:10:21 2019 -0600
METRON-2100 Update developer documentation for full dev management UI
parser aggregation feature gap (mmiklavc) closes apache/metron#1398
---
Upgrading.md | 7 ++++---
metron-deployment/README.md | 35 +++++++++++++++++++++++++++++++++++
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/Upgrading.md b/Upgrading.md
index b3ab8d2..d59aa57 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -19,13 +19,14 @@ limitations under the License.
This document constitutes a per-version listing of changes of
configuration which are non-backwards compatible.
-## 0.7.1 to 0.7.2
+## 0.7.0 to 0.7.1
+
+### [METRON-2100: Update developer documentation for full dev management UI
parser aggregation feature
gap](https://issues.apache.org/jira/browse/METRON-2100)
+The original full_dev environment change was actually introduced in Metron
0.7.0. This Jira addresses missing user documentation for the Management UI
feature gap for parser aggregation. See [Parser Aggregation
Feature](metron-deployment#parser-aggregation-feature) for more details on how
to work with and configure parsers with this feature change enabled in full_dev.
### [METRON-2053: Refactor metron-enrichment to decouple Storm
dependencies](https://issues.apache.org/jira/browse/METRON-2053)
`org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter` has had its
packaged changed to
`org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter`. It has also been
moved from the `metron-platform/metron-enrichment` module to a more appropriate
home in `metron-platform/metron-writer`.
-## 0.7.0 to 0.7.1
-
### [METRON-1929: Build GET_ASN Stellar
function](https://issues.apache.org/jira/browse/METRON-1929)
The script for `geo_enrichment_load.sh` has been renamed, and now is
`maxmind_enrichment_load.sh`. A couple changes should happen for users who are
upgrading.
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index b78a3e2..73cf602 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -73,6 +73,41 @@ To deploy Metron in a VM running on your computer, follow
the instructions at [d
We recommend looking at Ambari and shutting down any services you may not be
using. For example, we recommend turning off Metron Profiler, as this commonly
causes REST services to crash when running on a single VM.
+### Parser Aggregation Feature
+
+The [Parser
Aggregation](../metron-platform/metron-parsing/metron-parsing-storm#parser-aggregation)
feature does not currently exist in the management UI. In order to address
resource limitations in the full dev development environments, bro, yaf, and
snort have been aggregated into a single parser
+topology. However, the Management UI is not currently able to display its
status until the feature is added. Aggregated parsers can still be created via
Ambari and the command line scripts.
+
+Here are some tips for working with parser aggregation while the UI feature is
being developed.
+
+* **How are parsers picked up by the UI?:** This is based entirely on what is
currently stored in the Zookeeper configs. See [Management
Utility](../metron-platform/metron-common#management-utility) "DUMP" option
with "-c PARSER" to see all of what is currently loaded. The management UI does
not
+update the configurations stored locally on disk, so Zookeeper is the source
of truth.
+
+* **Removing an existing aggregation:** In the [Ambari UI](http://node1:8080)
click on the Metron serice and select "Metron Parsers." Select "stop" from the
dropdown for the parser component. Click "back," "configs," and then navigate
to "Parsers." In the text field option labeled "parsers".
+Remove the double quotes from around the listed parsers. Save and choose
"Restart" when prompted. This will deploy three individual parsers rather than
a single aggregated parser: bro, snort, and yaf. Be aware, you may need to shut
down other topologies to free up resources so that you can
+run the parsers without aggregation. Stopping the profiler, pcap, or
batch_indexing are a few options that will still allow data to pass through the
system end-to-end.
+
+* **Managing parser lifecycle:** Starting and stopping parsers in the
management UI will in no way affect a parser running as aggregated. The
exception to this is if you create a parser via the management UI that has the
same name as the
+aggregation, e.g. "bro__snort__yaf." We recommend against this. It will appear
as thought you now have the ability to manage the aggregated parser now, but
you will only be able to start/stop it.
+
+* **Editing parser configuration:** In order to modify the aggregated parsers'
configurations, you will need to first pull all of the configuration from
Zookeeper to the local configuration directory by executing the following
+commands
+
+ ```
+ source /etc/default/metron
+ $METRON_HOME/bin/zk_load_configs.sh -m PULL -o
${METRON_HOME}/config/zookeeper -z $ZOOKEEPER -f
+ ```
+
+ Make your changes to an individual parser's configuration json, e.g.
`${METRON_HOME}/config/zookeeper/parsers/bro.json`, save locally, and then push
them back up to Zookeeper
+
+ ```
+ $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
$METRON_HOME/config/zookeeper/ -z $ZOOKEEPER
+ ```
+
+ See [Management
Utility](../metron-platform/metron-common#management-utility) for more detail.
+
+* **Other gotchas:** Stop the aggregated parsers in Ambari before removing or
adding grouping quotes. Otherwise, you will end up with both the individual
parsers and the aggregated topology running concurrently. Ambari only manages
the parser topology lifecycle via the current parser name list provided,
+so changing that list removes Ambari's ability to reference the old topology
names.
How do I build RPM packages?
----------------------------
