Repository: metron Updated Branches: refs/heads/master f7b0a1602 -> c225fc49b
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c225fc49 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c225fc49 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c225fc49 Branch: refs/heads/master Commit: c225fc49b3e97b488a3b11770266b7ed5a19e984 Parents: f7b0a16 Author: JonZeolla <[email protected]> Authored: Wed Nov 8 09:55:11 2017 -0500 Committer: cstella <[email protected]> Committed: Wed Nov 8 09:55:11 2017 -0500 ---------------------------------------------------------------------- metron-platform/Performance-tuning-guide.md | 65 ++++++++++++---------- metron-platform/metron-parsers/README.md | 68 +++++++++++++----------- 2 files changed, 74 insertions(+), 59 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/c225fc49/metron-platform/Performance-tuning-guide.md ---------------------------------------------------------------------- diff --git a/metron-platform/Performance-tuning-guide.md b/metron-platform/Performance-tuning-guide.md index c016d55..a1fe738 100644 --- a/metron-platform/Performance-tuning-guide.md +++ b/metron-platform/Performance-tuning-guide.md @@ -219,38 +219,49 @@ And we ran our bro parser topology with the following options. We did not need t though you could certainly do so if necessary. Notice that we only needed 1 worker. ``` -/usr/metron/0.4.0/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s bro -ksp SASL_PLAINTEXT - -ot enrichments +/usr/metron/0.4.2/bin/start_parser_topology.sh \ -e ~metron/.storm/storm-bro.config \ -esc ~/.storm/spout-bro.config \ - -sp 24 \ - -snt 24 \ + -k $BROKERLIST \ + -ksp SASL_PLAINTEXT \ -nw 1 \ + -ot enrichments \ -pnt 24 \ -pp 24 \ -``` - -From the usage docs, here are the options we've used. The full reference can be found here - https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md -``` --e,--extra_topology_options <JSON_FILE> Extra options in the form - of a JSON file with a map - for content. --esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options - in the form of a JSON file - with a map for content. - Possible keys are: - retryDelayMaxMs,retryDelay - Multiplier,retryInitialDel - ayMs,stateUpdateIntervalMs - ,bufferSizeBytes,fetchMaxW - ait,fetchSizeBytes,maxOffs - etBehind,metricsTimeBucket - SizeInSecs,socketTimeoutMs --sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint --snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks --nw,--num_workers <NUM_WORKERS> Number of Workers --pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks --pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint + -s bro \ + -snt 24 \ + -sp 24 \ + -z $ZOOKEEPER \ +``` + +From the usage docs, here are the options we've used. The full reference can be found [here](../metron-platform/metron-parsers/README.md#Starting_the_Parser_Topology). +``` +usage: start_parser_topology.sh + -e,--extra_topology_options <JSON_FILE> Extra options in the form + of a JSON file with a map + for content. + -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options + in the form of a JSON file + with a map for content. + Possible keys are: + retryDelayMaxMs,retryDelay + Multiplier,retryInitialDel + ayMs,stateUpdateIntervalMs + ,bufferSizeBytes,fetchMaxW + ait,fetchSizeBytes,maxOffs + etBehind,metricsTimeBucket + SizeInSecs,socketTimeoutMs + -k,--kafka <BROKER_URL> Kafka Broker URL + -ksp,--kafka_security_protocol <SECURITY_PROTOCOL> Kafka Security Protocol + -nw,--num_workers <NUM_WORKERS> Number of Workers + -ot,--output_topic <KAFKA_TOPIC> Output Kafka Topic + -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks + -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint + -s,--sensor <SENSOR_TYPE> Sensor Type + -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks + -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint + -z,--zk <ZK_QUORUM> Zookeeper Quroum URL + (zk1:2181,zk2:2181,... ``` ### Enrichment Tuning http://git-wip-us.apache.org/repos/asf/metron/blob/c225fc49/metron-platform/metron-parsers/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md index 75559c0..54dbef2 100644 --- a/metron-platform/metron-parsers/README.md +++ b/metron-platform/metron-parsers/README.md @@ -373,37 +373,41 @@ The usage for `start_parser_topology.sh` is as follows: ``` usage: start_parser_topology.sh - -e,--extra_topology_options <JSON_FILE> Extra options in the form - of a JSON file with a map - for content. - -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options - in the form of a JSON file - with a map for content. - Possible keys are: - retryDelayMaxMs,retryDelay - Multiplier,retryInitialDel - ayMs,stateUpdateIntervalMs - ,bufferSizeBytes,fetchMaxW - ait,fetchSizeBytes,maxOffs - etBehind,metricsTimeBucket - SizeInSecs,socketTimeoutMs - -ewnt,--error_writer_num_tasks <NUM_TASKS> Error Writer Num Tasks - -ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism - Hint - -h,--help This screen - -k,--kafka <BROKER_URL> Kafka Broker URL - -mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds - -mtp,--max_task_parallelism <MAX_TASK> Max task parallelism - -na,--num_ackers <NUM_ACKERS> Number of Ackers - -nw,--num_workers <NUM_WORKERS> Number of Workers - -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks - -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint - -s,--sensor <SENSOR_TYPE> Sensor Type - -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks - -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint - -t,--test <TEST> Run in Test Mode - -z,--zk <ZK_QUORUM> Zookeeper Quroum URL - (zk1:2181,zk2:2181,... + -e,--extra_topology_options <JSON_FILE> Extra options in the form + of a JSON file with a map + for content. + -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options + in the form of a JSON file + with a map for content. + Possible keys are: + retryDelayMaxMs,retryDelay + Multiplier,retryInitialDel + ayMs,stateUpdateIntervalMs + ,bufferSizeBytes,fetchMaxW + ait,fetchSizeBytes,maxOffs + etBehind,metricsTimeBucket + SizeInSecs,socketTimeoutMs + -ewnt,--error_writer_num_tasks <NUM_TASKS> Error Writer Num Tasks + -ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism + Hint + -h,--help This screen + -iwnt,--invalid_writer_num_tasks <NUM_TASKS> Invalid Writer Num Tasks + -iwp,--invalid_writer_p <PARALLELISM_HINT> Invalid Message Writer Parallelism Hint + -k,--kafka <BROKER_URL> Kafka Broker URL + -ksp,--kafka_security_protocol <SECURITY_PROTOCOL> Kafka Security Protocol + -mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds + -mtp,--max_task_parallelism <MAX_TASK> Max task parallelism + -na,--num_ackers <NUM_ACKERS> Number of Ackers + -nw,--num_workers <NUM_WORKERS> Number of Workers + -ot,--output_topic <KAFKA_TOPIC> Output Kafka Topic + -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks + -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint + -s,--sensor <SENSOR_TYPE> Sensor Type + -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks + -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint + -t,--test <TEST> Run in Test Mode + -z,--zk <ZK_QUORUM> Zookeeper Quroum URL + (zk1:2181,zk2:2181,... ``` ## The `--extra_kafka_spout_config` Option @@ -453,7 +457,7 @@ are a few knobs to tune to get the most out of your system. In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors. -Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) +Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](../../metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) ## Kafka Queue The kafka queue associated with your parser is a collection point for
