[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441321#comment-16441321
 ] 

ASF GitHub Bot commented on METRON-1445:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/988


> Update performance tuning guide with more explicit parameter instructions
> -
>
> Key: METRON-1445
> URL: https://issues.apache.org/jira/browse/METRON-1445
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441252#comment-16441252
 ] 

ASF GitHub Bot commented on METRON-1445:


Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/988#discussion_r182170422
  
--- Diff: metron-platform/Performance-tuning-guide.md ---
@@ -111,15 +138,146 @@ __spout-config.json__
 }
 ```
 
-These are the spout recommended defaults from Storm and are currently the 
defaults provided in the Kafka spout itself. In fact, if you find the 
recommended defaults work fine for you,
+Above is a snippet for configuring parsers. These are the spout 
recommended defaults from Storm and are currently the defaults provided in the 
Kafka spout itself. In fact, if you find the recommended defaults work fine for 
you,
 then you can omit these settings altogether.
 
+ Where to Find Tuning Properties
+
+**Important:** The parser topologies are deployed via a builder pattern 
that takes parameters from the CLI as set via Ambari. The enrichment and 
indexing topologies are configured
+using a Storm Flux file, a configuration properties file, and Ambari. Here 
is a setting materialization summary for each of the topology types:
+
+- Parsers
+   - Management UI -> parser json config and CLI -> Storm
+- Enrichment
+   - Ambari UI -> properties file -> Flux -> Storm
+- Indexing
+   - Ambari UI -> properties file -> Flux -> Storm
+
+**Parsers**
+
+This is a mapping of the various performance tuning properties for parsers 
and how they are materialized.
+
+See more detail on starting parsers 
[here](https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md#starting-the-parser-topology)
+
+| Category| Management UI Property Name
| JSON Config File Property Name | CLI Option   
  | Storm Property Name 
|  Notes
|

+|-||||-|---|
+| Storm topology config   | Num Workers
| n/a| -nw,--num_workers   
  | topology.workers
|   
|
+| | Num Ackers 
| n/a| -na,--num_ackers 
  | 
topology.acker.executors|   
|
+| | Storm Config   
| topology.max.spout.pending | -e,--extra_topology_options , 
e.g. { "topology.max.spout.pending" : NUM }   | 
topology.max.spout.pending  | Put property in JSON format in a file named 
`storm--config.json`   |
+| Kafka spout | Spout Parallelism  
| n/a| -sp,--spout_p
  | n/a 
|   
|
+| | Spout Num Tasks
| n/a| -snt,--spout_num_tasks
  | n/a 
|   
|
+| | Spout Config   
| spout.pollTimeoutMs| -esc,--extra_kafka_spout_config 
, e.g. { "spout.pollTimeoutMs" : 200 }  | n/a
 | Put property in JSON format in a file named 
`spout--config.json`   |
+| | Spout Config   
| spout.maxUncommittedOffsets| -esc,--extra_kafka_spout_config 
, e.g. { "spout.maxUncommittedOffsets" : 1000 } | n/a
 | Put property in JSON format in a file named 
`spout--config.json`   |
+| | Spout Config   
| spout.offsetCommitPeriodMs | -esc,--extra_kafka_spout_config 
, e.g. { "spout.offsetCommitPeriodMs" : 

[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441243#comment-16441243
 ] 

ASF GitHub Bot commented on METRON-1445:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/988#discussion_r182168873
  
--- Diff: metron-platform/Performance-tuning-guide.md ---
@@ -111,15 +138,146 @@ __spout-config.json__
 }
 ```
 
-These are the spout recommended defaults from Storm and are currently the 
defaults provided in the Kafka spout itself. In fact, if you find the 
recommended defaults work fine for you,
+Above is a snippet for configuring parsers. These are the spout 
recommended defaults from Storm and are currently the defaults provided in the 
Kafka spout itself. In fact, if you find the recommended defaults work fine for 
you,
 then you can omit these settings altogether.
 
+ Where to Find Tuning Properties
+
+**Important:** The parser topologies are deployed via a builder pattern 
that takes parameters from the CLI as set via Ambari. The enrichment and 
indexing topologies are configured
+using a Storm Flux file, a configuration properties file, and Ambari. Here 
is a setting materialization summary for each of the topology types:
+
+- Parsers
+   - Management UI -> parser json config and CLI -> Storm
+- Enrichment
+   - Ambari UI -> properties file -> Flux -> Storm
+- Indexing
+   - Ambari UI -> properties file -> Flux -> Storm
+
+**Parsers**
+
+This is a mapping of the various performance tuning properties for parsers 
and how they are materialized.
+
+See more detail on starting parsers 
[here](https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md#starting-the-parser-topology)
+
+| Category| Management UI Property Name
| JSON Config File Property Name | CLI Option   
  | Storm Property Name 
|  Notes
|

+|-||||-|---|
+| Storm topology config   | Num Workers
| n/a| -nw,--num_workers   
  | topology.workers
|   
|
+| | Num Ackers 
| n/a| -na,--num_ackers 
  | 
topology.acker.executors|   
|
+| | Storm Config   
| topology.max.spout.pending | -e,--extra_topology_options , 
e.g. { "topology.max.spout.pending" : NUM }   | 
topology.max.spout.pending  | Put property in JSON format in a file named 
`storm--config.json`   |
+| Kafka spout | Spout Parallelism  
| n/a| -sp,--spout_p
  | n/a 
|   
|
+| | Spout Num Tasks
| n/a| -snt,--spout_num_tasks
  | n/a 
|   
|
+| | Spout Config   
| spout.pollTimeoutMs| -esc,--extra_kafka_spout_config 
, e.g. { "spout.pollTimeoutMs" : 200 }  | n/a
 | Put property in JSON format in a file named 
`spout--config.json`   |
+| | Spout Config   
| spout.maxUncommittedOffsets| -esc,--extra_kafka_spout_config 
, e.g. { "spout.maxUncommittedOffsets" : 1000 } | n/a
 | Put property in JSON format in a file named 
`spout--config.json`   |
+| | Spout Config   
| spout.offsetCommitPeriodMs | -esc,--extra_kafka_spout_config 
, e.g. { "spout.offsetCommitPeriodMs" : 

[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441186#comment-16441186
 ] 

ASF GitHub Bot commented on METRON-1445:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/988
  
+1 Thanks


> Update performance tuning guide with more explicit parameter instructions
> -
>
> Key: METRON-1445
> URL: https://issues.apache.org/jira/browse/METRON-1445
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441155#comment-16441155
 ] 

ASF GitHub Bot commented on METRON-1445:


Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/988#discussion_r182152983
  
--- Diff: metron-platform/metron-common/README.md ---
@@ -400,3 +401,34 @@ __Side Effects__
 Calling the mark() method multiple times simply resets the start time to 
the current nano time. Calling log() with a non-existent mark name will log 0 
ns elapsed time with a warning indicating that log has been invoked for a mark 
name that does not exist.
 The class is not thread-safe and makes no attempt at keeping multiple 
threads from modifying the same markers.
 
+# Metron Debugging
+
+A Python script is provided for gathering information useful in debugging 
your Metron cluster. Run from the node that has Metron installed on it. All 
options listed below are required.
+
+_Note:_ Be aware that no anonymization/scrubbing is performed on the 
captured configuration details.
+
+```
+# ./cluster_info.py -h
--- End diff --

@nickwallen fixed to reference $METRON_HOME/bin


> Update performance tuning guide with more explicit parameter instructions
> -
>
> Key: METRON-1445
> URL: https://issues.apache.org/jira/browse/METRON-1445
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1445) Update performance tuning guide with more explicit parameter instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433494#comment-16433494
 ] 

ASF GitHub Bot commented on METRON-1445:


GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/988

METRON-1445: Update performance tuning guide with more explicit parameter 
instructions

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1445

The current performance tuning guide can be challenging to follow for those 
not familiar with the system. This PR updates the guide to provide details 
about how and where to find tuning parameters along with details as to how the 
properties are materialized in a running topology.

Additionally, a supplement to the tuning guide has been provided to pull 
Metron cluster debugging info into a tarball. As pertains to the perf guide, it 
captures the state of Ambari configuration, flux files, properties files, and 
zk config. This helps users confirm system state as they make tuning changes.

## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
n/a

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron performance-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #988


commit 43120ee6825d52fd5074fa2ab2a8c8f6d8a71dba
Author: Michael Miklavcic 
Date:   2018-02-16T19:40:00Z

Add start of debugging utility. First pass at perf tuning doc updates.

commit f5c39d02c636ed7c7e08fddef8f0d1d9da4052d9
Author: Michael Miklavcic 
Date:   2018-04-03T21:54:12Z

Merge with master

commit 79ddc02767cec5a76b5d0955546bbedf96a4353f
Author: Michael Miklavcic 
Date:   2018-04-04T18:08:04Z

Merge branch 'master' into performance-doc

commit 4f4cf53699330257531df87a50f8c5dee13aca11
Author: Michael Miklavcic 
Date:   2018-04-05T15:21:03Z

Merge branch 'master' into performance-doc

commit fc17e1d1b00809c2c4a5514181221722a117cd5e
Author: Michael Miklavcic 
Date:   2018-04-11T06:47:18Z

Finish tuning guide parameter mappings. Move debug info script from Bash to 
Python.




> Update performance tuning guide with more explicit parameter instructions
> -
>
> Key: METRON-1445
> URL: https://issues.apache.org/jira/browse/METRON-1445
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)