[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127555429
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
 ---
@@ -37,3 +37,5 @@ SECURITY_ENABLED={{security_enabled|lower}}
 {% endif %}
 {% if metron_keytab_path is defined 
%}METRON_SERVICE_KEYTAB="{{metron_keytab_path}}"
 {% endif %}
+KAFKA_SECURITY_PROTOCOL="{{kafka_security_protocol}}"
+PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
--- End diff --

I think that's fine.  If you really wanted it to mirror the implementation 
in the MPack, you could do:
`PARSER_TOPOLOGY_OPTIONS="/home/{{metron_user}}/.storm/storm.config"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #654: METRON-1044: Disabled writers are not acking messa...

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/654


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127534192
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
 ---
@@ -37,3 +37,5 @@ SECURITY_ENABLED={{security_enabled|lower}}
 {% endif %}
 {% if metron_keytab_path is defined 
%}METRON_SERVICE_KEYTAB="{{metron_keytab_path}}"
 {% endif %}
+KAFKA_SECURITY_PROTOCOL="{{kafka_security_protocol}}"
+PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
--- End diff --

I did not find any existing property that defines the 
`PARSER_TOPOLOGY_OPTIONS` value.  It is hard coded in one of the MPack Python 
scripts (`parser_commands.py`) as `' -e ~' + self.__params.metron_user + 
'/.storm/storm.config'`.

I thought that if I just put the exact value here, we are half way to 
making it configurable in Ambari, if we choose to do so.

I also tried to use `~${METRON_HOME}/.storm/storm.config`, but the 
`ProcessBuilder` does not do tilde expansion for user's home directories.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #647: METRON-1031: Management UI Cannot Start Topologies in Ker...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/647
  
- I removed `-ksp` from Enrichment and Indexing.
- I added `-e` also to the Parsers.  This is also required to get them 
running.
- Updated the test plan.

I ran through a full test once successfully, but I changed a few minor 
things and just need to run through the manual test once more.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #654: METRON-1044: Disabled writers are not acking messa...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/654#discussion_r127531933
  
--- Diff: 
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
 ---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch() 
throws Exception {
   }
 
   @Test
+  public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws 
Exception {
--- End diff --

I combined the test for a disabled sensor into the previous test for 
enabled sensors.  Is this ok?  Otherwise we'll have 2 tests that look almost 
identical.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #654: METRON-1044: Disabled writers are not acking messa...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/654#discussion_r127527841
  
--- Diff: 
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
 ---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch() 
throws Exception {
   }
 
   @Test
+  public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws 
Exception {
--- End diff --

The first one (writer for one sensor is enabled and writer for another 
sensor is disabled).  Should have been more clear.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #654: METRON-1044: Disabled writers are not acking messa...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/654#discussion_r127527166
  
--- Diff: 
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
 ---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch() 
throws Exception {
   }
 
   @Test
+  public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws 
Exception {
--- End diff --

Do you mean write a test where a writer for one sensor is enabled and a 
writer for another sensor is disabled?  Or do you really want separate writers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #654: METRON-1044: Disabled writers are not acking messa...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/654#discussion_r127526102
  
--- Diff: 
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
 ---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch() 
throws Exception {
   }
 
   @Test
+  public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws 
Exception {
--- End diff --

It's not a huge deal, but is it possible to write a test where one writer 
is enabled and one is disabled?  I'm mostly concerned about making sure 
everything is spelled out, to avoid any regressions if anything in here gets 
refactored.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Matt Foley]

Yes, if the files are also stored in ZK, Curator can watch them, but it would 
require extension work in our Curator usage.  It currently manages a single 
tree cache.  Managing free-floating files would require careful design work.

Casey, my reference to JSON Pointers was itself the result of a 5-minute 
search; I inferred they might exist and searched for them :-)  But they should 
at least be looked into before we roll our own, especially if they do happen to 
work with Curator.  The initial pointers are:

Structuring a complex schema — Understanding JSON Schema 1.0 
...
https://spacetelescope.github.io/understanding-json-schema/structuring.html
Likewise in JSON Schema, for anything but the most trivial schema, it's really 
useful ... $ref can also be a relative or absolute URI, so if you prefer to 
include your ...

RapidJSON: 
Pointer
http://rapidjson.org/md_doc_pointer.html
(This feature was released in v1.1.0). JSON Pointer is a standardized (RFC6901) 
way to select a value inside a JSON Document (DOM). This can be analogous ...

RFC 6901 - JavaScript Object Notation (JSON) Pointer - IETF 
Tools
https://tools.ietf.org/html/rfc6901
by M Nottingham - ‎2013 - ‎Related 
articles
Abstract JSON Pointer defines a string syntax for identifying a specific value 
within a JavaScript Object Notation (JSON) document. Status of This Memo This 
is ...

JSON API — Latest Specification 
(v1.0)
http://jsonapi.org/format/
This page represents the latest published version of JSON API, which is 
currently 1.0 ... This section describes the structure of a JSON API document, 
which is ...
[Note: this references JSON Pointer as a standard entity [RFC6901]  but not as 
part of the JSON spec.]


From: Otto Fowler 
Date: Friday, July 14, 2017 at 10:42 AM
To: Matt Foley , "dev@metron.apache.org" 

Subject: Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement 
execution

I think the ‘files’ should be stored in zk, and updated with the same mechanism.


On July 14, 2017 at 10:34:36, Casey Stella 
ceste...@gmail.com wrote:

Just chiming in on a part of this: definitely we do not want to lose
automatic config updates (at least, I'd be strongly, strongly STRONGLY
against it).

I definitely agree that JSON files could easily get unwieldy.  I don't know
anything about JSON pointers, could you cover that briefly, Matt?  Even a
URL or two to get started would be great.  Basic googling (while on
vacation) yielded that it was something like xpath for json, but I probably
just googled the wrong thing.

Casey


On July 14, 2017 at 13:27:36, Matt Foley 
(mfo...@hortonworks.com) wrote:
In the abstract, this is a good idea. I see it as related to METRON-987, which 
was the first step in allowing sequences of Stellar statements (aka "programs" 
:-) ) instead of just unrelated groups of single statements. Your proposal lets 
us really work with programs as first-class entities.

However, some concerns need to be resolved:

1. Syntax.

Currently Stellar syntax and JSON fit neatly together. Where would be the cut 
line for file substitutions? Referencing METRON-987, would you only allow a 
file substitution where we currently allow square-bracketed Stellar string 
sequences? What about Profile config syntax, where several chunks of code are 
intimately related (hence want to be located in the same file), but don't all 
get executed at the same time? (This is not a showstopper question because 
Profile configs are usually simple and don't really need file substitution. The 
need is much greater in Enrichment.)

2. Config Updates.

Currently Metron configuration is stored in ZK, but managed through Curator 
libraries. In return for considerable complexity, this gives instant update 
whenever a config changes, without effort in the BI part of the application. 
This differs sharply from file-based configuration, where updates in response 
to config changes require either a restart, an explicit reload command from the 
user, or frequent state-checking in the application.

So currently people trying to develop a new enrichment can update the config, 
and immediately test the result, without restarting and without any explicit 
reload command. We probably want to not lose this.

Rather than roll our own file pointer model, can we use JSON Pointers? Will 
they work with 

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Ryan Merriman]

A couple things I would like to point out.  You can test Stellar statements
without having to send data through parser/enrichment topologies.  There is
a REST endpoint that allows you to pass in a sample message and parser
config and returns a message with Stellar statements applied.  This could
easily be expanded to enrichment configs or testing generic stellar
statements against test messages.

Moving statements to a separate file is going to require a lot of work and
will make our mechanism for managing configuration in bolts more complex.
We would have to also listen for changes in these files and reconcile which
parser/enrichment configs are affected.



On Fri, Jul 14, 2017 at 12:42 PM, Otto Fowler 
wrote:

> I think the ‘files’ should be stored in zk, and updated with the same
> mechanism.
>
> On July 14, 2017 at 13:27:36, Matt Foley (mfo...@hortonworks.com) wrote:
>
> In the abstract, this is a good idea. I see it as related to METRON-987,
> which was the first step in allowing sequences of Stellar statements (aka
> "programs" :-) ) instead of just unrelated groups of single statements.
> Your proposal lets us really work with programs as first-class entities.
>
> However, some concerns need to be resolved:
>
> 1. Syntax.
>
> Currently Stellar syntax and JSON fit neatly together. Where would be the
> cut line for file substitutions? Referencing METRON-987, would you only
> allow a file substitution where we currently allow square-bracketed Stellar
> string sequences? What about Profile config syntax, where several chunks of
> code are intimately related (hence want to be located in the same file),
> but don't all get executed at the same time? (This is not a showstopper
> question because Profile configs are usually simple and don't really need
> file substitution. The need is much greater in Enrichment.)
>
> 2. Config Updates.
>
> Currently Metron configuration is stored in ZK, but managed through Curator
> libraries. In return for considerable complexity, this gives instant update
> whenever a config changes, without effort in the BI part of the
> application. This differs sharply from file-based configuration, where
> updates in response to config changes require either a restart, an explicit
> reload command from the user, or frequent state-checking in the
> application.
>
> So currently people trying to develop a new enrichment can update the
> config, and immediately test the result, without restarting and without any
> explicit reload command. We probably want to not lose this.
>
> Rather than roll our own file pointer model, can we use JSON Pointers? Will
> they work with Curator? Both of those get into some fairly obscure
> features, that would need to be studied. It also actually relates to the
> syntax question presented above.
>
>
> On 7/14/17, 6:17 AM, "Otto Fowler"  wrote:
>
> https://issues.apache.org/jira/browse/METRON-1046
>
> I was thinking this morning that managing stellar statements in the config
> json could become, and maybe is kind of unwieldy.
> To that end, if in say a parser configuration I can refer to a ‘file’ in
> zookeeper as an alternative, we would add the capability to execute and
> manage more complex statements, and even chain multiple statements
> together.
>
> These files could be shared as well.
>
> This could be a Bad Idea™, so I thought I’d throw it out to the list.
>
> Please check out the jira, give some thought, and comment there or on the
> list or both.
>
> O
>

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Otto Fowler]

I think the ‘files’ should be stored in zk, and updated with the same
mechanism.

On July 14, 2017 at 13:27:36, Matt Foley (mfo...@hortonworks.com) wrote:

In the abstract, this is a good idea. I see it as related to METRON-987,
which was the first step in allowing sequences of Stellar statements (aka
"programs" :-) ) instead of just unrelated groups of single statements.
Your proposal lets us really work with programs as first-class entities.

However, some concerns need to be resolved:

1. Syntax.

Currently Stellar syntax and JSON fit neatly together. Where would be the
cut line for file substitutions? Referencing METRON-987, would you only
allow a file substitution where we currently allow square-bracketed Stellar
string sequences? What about Profile config syntax, where several chunks of
code are intimately related (hence want to be located in the same file),
but don't all get executed at the same time? (This is not a showstopper
question because Profile configs are usually simple and don't really need
file substitution. The need is much greater in Enrichment.)

2. Config Updates.

Currently Metron configuration is stored in ZK, but managed through Curator
libraries. In return for considerable complexity, this gives instant update
whenever a config changes, without effort in the BI part of the
application. This differs sharply from file-based configuration, where
updates in response to config changes require either a restart, an explicit
reload command from the user, or frequent state-checking in the
application.

So currently people trying to develop a new enrichment can update the
config, and immediately test the result, without restarting and without any
explicit reload command. We probably want to not lose this.

Rather than roll our own file pointer model, can we use JSON Pointers? Will
they work with Curator? Both of those get into some fairly obscure
features, that would need to be studied. It also actually relates to the
syntax question presented above.


On 7/14/17, 6:17 AM, "Otto Fowler"  wrote:

https://issues.apache.org/jira/browse/METRON-1046

I was thinking this morning that managing stellar statements in the config
json could become, and maybe is kind of unwieldy.
To that end, if in say a parser configuration I can refer to a ‘file’ in
zookeeper as an alternative, we would add the capability to execute and
manage more complex statements, and even chain multiple statements
together.

These files could be shared as well.

This could be a Bad Idea™, so I thought I’d throw it out to the list.

Please check out the jira, give some thought, and comment there or on the
list or both.

O

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Casey Stella]

Just chiming in on a part of this: definitely we do not want to lose
automatic config updates (at least, I'd be strongly, strongly STRONGLY
against it).

I definitely agree that JSON files could easily get unwieldy.  I don't know
anything about JSON pointers, could you cover that briefly, Matt?  Even a
URL or two to get started would be great.  Basic googling (while on
vacation) yielded that it was something like xpath for json, but I probably
just googled the wrong thing.

Casey

On Fri, Jul 14, 2017 at 6:27 PM, Matt Foley  wrote:

> In the abstract, this is a good idea.  I see it as related to METRON-987,
> which was the first step in allowing sequences of Stellar statements (aka
> "programs" :-) ) instead of just unrelated groups of single statements.
> Your proposal lets us really work with programs as first-class entities.
>
> However, some concerns need to be resolved:
>
> 1. Syntax.
>
> Currently Stellar syntax and JSON fit neatly together.  Where would be the
> cut line for file substitutions?  Referencing METRON-987, would you only
> allow a file substitution where we currently allow square-bracketed Stellar
> string sequences?  What about Profile config syntax, where several chunks
> of code are intimately related (hence want to be located in the same file),
> but don't all get executed at the same time? (This is not a showstopper
> question because Profile configs are usually simple and don't really need
> file substitution.  The need is much greater in Enrichment.)
>
> 2. Config Updates.
>
> Currently Metron configuration is stored in ZK, but managed through
> Curator libraries.  In return for considerable complexity, this gives
> instant update whenever a config changes, without effort in the BI part of
> the application.  This differs sharply from file-based configuration, where
> updates in response to config changes require either a restart, an explicit
> reload command from the user, or frequent state-checking in the application.
>
> So currently people trying to develop a new enrichment can update the
> config, and immediately test the result, without restarting and without any
> explicit reload command.  We probably want to not lose this.
>
> Rather than roll our own file pointer model, can we use JSON Pointers?
> Will they work with Curator?  Both of those get into some fairly obscure
> features, that would need to be studied.  It also actually relates to the
> syntax question presented above.
>
>
> On 7/14/17, 6:17 AM, "Otto Fowler"  wrote:
>
> https://issues.apache.org/jira/browse/METRON-1046
>
> I was thinking this morning that managing stellar statements in the
> config
> json could become, and maybe is kind of unwieldy.
> To that end, if in say a parser configuration I can refer to a ‘file’
> in
> zookeeper as an alternative, we would add the capability to execute and
> manage more complex statements, and even chain multiple statements
> together.
>
> These files could be shared as well.
>
> This could be a Bad Idea™, so I thought I’d throw it out to the list.
>
> Please check out the jira, give some thought, and comment there or on
> the
> list or both.
>
> O
>
>
>

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127474849
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
 ---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws 
RestException {
   protected int runCommand(String[] command) throws RestException {
 ProcessBuilder pb = getProcessBuilder(command);
 pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
 try {
   process = pb.start();
   process.waitFor();
+
 } catch (Exception e) {
   throw new RestException(e);
 }
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", 
command), exitValue);
+
+return exitValue;
   }
 
   protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
 command[0] = 
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
 command[1] = "-k";
 command[2] = 
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
 command[3] = "-z";
 command[4] = 
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
 command[5] = "-s";
 command[6] = name;
+command[7] = "-ksp";
+command[8] = 
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
 return command;
   }
 
   protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --

right, my bad, it was just parser script I had to change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127473184
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
 ---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws 
RestException {
   protected int runCommand(String[] command) throws RestException {
 ProcessBuilder pb = getProcessBuilder(command);
 pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
 try {
   process = pb.start();
   process.waitFor();
+
 } catch (Exception e) {
   throw new RestException(e);
 }
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", 
command), exitValue);
+
+return exitValue;
   }
 
   protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
 command[0] = 
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
 command[1] = "-k";
 command[2] = 
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
 command[3] = "-z";
 command[4] = 
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
 command[5] = "-s";
 command[6] = name;
+command[7] = "-ksp";
+command[8] = 
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
 return command;
   }
 
   protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --

No it's always been like that.  The parser topology start script is the 
only one that supports the ksp flag.  The enrichment and elasticsearch topology 
start scripts don't expect any input parameters.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127472746
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
 ---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws 
RestException {
   protected int runCommand(String[] command) throws RestException {
 ProcessBuilder pb = getProcessBuilder(command);
 pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
 try {
   process = pb.start();
   process.waitFor();
+
 } catch (Exception e) {
   throw new RestException(e);
 }
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", 
command), exitValue);
+
+return exitValue;
   }
 
   protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
 command[0] = 
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
 command[1] = "-k";
 command[2] = 
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
 command[3] = "-z";
 command[4] = 
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
 command[5] = "-s";
 command[6] = name;
+command[7] = "-ksp";
+command[8] = 
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
 return command;
   }
 
   protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --

Has this changed since 0.4.0 RC1? Didn't work when I tried it until ksp 
added in the start script.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127469430
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
 ---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws 
RestException {
   protected int runCommand(String[] command) throws RestException {
 ProcessBuilder pb = getProcessBuilder(command);
 pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
 try {
   process = pb.start();
   process.waitFor();
+
 } catch (Exception e) {
   throw new RestException(e);
 }
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", 
command), exitValue);
+
+return exitValue;
   }
 
   protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
 command[0] = 
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
 command[1] = "-k";
 command[2] = 
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
 command[3] = "-z";
 command[4] = 
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
 command[5] = "-s";
 command[6] = name;
+command[7] = "-ksp";
+command[8] = 
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
 return command;
   }
 
   protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --

Makes sense now.  I will remove -ksp from both Enrichment and Indexing 
start commands.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #647: METRON-1031: Management UI Cannot Start Topologies...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/647#discussion_r127466486
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
 ---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws 
RestException {
   protected int runCommand(String[] command) throws RestException {
 ProcessBuilder pb = getProcessBuilder(command);
 pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
 try {
   process = pb.start();
   process.waitFor();
+
 } catch (Exception e) {
   throw new RestException(e);
 }
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", 
command), exitValue);
+
+return exitValue;
   }
 
   protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
 command[0] = 
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
 command[1] = "-k";
 command[2] = 
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
 command[3] = "-z";
 command[4] = 
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
 command[5] = "-s";
 command[6] = name;
+command[7] = "-ksp";
+command[8] = 
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
 return command;
   }
 
   protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --

I don't think the -ksp flag is needed for the enrichment or indexing 
topologies.  This setting comes from enrichment.properties and 
elasticsearch.properties.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Metron REST - Logging Config

[Nick Allen]

Actually per step 3, this is what the property should look-like. Note that
"file:" has to precede the path.

-Dlog4j.configuration=file:/path/to/log4j.properties


On Fri, Jul 14, 2017 at 10:02 AM, Nick Allen  wrote:

> Yes, this is what I did.  Pulling this from memory, so I might be slightly
> off on the exact commands used.
>
> 1. I deleted the `log4j.properties` file from the Metron REST jar.  I
> believe I used `zip -d ...`.
>
> 2. I created my own log4j properties file.  Something like...
> `/usr/metron/0.4.1/config/log4j/metron-rest/log4j.properties`
>
> 3. I then added 
> `-Dlog4j.properties=/usr/metron/0.4.1/config/log4j/metron-rest/log4j.properties`
> as a JVM argument in one of the properties in /etc/sysconfig/metron.
>
>
> I think ultimately, we don't want to ship a log4j properties file in the
> JAR itself.
>
>
>
>
> On Fri, Jul 14, 2017 at 9:52 AM, Ryan Merriman 
> wrote:
>
>> The only way I know of is to change log4j.properites.  Did you every
>> figure
>> out a better way?
>>
>> On Tue, Jul 11, 2017 at 2:10 PM, Nick Allen  wrote:
>>
>> > How do I configure logging for Metron REST on a deployed host?
>> >
>> > Right now a log4j.properties file gets packaged into the metron-rest JAR
>> > itself.  Is there is an easy way that I am missing?
>> >
>>
>
>

Re: Metron REST - Logging Config

[Nick Allen]

Yes, this is what I did.  Pulling this from memory, so I might be slightly
off on the exact commands used.

1. I deleted the `log4j.properties` file from the Metron REST jar.  I
believe I used `zip -d ...`.

2. I created my own log4j properties file.  Something like...
`/usr/metron/0.4.1/config/log4j/metron-rest/log4j.properties`

3. I then added
`-Dlog4j.properties=/usr/metron/0.4.1/config/log4j/metron-rest/log4j.properties`
as a JVM argument in one of the properties in /etc/sysconfig/metron.


I think ultimately, we don't want to ship a log4j properties file in the
JAR itself.




On Fri, Jul 14, 2017 at 9:52 AM, Ryan Merriman  wrote:

> The only way I know of is to change log4j.properites.  Did you every figure
> out a better way?
>
> On Tue, Jul 11, 2017 at 2:10 PM, Nick Allen  wrote:
>
> > How do I configure logging for Metron REST on a deployed host?
> >
> > Right now a log4j.properties file gets packaged into the metron-rest JAR
> > itself.  Is there is an easy way that I am missing?
> >
>

Re: [REQUEST] Contributor rights in Jira

[Nick Allen]

Hi Laurens - It seems that I do not have the access that I should in JIRA.
I soon as I get my access worked out, I can take care of this for you.

On Wed, Jul 12, 2017 at 5:21 PM, Laurens Vets  wrote:

> Hello,
>
> Could a PMC member please grant my Jira account contributor rights? I'd
> like to start helping out with various smaller tasks. I promise I won't
> mess stuff up and go to IRC first for any questions/comments/additions.
> Otto has been extremely helpful there already :)
>
> Thanks,
> Laurens
>

Re: Metron REST - Logging Config

[Ryan Merriman]

The only way I know of is to change log4j.properites.  Did you every figure
out a better way?

On Tue, Jul 11, 2017 at 2:10 PM, Nick Allen  wrote:

> How do I configure logging for Metron REST on a deployed host?
>
> Right now a log4j.properties file gets packaged into the metron-rest JAR
> itself.  Is there is an easy way that I am missing?
>

[DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON-1046

I was thinking this morning that managing stellar statements in the config
json could become, and maybe is kind of unwieldy.
To that end, if in say a parser configuration I can refer to a ‘file’ in
zookeeper as an alternative, we would add the capability to execute and
manage more complex statements, and even chain multiple statements together.

These files could be shared as well.

This could be a Bad Idea™, so I thought I’d throw it out to the list.

Please check out the jira, give some thought, and comment there or on the
list or both.

O

[GitHub] metron pull request #633: METRON-1008: Updating travis to use trusty, moved ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/633


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #633: METRON-1008: Updating travis to use trusty, moved install...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/633
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #572: METRON-633: Create better logging for HbaseEnrichm...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/572


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #572: METRON-633: Create better logging for HbaseEnrichmentWrit...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/572
  
+1, thanks again for the contribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #528: METRON-838 Incorrect set of ts in FireEye parser

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/528
  
@bjigmp I just merged in https://github.com/apache/metron/pull/623, so you 
should be able to continue this now. Thanks again for the submissions!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #623: METRON-1003 ParserUtil parses dates incorrect

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/623


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r127438525
  
--- Diff: 
metron-interface/metron-alerts/src/app/utils/elasticsearch-utils.ts ---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {ColumnMetadata} from '../model/column-metadata';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+
+export class ElasticsearchUtils {
+
+  public static excludeIndexName = 'kibana';
+
+  private static createColumMetaData(properties: any, columnMetadata: 
ColumnMetadata[], seen: string[]) {
+ try {
+   let columnNames = Object.keys(properties);
+   for (let columnName of columnNames) {
+ if (seen.indexOf(columnName) === -1) {
+   seen.push(columnName);
+   columnMetadata.push(
+ new ColumnMetadata(columnName, (properties[columnName].type ? 
properties[columnName].type.toUpperCase() : ''))
--- End diff --

Forcing upper case in this model prevents sorting from working later... 
e.g. sorting by threat.triage.level leads to failed searches with: Caused by: 
java.lang.IllegalArgumentException: No mapper found for type [DOUBLE] 

The type mapping in elastic appears to be case sensitive, so squashing case 
here is going to break things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r127432908
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/elasticsearch-localstorage-impl.ts
 ---
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Observable} from 'rxjs/Rx';
+import {Headers, RequestOptions} from '@angular/http';
+
+import {HttpUtil} from '../utils/httpUtil';
+import {DataSource} from './data-source';
+import {Alert} from '../model/alert';
+import {ColumnMetadata} from '../model/column-metadata';
+import {ElasticsearchUtils} from '../utils/elasticsearch-utils';
+import {
+  ALERTS_COLUMN_NAMES, ALERTS_TABLE_METADATA, ALERTS_RECENT_SEARCH,
+  ALERTS_SAVED_SEARCH, NUM_SAVED_SEARCH
+} from '../utils/constants';
+import {ColumnNames} from '../model/column-names';
+import {ColumnNamesService} from './column-names.service';
+import {TableMetadata} from '../model/table-metadata';
+import {SaveSearch} from '../model/save-search';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+import {SearchRequest} from '../model/search-request';
+
+export class ElasticSearchLocalstorageImpl extends DataSource {
+
+  private defaultColumnMetadata = [
+new ColumnMetadata('_id', 'string'),
+new ColumnMetadata('timestamp', 'date'),
+new ColumnMetadata('source:type', 'string'),
+new ColumnMetadata('ip_src_addr', 'ip'),
+new ColumnMetadata('enrichments:geo:ip_dst_addr:country', 'string'),
+new ColumnMetadata('ip_dst_addr', 'ip'),
+new ColumnMetadata('host', 'string'),
+new ColumnMetadata('alert_status', 'string')
+  ];
+
+  getAlerts(searchRequest: SearchRequest): 
Observable {
+let url = '/search/*,-*' + ElasticsearchUtils.excludeIndexName + 
'/_search';
+let request: any  = searchRequest;
+request.query = { query_string: { query: searchRequest.query } };
+
+return this.http.post(url, request, new RequestOptions({headers: new 
Headers(this.defaultHeaders)}))
+  .map(HttpUtil.extractData)
+  .map(ElasticsearchUtils.extractAlertsData)
+  .catch(HttpUtil.handleError)
+  .onErrorResumeNext();
+  }
+
+  getAlert(index: string, type: string, alertId: string): 
Observable {
+return this.http.get('/search/' + index + '/' + type + '/' + alertId, 
new RequestOptions({headers: new Headers(this.defaultHeaders)}))
+  .map(HttpUtil.extractData);
+  }
+
+  updateAlertState(request: any) {
+return this.http.post('/search/_bulk', request, new 
RequestOptions({headers: new Headers(this.defaultHeaders)}))
+  .map(HttpUtil.extractData)
+  .catch(HttpUtil.handleError);
+  }
+
+  getDefaultAlertTableColumnNames(): Observable {
+return Observable.create(observer => {
+  
observer.next(JSON.parse(JSON.stringify(this.defaultColumnMetadata)));
+  observer.complete();
+});
+  }
+
+  getAllFieldNames(): Observable {
+let url = '_cluster/state';
--- End diff --

using cluster state here to enumerate indexes adds a lot of overhead, and 
brings back a lot of unnecessary meta data. Of course really this should 
ultimately be fronted with its own API from the REST service, but it might also 
be worth considering a simple '*' as url, which will just bring back index 
details without all the node and replica details.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #653: METRON-1040 Create Installation Instructions for t...

[MohanDV]

Github user MohanDV commented on a diff in the pull request:

https://github.com/apache/metron/pull/653#discussion_r127391226
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -69,7 +142,7 @@ More information on configuring and using the client can 
be found [here](../metr
 It is assumed that the `PROFILE_GET` client is correctly configured before 
using it.
 ```
 $ bin/stellar -z node1:2181
-[Stellar]>>> PROFILE_GET( "test", "10.0.0.1", PROFILE_FIXED(30, 
"MINUTES"))
+[Stellar]>>> PROFILE_GET( "hello-world", "10.0.0.1", PROFILE_FIXED(30, 
"MINUTES"))
 [451, 448]
--- End diff --

It will be more clear if we can explain what the output here. 
ie , we have got 451  messages with ip_src_addr in the first interval of 
15mins and 448 in the next interval ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #653: METRON-1040 Create Installation Instructions for t...

[MohanDV]

Github user MohanDV commented on a diff in the pull request:

https://github.com/apache/metron/pull/653#discussion_r127390120
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -6,36 +6,94 @@ This is achieved by summarizing the streaming telemetry 
data consumed by Metron
 
 Any field contained within a message can be used to generate a profile.  A 
profile can even be produced by combining fields that originate in different 
data sources.  A user has considerable power to transform the data used in a 
profile by leveraging the Stellar language. A user only need configure the 
desired profiles and ensure that the Profiler topology is running.
 
+* [Installation](#installation)
 * [Getting Started](#getting-started)
 * [Creating Profiles](#creating-profiles)
 * [Configuring the Profiler](#configuring-the-profiler)
 * [Examples](#examples)
 * [Implementation](#implementation)
 
-## Getting Started
+## Installation
+
+Follow these instructions to install the Profiler.  This assumes that core 
Metron has already been installed and validated.  
+
+1. Build the Metron RPMs by [following these 
instructions](../../metron-deployment#rpm).  
 
-This section will describe the steps required to get your first profile 
running.
+You may have already built the Metron RPMs when core Metron was 
installed.
+
+```
+$ find metron-deployment/ -name "metron-profiler*.rpm"
+
metron-deployment//packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.4.1-201707131420.noarch.rpm
+```
+
+1. Copy the Profiler RPM to the installation host.  
+
+The installation host must be the same host on which core Metron was 
installed.  Depending on how you installed Metron, the Profiler RPM might have 
already been copied to this host with the other Metron RPMs.
+
+```
+[root@node1 ~]# find /localrepo/  -name "metron-profiler*.rpm"
+/localrepo/metron-profiler-0.4.0-201707112313.noarch.rpm
+```
 
-1. Stand-up a Metron environment.  For this example, we will use the 
'Quick Dev' environment.  Follow the instructions included with [Quick 
Dev](../../metron-deployment/vagrant/quick-dev-platform) or build your own.
+1. Install the RPM.
+
+```
+[root@node1 ~]# rpm -ivh metron-profiler-*.noarch.rpm
+Preparing...
### [100%]
+   1:metron-profiler
### [100%]
+```
+
+```
+[root@node1 ~]# rpm -ql metron-profiler
+/usr/metron
+/usr/metron/0.4.1
+/usr/metron/0.4.1/bin
+/usr/metron/0.4.1/bin/start_profiler_topology.sh
+/usr/metron/0.4.1/config
+/usr/metron/0.4.1/config/profiler.properties
+/usr/metron/0.4.1/flux
+/usr/metron/0.4.1/flux/profiler
+/usr/metron/0.4.1/flux/profiler/remote.yaml
+/usr/metron/0.4.1/lib
+/usr/metron/0.4.1/lib/metron-profiler-0.4.0-uber.jar
+```
+
+1. Create a table within HBase that will store the profile data. The table 
name and column family must match the [Profiler's 
configuration](#configuring-the-profiler).  By default, the table is named 
`profiler` with a column family `P`.
 
-1. Create a table within HBase that will store the profile data. The table 
name and column family must match the [Profiler's 
configuration](#configuring-the-profiler).
 ```
 $ /usr/hdp/current/hbase-client/bin/hbase shell
 hbase(main):001:0> create 'profiler', 'P'
 ```
 
-1. Edit the configuration file located at 
`$METRON_HOME/config/profiler.properties`.  Change the kafka.zk and 
kafka.broker values from "node1" to the appropriate host name.  Keep the same 
port numbers:
+1. Edit the configuration file located at 
`$METRON_HOME/config/profiler.properties`.  
 ```
 kafka.zk=node1:2181
 kafka.broker=node1:6667
 ```
+Change `kafka.zk` to refer to Zookeeper in your environment.  
+Change `kafka.broker` to refer to a Kafka Broker in your environment.
+
+1. Start the Profiler topology.
+```
+$ cd $METRON_HOME
+$ bin/start_profiler_topology.sh
+```
+
+At this point the Profiler is running and consuming telemetry messages.  
We have not defined any profiles yet, so it is not doing anything very useful.  
The next section walks you through the steps to create your very first "Hello, 
World!" profile.
+
+## Getting Started
 
-1. Define the profile in a file located at 
`$METRON_HOME/config/zookeeper/profiler.json`.  The following example JSON will 
create a profile that simply counts the number of messages per `ip_src_addr`, 
during each sampling interval.
+This section will describe the steps required to get your