[jira] [Commented] (METRON-1572) Enhance KAFKA_PUT function
[ https://issues.apache.org/jira/browse/METRON-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505666#comment-16505666 ] ASF GitHub Bot commented on METRON-1572: Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/1024 +1, executed test instructions, got correct output > Enhance KAFKA_PUT function > -- > > Key: METRON-1572 > URL: https://issues.apache.org/jira/browse/METRON-1572 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > Enhance the KAFKA_PUT function to accept either a List of String or a String. > This makes it simpler to send either 1 message or a bunch of messages. > KAFKA_PUT should queue up all messages to be sent before waiting on a > response. This improves response when sending a large number of messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1024: METRON-1572 Enhance KAFKA_PUT function
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/1024 +1, executed test instructions, got correct output ---
[jira] [Commented] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505665#comment-16505665
]
ASF GitHub Bot commented on METRON-1533:
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/1025
+1 There are minor errors in your test instructions.
CONFIG_PUT("ENRICHMENTS", e, "bro") should be CONFIG_PUT("ENRICHMENT", conf,
"bro")
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1025: METRON-1533 Create KAFKA_FIND Stellar function
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/1025
+1 There are minor errors in your test instructions.
CONFIG_PUT("ENRICHMENTS", e, "bro") should be CONFIG_PUT("ENRICHMENT", conf,
"bro")
---
[jira] [Commented] (METRON-1573) Enhance KAFKA_* functions to return partition and offset details
[
https://issues.apache.org/jira/browse/METRON-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505661#comment-16505661
]
ASF GitHub Bot commented on METRON-1573:
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/1030
When I try:
[Stellar]>>> KAFKA_PUT("mytopic", "{ message! }")
[{partition=0, offset=0, topic=mytopic, timestamp=1528428506817}]
[Stellar]>>> KAFKA_PUT("mytopic", "{ message! }")
[{partition=0, offset=1, topic=mytopic, timestamp=1528428552606}]
[Stellar]>>> KAFKA_TAIL("mytopic")
[]
[Stellar]>>> KAFKA_GET("mytopic")
[]
So nothing is pulled back. But the topic and the data does exist:
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic mytopic
--zookeeper localhost:2181 --from-beginning
{ message! }
{ message! }
> Enhance KAFKA_* functions to return partition and offset details
>
>
> Key: METRON-1573
> URL: https://issues.apache.org/jira/browse/METRON-1573
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The KAFKA_* functions currently simply return the value of the message.
> There are often times when it would be useful to get more detailed
> information including the message partition, offset, key, etc.
> The functions should be enhanced to allow a user to optionally get a more
> detailed view.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1030: METRON-1573 Enhance KAFKA_* functions to return partitio...
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/1030
When I try:
[Stellar]>>> KAFKA_PUT("mytopic", "{ message! }")
[{partition=0, offset=0, topic=mytopic, timestamp=1528428506817}]
[Stellar]>>> KAFKA_PUT("mytopic", "{ message! }")
[{partition=0, offset=1, topic=mytopic, timestamp=1528428552606}]
[Stellar]>>> KAFKA_TAIL("mytopic")
[]
[Stellar]>>> KAFKA_GET("mytopic")
[]
So nothing is pulled back. But the topic and the data does exist:
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic mytopic
--zookeeper localhost:2181 --from-beginning
{ message! }
{ message! }
---
[jira] [Commented] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
[ https://issues.apache.org/jira/browse/METRON-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505414#comment-16505414 ] ASF GitHub Bot commented on METRON-1609: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1056 Thanks @merrimanr. Good catch. I will fix that. > Elasticsearch settings in Ambari should not be required if Solr is the indexer > -- > > Key: METRON-1609 > URL: https://issues.apache.org/jira/browse/METRON-1609 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: Screen Shot 2018-06-06 at 12.34.13 PM.png, Screen Shot > 2018-06-06 at 12.37.08 PM.png > > > When a user deploys Metron with Solr, the Mpack requires that the user > populate the Index Settings > Elasticsearch Hosts field to continue, even > when deploying a Metron cluster that indexes to Solr. > The other Elasticsearch specific fields on the Index Settings page are also > required, although defaults are provided and thus the user is not required to > enter anything here. If you remove the provided defaults, you can see that > each of the fields are required, even when Elasticsearch will not be used. > * The Elasticsearch related properties under the Ambari 'Index settings' tab > should only be required if Elasticsearch is chosen as the indexer. > * The Solr related properties under the 'Index Settings' tab should only be > required if Solr is chosen as the indexer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1056 Thanks @merrimanr. Good catch. I will fix that. ---
[jira] [Commented] (METRON-1601) Rename metaalert "alert" nested field to "metron_alert" to avoid collision
[ https://issues.apache.org/jira/browse/METRON-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505407#comment-16505407 ] ASF GitHub Bot commented on METRON-1601: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1049 +1 by inspection. > Rename metaalert "alert" nested field to "metron_alert" to avoid collision > -- > > Key: METRON-1601 > URL: https://issues.apache.org/jira/browse/METRON-1601 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Currently we add a nested field called "alert" to our ES templates to enable > meta alert querying. This field name is too common and should be something > like "metron_alert" instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1049: METRON-1601: Rename metaalert "alert" nested field to "m...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1049 +1 by inspection. ---
[jira] [Created] (METRON-1610) Evaluate removing ConfiguredIndexingBolt
Michael Miklavcic created METRON-1610: - Summary: Evaluate removing ConfiguredIndexingBolt Key: METRON-1610 URL: https://issues.apache.org/jira/browse/METRON-1610 Project: Metron Issue Type: Task Reporter: Michael Miklavcic Changes in [https://github.com/apache/metron/pull/1045] potentially removed any need for ConfiguredIndexingBolt. This ticket is to verify no remaining bolts utilize this class and potentially remove it from the code base. https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/bolt/ConfiguredIndexingBolt.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
[ https://issues.apache.org/jira/browse/METRON-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505387#comment-16505387 ] ASF GitHub Bot commented on METRON-1609: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1056 I spun this up in full dev and the only issue I could find was that starting REST doesn't throw an error when a required parameter is missing. The action succeeds but the app won't start and Ambari will still report it as down. Other than that everything else worked. > Elasticsearch settings in Ambari should not be required if Solr is the indexer > -- > > Key: METRON-1609 > URL: https://issues.apache.org/jira/browse/METRON-1609 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Attachments: Screen Shot 2018-06-06 at 12.34.13 PM.png, Screen Shot > 2018-06-06 at 12.37.08 PM.png > > > When a user deploys Metron with Solr, the Mpack requires that the user > populate the Index Settings > Elasticsearch Hosts field to continue, even > when deploying a Metron cluster that indexes to Solr. > The other Elasticsearch specific fields on the Index Settings page are also > required, although defaults are provided and thus the user is not required to > enter anything here. If you remove the provided defaults, you can see that > each of the fields are required, even when Elasticsearch will not be used. > * The Elasticsearch related properties under the Ambari 'Index settings' tab > should only be required if Elasticsearch is chosen as the indexer. > * The Solr related properties under the 'Index Settings' tab should only be > required if Solr is chosen as the indexer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1056 I spun this up in full dev and the only issue I could find was that starting REST doesn't throw an error when a required parameter is missing. The action succeeds but the app won't start and Ambari will still report it as down. Other than that everything else worked. ---
[jira] [Commented] (METRON-1555) Update REST to run YARN and MR jobs
[ https://issues.apache.org/jira/browse/METRON-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505266#comment-16505266 ] ASF GitHub Bot commented on METRON-1555: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1019 I added a link to the discuss thread to the description. There is also a link in the top-level Jira: https://issues.apache.org/jira/browse/METRON-1554. The latest commit removes the Service classes so that only a sample controller remains. I added a comment to that controller stating it should be replaced. There was a Jira created for service development and API design: https://issues.apache.org/jira/browse/METRON-1559. Hopefully that makes it clearer that API design is out of scope for this PR. I can also add a comment to that Jira once this PR gets accepted. I feel like spinning up and testing this in full dev should be part of the review. Most of the changes here are Ambari and Maven related. To test this properly there will need to be some way to submit a MR job (any job) from REST, hence the throwaway sample controller. If you want to create your own sample controller and test that way I can remove the one in this PR. Is there an easier option I'm not thinking of? Maintaining a needs addressed list sounds like a good idea. I'm not clear on how this would work in practice though. Where would we keep it? Would Jira tasks serve this purpose? This seems like it could be applicable to all tasks in this feature branch (all feature branches really) so I don't think this is the appropriate place to discuss it. Can we move it back to a discuss thread so you can clarify and expand on your idea? I think it needs some more thought if we're to adopt it as a process. > Update REST to run YARN and MR jobs > --- > > Key: METRON-1555 > URL: https://issues.apache.org/jira/browse/METRON-1555 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > This task involves enabling REST to submit YARN or MR jobs. We will likely > need to: > * update Maven dependencies to include YARN and MR libraries in the > classpath and resolve any version conflicts > * update REST start script to include properties required for YARN > * update the MPack for any additional setup work (create user HDFS directory > for example) and properties needed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1019 I added a link to the discuss thread to the description. There is also a link in the top-level Jira: https://issues.apache.org/jira/browse/METRON-1554. The latest commit removes the Service classes so that only a sample controller remains. I added a comment to that controller stating it should be replaced. There was a Jira created for service development and API design: https://issues.apache.org/jira/browse/METRON-1559. Hopefully that makes it clearer that API design is out of scope for this PR. I can also add a comment to that Jira once this PR gets accepted. I feel like spinning up and testing this in full dev should be part of the review. Most of the changes here are Ambari and Maven related. To test this properly there will need to be some way to submit a MR job (any job) from REST, hence the throwaway sample controller. If you want to create your own sample controller and test that way I can remove the one in this PR. Is there an easier option I'm not thinking of? Maintaining a needs addressed list sounds like a good idea. I'm not clear on how this would work in practice though. Where would we keep it? Would Jira tasks serve this purpose? This seems like it could be applicable to all tasks in this feature branch (all feature branches really) so I don't think this is the appropriate place to discuss it. Can we move it back to a discuss thread so you can clarify and expand on your idea? I think it needs some more thought if we're to adopt it as a process. ---
[jira] [Commented] (METRON-1607) update public web site to point at 0.5.0 new release
[ https://issues.apache.org/jira/browse/METRON-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505139#comment-16505139 ] ASF GitHub Bot commented on METRON-1607: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1053 +1 Thanks @justinleet. I spun-up the site locally and poked around. Everything looks kosher. > update public web site to point at 0.5.0 new release > > > Key: METRON-1607 > URL: https://issues.apache.org/jira/browse/METRON-1607 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Update the site for the 0.5.0 release per the process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1053: METRON-1607: update public web site to point at 0.5.0 ne...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1053 +1 Thanks @justinleet. I spun-up the site locally and poked around. Everything looks kosher. ---
[jira] [Commented] (METRON-1608) Add configuration for threat.triage.field name
[ https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505071#comment-16505071 ] ASF GitHub Bot commented on METRON-1608: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1055 FYI - There is also another `source.type.field` reference that would be an easy update [here](https://github.com/apache/metron/blob/71a015fabc47b23ea0c5400fa239fee9f5c0d194/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/SearchServiceImplTest.java#L186) > Add configuration for threat.triage.field name > -- > > Key: METRON-1608 > URL: https://issues.apache.org/jira/browse/METRON-1608 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Priority: Major > > Currently there is an option for replacing '.'s with ':'s in Elasticsearch > field names. This is the default behavior. However our current version of > Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s > instead. In the DAO implementation (metaalerts specifically), the > threat.triage.field is hardcoded with ':'s and will not work properly if a > user switches to using '.'s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1055: METRON-1608: Add configuration for threat.triage.field n...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1055 FYI - There is also another `source.type.field` reference that would be an easy update [here](https://github.com/apache/metron/blob/71a015fabc47b23ea0c5400fa239fee9f5c0d194/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/SearchServiceImplTest.java#L186) ---
[GitHub] metron pull request #1055: METRON-1608: Add configuration for threat.triage....
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1055#discussion_r193850059
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/SearchServiceImpl.java
---
@@ -151,7 +152,7 @@ public GroupResponse group(GroupRequest groupRequest)
throws RestException {
String sourceTypeField = Constants.SENSOR_TYPE.replace('.', ':');
--- End diff --
Just another point of duplication. We define what the default value is
here and in `ElasticsearchMetaAlertDao`. Could be extracted out along with the
other bits.
---
[jira] [Commented] (METRON-1608) Add configuration for threat.triage.field name
[
https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505067#comment-16505067
]
ASF GitHub Bot commented on METRON-1608:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1055#discussion_r193850059
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/SearchServiceImpl.java
---
@@ -151,7 +152,7 @@ public GroupResponse group(GroupRequest groupRequest)
throws RestException {
String sourceTypeField = Constants.SENSOR_TYPE.replace('.', ':');
--- End diff --
Just another point of duplication. We define what the default value is
here and in `ElasticsearchMetaAlertDao`. Could be extracted out along with the
other bits.
> Add configuration for threat.triage.field name
> --
>
> Key: METRON-1608
> URL: https://issues.apache.org/jira/browse/METRON-1608
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently there is an option for replacing '.'s with ':'s in Elasticsearch
> field names. This is the default behavior. However our current version of
> Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s
> instead. In the DAO implementation (metaalerts specifically), the
> threat.triage.field is hardcoded with ':'s and will not work properly if a
> user switches to using '.'s.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1608) Add configuration for threat.triage.field name
[
https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505063#comment-16505063
]
ASF GitHub Bot commented on METRON-1608:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1055#discussion_r193849566
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -720,4 +719,12 @@ public int getPageSize() {
public void setPageSize(int pageSize) {
this.pageSize = pageSize;
}
+
+ private String getField(String globalConfigKey, String defaultField) {
--- End diff --
It seems like we have duplicated this logic here and in
`SearchServiceImpl`. Would it make sense to extract this out?
> Add configuration for threat.triage.field name
> --
>
> Key: METRON-1608
> URL: https://issues.apache.org/jira/browse/METRON-1608
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently there is an option for replacing '.'s with ':'s in Elasticsearch
> field names. This is the default behavior. However our current version of
> Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s
> instead. In the DAO implementation (metaalerts specifically), the
> threat.triage.field is hardcoded with ':'s and will not work properly if a
> user switches to using '.'s.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1055: METRON-1608: Add configuration for threat.triage....
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1055#discussion_r193849566
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -720,4 +719,12 @@ public int getPageSize() {
public void setPageSize(int pageSize) {
this.pageSize = pageSize;
}
+
+ private String getField(String globalConfigKey, String defaultField) {
--- End diff --
It seems like we have duplicated this logic here and in
`SearchServiceImpl`. Would it make sense to extract this out?
---
[jira] [Commented] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
[ https://issues.apache.org/jira/browse/METRON-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504993#comment-16504993 ] ASF GitHub Bot commented on METRON-1609: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1056 METRON-1609 Elasticsearch settings in Ambari should not be required if Solr is the Indexer NOTE: This PR has been submitted against the Solr feature branch; not master. ## Description When a user deploys Metron with Solr, the Mpack requires that the user populate the Index Settings > Elasticsearch Hosts field to continue, even when deploying a Metron cluster that indexes to Solr and does not use Elasticsearch. * The Elasticsearch related properties under the Ambari 'Index settings' tab should only be required if Elasticsearch is chosen as the indexer. * The Solr related properties under the 'Index Settings' tab should only be required if Solr is chosen as the indexer. ## Changes * Removed the static validation checks for each of the parameters on the 'Index Settings' tab. Static validation no longer works because validation depends on which Indexer the user has chosen. * Added a function `check_indexing_parameters` that validates the parameters based on the Indexer that has been chosen by the user on the "Indexing" tab. * The parameters are checked before any of the following actions: * Start indexing * Stop indexing * Status check indexing * Service checking indexing * Install Elasticsearch Template * Delete Elasticsearch Template * Install Kibana Dashboard * Update global properties * When updating the global properties, we only update either Solr or Elasticsearch, not both as before. * It seems logical to move the 'Indexing Settings' tab parameters to the `Index Settings` tab, especially when it on this page where the user selects the Indexer; Elasticsearch or Solr. I am not sure if there are other implications in doing this, so I did not do that as part of this PR. ## Testing 1. Start the development environment. 1. Validate that the Alerts UI is receiving alerts and that the Ambari 'Service Check' passes. **Missing Elasticsearch parameters should be caught** 1. Stop the Indexing topology in Ambari. 1. Using Ambari, delete (define a blank value for) the `metron-env/es_binary_port` parameter in Metron > Index Settings. 1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message. ``` resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Elasticsearch, missing=['metron-env/es_binary_port'] ``` 1. Attempt the Elasticsearch Template Install. This should fail with the same error message. 1. Attempt the Elasticsearch Template Delete. This should fail with the same error message. 1. Attempt the Kibana Dashboard Install. This should fail with the same error message. **Missing Elasticsearch parameters should not impact unrelated functions** 1. Restart another unrelated topology like the Profiler or Enrichment. These should restart successfully as they are not impacted by the missing Indexing parameters. **Missing Elasticsearch parameters should not impact Solr** 1. Go to Metron > Indexing and change "Random Access Search Engine" to "Solr". 1. Attempt to start the Indexing topology in Ambari. This should succeed even though we are missing Elasticsearch parameters. **Missing Solr parameters should be caught** 1. Stop the Indexing topology in Ambari. 1. Using Ambari, delete (define a blank value for) the `metron-env/solr_zookeeper_url` parameter in Metron > Index Settings. 1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message. ``` resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Solr, missing=['metron-env/solr_zookeeper_url'] ``` ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to
[GitHub] metron pull request #1056: METRON-1609 Elasticsearch settings in Ambari shou...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1056 METRON-1609 Elasticsearch settings in Ambari should not be required if Solr is the Indexer NOTE: This PR has been submitted against the Solr feature branch; not master. ## Description When a user deploys Metron with Solr, the Mpack requires that the user populate the Index Settings > Elasticsearch Hosts field to continue, even when deploying a Metron cluster that indexes to Solr and does not use Elasticsearch. * The Elasticsearch related properties under the Ambari 'Index settings' tab should only be required if Elasticsearch is chosen as the indexer. * The Solr related properties under the 'Index Settings' tab should only be required if Solr is chosen as the indexer. ## Changes * Removed the static validation checks for each of the parameters on the 'Index Settings' tab. Static validation no longer works because validation depends on which Indexer the user has chosen. * Added a function `check_indexing_parameters` that validates the parameters based on the Indexer that has been chosen by the user on the "Indexing" tab. * The parameters are checked before any of the following actions: * Start indexing * Stop indexing * Status check indexing * Service checking indexing * Install Elasticsearch Template * Delete Elasticsearch Template * Install Kibana Dashboard * Update global properties * When updating the global properties, we only update either Solr or Elasticsearch, not both as before. * It seems logical to move the 'Indexing Settings' tab parameters to the `Index Settings` tab, especially when it on this page where the user selects the Indexer; Elasticsearch or Solr. I am not sure if there are other implications in doing this, so I did not do that as part of this PR. ## Testing 1. Start the development environment. 1. Validate that the Alerts UI is receiving alerts and that the Ambari 'Service Check' passes. **Missing Elasticsearch parameters should be caught** 1. Stop the Indexing topology in Ambari. 1. Using Ambari, delete (define a blank value for) the `metron-env/es_binary_port` parameter in Metron > Index Settings. 1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message. ``` resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Elasticsearch, missing=['metron-env/es_binary_port'] ``` 1. Attempt the Elasticsearch Template Install. This should fail with the same error message. 1. Attempt the Elasticsearch Template Delete. This should fail with the same error message. 1. Attempt the Kibana Dashboard Install. This should fail with the same error message. **Missing Elasticsearch parameters should not impact unrelated functions** 1. Restart another unrelated topology like the Profiler or Enrichment. These should restart successfully as they are not impacted by the missing Indexing parameters. **Missing Elasticsearch parameters should not impact Solr** 1. Go to Metron > Indexing and change "Random Access Search Engine" to "Solr". 1. Attempt to start the Indexing topology in Ambari. This should succeed even though we are missing Elasticsearch parameters. **Missing Solr parameters should be caught** 1. Stop the Indexing topology in Ambari. 1. Using Ambari, delete (define a blank value for) the `metron-env/solr_zookeeper_url` parameter in Metron > Index Settings. 1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message. ``` resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Solr, missing=['metron-env/solr_zookeeper_url'] ``` ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed i
[jira] [Created] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
Nick Allen created METRON-1609: -- Summary: Elasticsearch settings in Ambari should not be required if Solr is the indexer Key: METRON-1609 URL: https://issues.apache.org/jira/browse/METRON-1609 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Attachments: Screen Shot 2018-06-06 at 12.34.13 PM.png, Screen Shot 2018-06-06 at 12.37.08 PM.png When a user deploys Metron with Solr, the Mpack requires that the user populate the Index Settings > Elasticsearch Hosts field to continue, even when deploying a Metron cluster that indexes to Solr. The other Elasticsearch specific fields on the Index Settings page are also required, although defaults are provided and thus the user is not required to enter anything here. If you remove the provided defaults, you can see that each of the fields are required, even when Elasticsearch will not be used. * The Elasticsearch related properties under the Ambari 'Index settings' tab should only be required if Elasticsearch is chosen as the indexer. * The Solr related properties under the 'Index Settings' tab should only be required if Solr is chosen as the indexer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504815#comment-16504815 ] ASF GitHub Bot commented on METRON-1587: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 @justinleet is the documentation I added comprehensive enough for this PR? > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 @justinleet is the documentation I added comprehensive enough for this PR? ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504792#comment-16504792 ] ASF GitHub Bot commented on METRON-1594: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 I assume you are talking to @nickwallen there @mmiklavc ? > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 I assume you are talking to @nickwallen there @mmiklavc ? ---
[jira] [Commented] (METRON-1608) Add configuration for threat.triage.field name
[
https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504786#comment-16504786
]
ASF GitHub Bot commented on METRON-1608:
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1055
METRON-1608: Add configuration for threat.triage.field name
## Contributor Comments
This PR adds a configuration to the global config for the
`threat.triage.score` field name, similar to what was done in
https://github.com/apache/metron/pull/1010 for `source.type`, minus the UI
changes.
I also opportunistically fixed a bug where the `source.type` field name
wasn't being read from the global config. Normally this would be a separate PR
but the fix overlaps with what was done here so I included it. I also added a
constant for the `source.type.field` property.
## Changes
- Added constants for `source.type.field` and `threat.triage.field` (would
`threat.triage.score.field` be better?)
- Added getter/setter for AccessConfig in the ElasticsearchDao (necessary
for testing)
- Refactored the default threat triage field name in
ElasticsearchMetaAlertDao to match source type field name pattern
- Added a `getField` method that gets the field name from the global config
or returns the default field name if not found
- Added unit and integration tests for both `source.type.field` and
`threat.triage.field`
## Testing
Testing this in full dev requires changing our indexing topology to use
fields with '.'s rather than ':'s. To do that:
1. Stop the snort sensor with `service sensor-stubs stop snort`
2. Delete the snort ES index with `curl -XDELETE
http://node1:9200/snort_index*`
3. Edit the template at
`/var/lib/ambari-agent/cache/common-services/METRON/0.5.0/package/files/snort_index.template`
by changing `source:type` to `source.type` and `threat:triage:*score` to
`threat.triage.*score`
4. Reinstall the template using the "Elasticsearch Template Install" in
Ambari
5. Update the global config with the new field names:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"es.clustername": "metron",
"es.ip": "node1:9300",
"es.date.format": ".MM.dd.HH",
"parser.error.topic": "indexing",
"update.hbase.table": "metron_update",
"update.hbase.cf": "t",
"es.client.settings": {
"client.transport.ping_timeout": "500s"
},
"profiler.client.period.duration": "15",
"profiler.client.period.duration.units": "MINUTES",
"user.settings.hbase.table": "user_settings",
"user.settings.hbase.cf": "cf",
"bootstrap.servers": "node1:6667",
"geo.hdfs.file": "/apps/metron/geo/default/GeoLite2-City.mmdb.gz",
"source.type.field":"source.type",
"threat.triage.score.field":"threat.triage.score"
}' 'http://node1:8082/api/v1/global/config'
```
6. Update the snort indexing config to not DEDOT the field names:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"hdfs": {
"index": "snort",
"batchSize": 1,
"enabled": true
},
"elasticsearch": {
"index": "snort",
"batchSize": 1,
"enabled": true,
"fieldNameConverter": "NOOP"
},
"solr": {
"index": "snort",
"batchSize": 1,
"enabled": true
}
}' 'http://node1:8082/api/v1/sensor/indexing/config/snort'
```
7. Start the snort sensor again with `service sensor-stubs start snort`
8. Wait for data to appear in the snort ES index and then perform a search:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"facetFields":[],
"from": 0,
"indices": [
"snort"
],
"query": "*",
"size": 5
}' 'http://node1:8082/api/v1/search/search'
```
9. Grab a guid from one of the snort search results and create a metaalert:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "1c082f1a-ad3a-4e46-ae1e-828b73c9a016",
"sensorType": "snort"
}
],
"groups": [
"string"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
10. Now find the metaalert from the guid returned in the previous step:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "1ad8be85-3164-4be9-a773-379abb044d0a",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
Before this PR there were a couple problems. First the metaalert had the
`source:type` field even though we've switched to "so
[GitHub] metron pull request #1055: METRON-1608: Add configuration for threat.triage....
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1055
METRON-1608: Add configuration for threat.triage.field name
## Contributor Comments
This PR adds a configuration to the global config for the
`threat.triage.score` field name, similar to what was done in
https://github.com/apache/metron/pull/1010 for `source.type`, minus the UI
changes.
I also opportunistically fixed a bug where the `source.type` field name
wasn't being read from the global config. Normally this would be a separate PR
but the fix overlaps with what was done here so I included it. I also added a
constant for the `source.type.field` property.
## Changes
- Added constants for `source.type.field` and `threat.triage.field` (would
`threat.triage.score.field` be better?)
- Added getter/setter for AccessConfig in the ElasticsearchDao (necessary
for testing)
- Refactored the default threat triage field name in
ElasticsearchMetaAlertDao to match source type field name pattern
- Added a `getField` method that gets the field name from the global config
or returns the default field name if not found
- Added unit and integration tests for both `source.type.field` and
`threat.triage.field`
## Testing
Testing this in full dev requires changing our indexing topology to use
fields with '.'s rather than ':'s. To do that:
1. Stop the snort sensor with `service sensor-stubs stop snort`
2. Delete the snort ES index with `curl -XDELETE
http://node1:9200/snort_index*`
3. Edit the template at
`/var/lib/ambari-agent/cache/common-services/METRON/0.5.0/package/files/snort_index.template`
by changing `source:type` to `source.type` and `threat:triage:*score` to
`threat.triage.*score`
4. Reinstall the template using the "Elasticsearch Template Install" in
Ambari
5. Update the global config with the new field names:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"es.clustername": "metron",
"es.ip": "node1:9300",
"es.date.format": ".MM.dd.HH",
"parser.error.topic": "indexing",
"update.hbase.table": "metron_update",
"update.hbase.cf": "t",
"es.client.settings": {
"client.transport.ping_timeout": "500s"
},
"profiler.client.period.duration": "15",
"profiler.client.period.duration.units": "MINUTES",
"user.settings.hbase.table": "user_settings",
"user.settings.hbase.cf": "cf",
"bootstrap.servers": "node1:6667",
"geo.hdfs.file": "/apps/metron/geo/default/GeoLite2-City.mmdb.gz",
"source.type.field":"source.type",
"threat.triage.score.field":"threat.triage.score"
}' 'http://node1:8082/api/v1/global/config'
```
6. Update the snort indexing config to not DEDOT the field names:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"hdfs": {
"index": "snort",
"batchSize": 1,
"enabled": true
},
"elasticsearch": {
"index": "snort",
"batchSize": 1,
"enabled": true,
"fieldNameConverter": "NOOP"
},
"solr": {
"index": "snort",
"batchSize": 1,
"enabled": true
}
}' 'http://node1:8082/api/v1/sensor/indexing/config/snort'
```
7. Start the snort sensor again with `service sensor-stubs start snort`
8. Wait for data to appear in the snort ES index and then perform a search:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"facetFields":[],
"from": 0,
"indices": [
"snort"
],
"query": "*",
"size": 5
}' 'http://node1:8082/api/v1/search/search'
```
9. Grab a guid from one of the snort search results and create a metaalert:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"alerts": [
{
"guid": "1c082f1a-ad3a-4e46-ae1e-828b73c9a016",
"sensorType": "snort"
}
],
"groups": [
"string"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
10. Now find the metaalert from the guid returned in the previous step:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"guid": "1ad8be85-3164-4be9-a773-379abb044d0a",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
Before this PR there were a couple problems. First the metaalert had the
`source:type` field even though we've switched to "source.type" in the global
config. Second, the `threat.triage.score` field in the metaalert would be 0
because the wrong threat triage field name is used to get scores from alerts.
With this PR the metalaert should correctly have the `source.type` field and
[jira] [Commented] (METRON-1606) Add a 'wrap' to incoming messages in the metron json parser
[
https://issues.apache.org/jira/browse/METRON-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504782#comment-16504782
]
ASF GitHub Bot commented on METRON-1606:
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1054
METRON-1606 Add capability to wrap json message as entity arrays
This PR adds the ability to configure the JSONMap parser to wrap messages
when using JSON Path queries in an entity with an array.
For example, if the json 'document' is
`{"foo": "one"}, {"bar","two"}`
you need to have it wrapped to work with it with jsonpath.
This PR allows this, resulting in
`{"configured name or default of message" : [ {{"foo": "one"},
{"bar","two"}]}`
Tests where added, along with sample data for integration, and a new sample
configuration.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron json-wrap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1054.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1054
commit e991d8d43865149f957d530624a69539de5d7bb2
Author: Otto Fowler
Date: 2018-06-07T14:52:07Z
METRON-1606 Add capability to wrap json message as entity arrays
> Add a 'wrap' to incoming messages in the metron json parser
> ---
>
> Key: METRON-1606
> URL: https://issues.apache.org/jira/browse/METRON-1606
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> For many use cases, the user may not have control over the format of the json
> that is coming in. Some services send 'documents' with many atomic json
> entries, that are not in a coherent array, and cannot be referenced or parsed.
>
> It would be useful to simply have the option for the JSONMap parser to be
> configured to 'wrap' the incoming document in an array and holding entity.
> This would allow it to be parsed and addressed by JSONPath.
>
> {data},
> {data},
> {data},
>
> wraps to:
>
> { "wrapName" : [
> {data},
> {data},
> {data}
> ]
> }
>
> that can be then referenced by a jsonPath statement : $.wrapperName[*] or
> something more complex
>
> So you would configure the parser
> {code:java}
> wrapIncomming : 'true' # This wraps the incomming bytes in a json construct,
> { NAME : [ CONTENT ] }{code}
> {code:java}
> .wrapContructName : 'messages' # default nam
[GitHub] metron pull request #1054: METRON-1606 Add capability to wrap json message a...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1054
METRON-1606 Add capability to wrap json message as entity arrays
This PR adds the ability to configure the JSONMap parser to wrap messages
when using JSON Path queries in an entity with an array.
For example, if the json 'document' is
`{"foo": "one"}, {"bar","two"}`
you need to have it wrapped to work with it with jsonpath.
This PR allows this, resulting in
`{"configured name or default of message" : [ {{"foo": "one"},
{"bar","two"}]}`
Tests where added, along with sample data for integration, and a new sample
configuration.
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron json-wrap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1054.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1054
commit e991d8d43865149f957d530624a69539de5d7bb2
Author: Otto Fowler
Date: 2018-06-07T14:52:07Z
METRON-1606 Add capability to wrap json message as entity arrays
---
[jira] [Commented] (METRON-1568) Stellar should have a _ special variable which returns the message in map form
[ https://issues.apache.org/jira/browse/METRON-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504774#comment-16504774 ] ASF GitHub Bot commented on METRON-1568: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1021 > Stellar should have a _ special variable which returns the message in map form > -- > > Key: METRON-1568 > URL: https://issues.apache.org/jira/browse/METRON-1568 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > In order to support functions which operate on the whole message, we should > have a special variable (_, keeping with the vaguely scala theme) which can > return the entire underlying message. This map should be immutable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1021: METRON-1568: Stellar should have a _ special vari...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1021 ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504758#comment-16504758 ] ASF GitHub Bot commented on METRON-1594: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 I didn't know you were still actively reviewing. The last comment I received besides a +1 was "that looks great, thanks." We do have unit and integration tests around all of this in addition to the batch performance testing performed on enrichments. They use the same KafkaWriter. Below is verbatim from our bylaws. This is also the handling that I've personally seen on most PRs that don't have dissenting/outstanding concerns to be addressed. > Code Change > > A change made to a codebase of the project requires lazy consensus of active committers other than the author of the patch. The code can be committed after the first +1. > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 I didn't know you were still actively reviewing. The last comment I received besides a +1 was "that looks great, thanks." We do have unit and integration tests around all of this in addition to the batch performance testing performed on enrichments. They use the same KafkaWriter. Below is verbatim from our bylaws. This is also the handling that I've personally seen on most PRs that don't have dissenting/outstanding concerns to be addressed. > Code Change > > A change made to a codebase of the project requires lazy consensus of active committers other than the author of the patch. The code can be committed after the first +1. ---
[jira] [Created] (METRON-1608) Add configuration for threat.triage.field name
Ryan Merriman created METRON-1608: - Summary: Add configuration for threat.triage.field name Key: METRON-1608 URL: https://issues.apache.org/jira/browse/METRON-1608 Project: Metron Issue Type: Bug Reporter: Ryan Merriman Currently there is an option for replacing '.'s with ':'s in Elasticsearch field names. This is the default behavior. However our current version of Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s instead. In the DAO implementation (metaalerts specifically), the threat.triage.field is hardcoded with ':'s and will not work properly if a user switches to using '.'s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504724#comment-16504724 ] ASF GitHub Bot commented on METRON-1594: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 My comment was just about calling out a possible need for more shutdown orchestration. I am not reviewing. > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1045 My comment was just about calling out a possible need for more shutdown orchestration. I am not reviewing. ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504718#comment-16504718 ] ASF GitHub Bot commented on METRON-1594: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1045 I was still going through it. Not sure where @ottobackwards landed on this. Besides the open TODO comment, I am not sure how much testing we did around the Profiler or testing changes made to the new configuration elements that you added. I didn't see a test plan around much of this besides spinning up Full Dev, which doesn't exercise the Profiler. I guess its just a community courtesy that we've followed, no matter how long its open. Kind of like how I followed up on #1036. I don't want to make a stink of it. Like I said, its just a courtesy I suppose. > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1045 I was still going through it. Not sure where @ottobackwards landed on this. Besides the open TODO comment, I am not sure how much testing we did around the Profiler or testing changes made to the new configuration elements that you added. I didn't see a test plan around much of this besides spinning up Full Dev, which doesn't exercise the Profiler. I guess its just a community courtesy that we've followed, no matter how long its open. Kind of like how I followed up on #1036. I don't want to make a stink of it. Like I said, its just a courtesy I suppose. ---
[jira] [Commented] (METRON-1607) update public web site to point at 0.5.0 new release
[ https://issues.apache.org/jira/browse/METRON-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504706#comment-16504706 ] ASF GitHub Bot commented on METRON-1607: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1053 METRON-1607: update public web site to point at 0.5.0 new release ## Contributor Comments Regenerated the site. Updated the download links to point to 0.5.0 instead of 0.4.2. Removed MD5 sum from links because we will no longer publish it per https://www.apache.org/dev/release-distribution#sigs-and-sums https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge contains instructions for running the site locally. Ensure the download links are correct, MD5 is missing, and that the site book link is correct. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron METRON-1607 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1053.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1053 commit c79d878d36e5aa9382c73b70a0733c98b7ee6910 Author: justinjleet Date: 2018-06-07T13:43:19Z Updating website. Removing MD5 sum > update public web site to point at 0.5.0 new release > > > Key: METRON-1607 > URL: https://issues.apache.org/jira/browse/METRON-1607 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Update the site for the 0.5.0 release per the process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1053: METRON-1607: update public web site to point at 0...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1053 METRON-1607: update public web site to point at 0.5.0 new release ## Contributor Comments Regenerated the site. Updated the download links to point to 0.5.0 instead of 0.4.2. Removed MD5 sum from links because we will no longer publish it per https://www.apache.org/dev/release-distribution#sigs-and-sums https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge contains instructions for running the site locally. Ensure the download links are correct, MD5 is missing, and that the site book link is correct. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron METRON-1607 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1053.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1053 commit c79d878d36e5aa9382c73b70a0733c98b7ee6910 Author: justinjleet Date: 2018-06-07T13:43:19Z Updating website. Removing MD5 sum ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504697#comment-16504697 ] ASF GitHub Bot commented on METRON-1594: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 @nickwallen The PR has been up for 7 days, I addressed all community comments days ago, and no comments appeared to be dissenting. Was there a concern or issue you had before this was merged? > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 @nickwallen The PR has been up for 7 days, I addressed all community comments days ago, and no comments appeared to be dissenting. Was there a concern or issue you had before this was merged? ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504658#comment-16504658 ] ASF GitHub Bot commented on METRON-1594: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1045 @mmiklavc FYI You've got a TODO comment in `ConfiguredIndexingBolt.java` that seems like something you wanted to address before merging. Usually good to at least give a chance for all those who have chimed in on a PR to sign-off before merging it in. At least that's what I try to do as a courtesy. > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1045 @mmiklavc FYI You've got a TODO comment in `ConfiguredIndexingBolt.java` that seems like something you wanted to address before merging. Usually good to at least give a chance for all those who have chimed in on a PR to sign-off before merging it in. At least that's what I try to do as a courtesy. ---
[jira] [Created] (METRON-1607) update public web site to point at 0.5.0 new release
Justin Leet created METRON-1607: --- Summary: update public web site to point at 0.5.0 new release Key: METRON-1607 URL: https://issues.apache.org/jira/browse/METRON-1607 Project: Metron Issue Type: Task Reporter: Justin Leet Assignee: Justin Leet Update the site for the 0.5.0 release per the process at: https://cwiki.apache.org/confluence/display/METRON/Release+Process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1604) Add power pc to OS family for the HCP management pack repo info
[ https://issues.apache.org/jira/browse/METRON-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504335#comment-16504335 ] ASF GitHub Bot commented on METRON-1604: GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1052 METRON-1604 : Add power pc to OS family for the HCP management pack repo info ## Contributor Comments Add power pc to OS family for the HCP management pack repo info ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1052.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1052 commit 7b503d7d2f9e08bf7f223aae05b036efeb18dc16 Author: Mohan Venkateshaiah Date: 2018-06-07T07:14:49Z METRON-1604 : Add power pc to OS family for the HCP management pack repo info > Add power pc to OS family for the HCP management pack repo info > > > Key: METRON-1604 > URL: https://issues.apache.org/jira/browse/METRON-1604 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Minor > > Add power pc to OS family for the HCP management pack repo info otherwise we > don't get "redhat-ppc" under stack selection while installing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1052: METRON-1604 : Add power pc to OS family for the H...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1052 METRON-1604 : Add power pc to OS family for the HCP management pack repo info ## Contributor Comments Add power pc to OS family for the HCP management pack repo info ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1052.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1052 commit 7b503d7d2f9e08bf7f223aae05b036efeb18dc16 Author: Mohan Venkateshaiah Date: 2018-06-07T07:14:49Z METRON-1604 : Add power pc to OS family for the HCP management pack repo info ---
