[GitHub] incubator-metron pull request #540: METRON-869 Include build instructions fo...
Github user anandsubbu closed the pull request at: https://github.com/apache/incubator-metron/pull/540 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #678: METRON-1076: Management UI status check always reports th...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/678 +1 (non-binding) Validated that with this fix, Ambari displays the correct status as opposed to the earlier behavior. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #688: METRON-1094: MaaS will not start due to classpath error r...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/688 I'm +1 (non-binding) Fired up a full dev and was able to test that `maas_service.sh` starts up fine. No errors seen in the YARN application logs. A few observations though: a) I am not sure if its my setup, but the maas uber jar, maas_deploy and maas_service scripts were missing from `$METRON_HOME/bin` folder on my quick dev. I had to SCP `metron-maas-service-0.4.1-archive.tar.gz` and untar to get these binaries. Is this expected? b) The `maas_service.sh` script can only be started as *hdfs* user. Please let me know if this needs to be updated in the README, and I can add this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/680 +1 (non-binding). Validated the latest fix to be working great. Thank you, @dlyle65535 . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/680 Sounds great, @dlyle65535 ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/680 Hi @dlyle65535 can it be a two step validation as follows: a) We first verify if we are able to hit the storm URL, if that fails -> then we mark the service as stopped. b) If the storm URL is up, and the service/topology (indexing, enrichment, parsers etc) is missing, then we mark the service as stopped. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #583: METRON-949 Change from error to warning ...
GitHub user anandsubbu opened a pull request: https://github.com/apache/incubator-metron/pull/583 METRON-949 Change from error to warning message for Metron REST in Ambari service_advisor ## Contributor Comments Reduced the level to WARN in service_advisor.py. Steps to validate: - Fire up Ambari wizard after installing Mpack - In the Assign Masters page, Metron REST should now show a 'WARN' icon instead of 'ERROR' I have not carried out the full suite of tests, since this is a minor rename from ERROR to WARN. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [NA] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [NA] Have you written or updated unit tests and or integration tests to verify your changes? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [NA] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [NA] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-949 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/583.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #583 commit e7c158d24ccb27f0973332d76c598233381f6d3e Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-05-12T09:26:16Z Changed from ERROR to WARN --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #583: METRON-949 Change from error to warning message for Metro...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/583 Could I get some love for this PR? :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #583: METRON-949 Change from error to warning ...
Github user anandsubbu closed the pull request at: https://github.com/apache/incubator-metron/pull/583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #583: METRON-949 Change from error to warning ...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/incubator-metron/pull/583 METRON-949 Change from error to warning message for Metron REST in Ambari service_advisor ## Contributor Comments Reduced the level to WARN in service_advisor.py. Steps to validate: - Fire up Ambari wizard after installing Mpack - In the Assign Masters page, Metron REST should now show a 'WARN' icon instead of 'ERROR' I have not carried out the full suite of tests, since this is a minor rename from ERROR to WARN. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [NA] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [NA] Have you written or updated unit tests and or integration tests to verify your changes? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [NA] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [NA] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-949 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/583.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #583 commit e7c158d24ccb27f0973332d76c598233381f6d3e Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-05-12T09:26:16Z Changed from ERROR to WARN --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #583: METRON-949 Change from error to warning message for Metro...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/583 Thank you, @merrimanr . Can you please help merge this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #583: METRON-949 Change from error to warning message for Metro...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/583 @merrimanr , this is because of the following reason: In the "Assign Masters" page of the Ambari deployment wizard, there is a check for the Metron REST host to have storm supervisor installed. This is currently displayed as an Error message. However in the flow of the wizard, the storm supervisor selection can only be done in the subsequent screen. So, this message should be changed into a Warning instead. This will be inline with the check that we have for Metron Parsers host co-location with Storm supervisor, which is also a WARN. See [here](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py#L65). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #583: METRON-949 Change from error to warning message for Metro...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/583 @merrimanr, let me know if the above comment answers your question. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #604: METRON-978: Management UI init script uses hard-coded MET...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/604 +1 (non-binding) since this ensures that the Metron UI service starts up properly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/759 +1 (non-binding) Indeed, this was a tricky one. Thank you @nickwallen for your root cause and fix!! I spun up a 12 node cluster with this patch, kerberized the cluster and confirmed that all of the topologies come up fine (esp. indexing and profiler). ---
[GitHub] metron pull request #723: METRON-1141: NameError: metronProfilerHost is not ...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/723 METRON-1141: NameError: metronProfilerHost is not defined in stack_advisor.py ## Contributor Comments ** Testing Done ** - Modified `service_advisor.py` on the Ambari host with the mpack installed - Restarted Ambari and navigated through deployment wizard - "Next" button is now seen to be enabled on the Assign Masters page. - No errors seen in `ambari-server.log` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1141 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/723.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #723 commit b14c25c7e6fb114886338056d4a015c712e98813 Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-08-30T07:35:55Z Rename metronProfilerMaster to metronProfilerHost --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #731: METRON-1146 Refactor for ParserExtensionConfig and Rest g...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/731 Hi @ottobackwards , I guess you wanted to name the PR title as METRON-1136 instead of METRON-1146? ---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362796
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -506,29 +506,27 @@ public Object apply(List strings) {
}
}
- @Stellar( name = "PARSE_JSON_STRING"
+ @Stellar(name = "JSON_PARSE"
, description = "Returns a JSON object for the specified JSON
string"
, params = {
"str - the JSON String to convert, may be null"
}
, returns = "an Object containing the parsed JSON string"
)
- public static class ParseJsonString extends BaseStellarFunction {
+ public static class JsonParse extends BaseStellarFunction {
@Override
public Object apply(List strings) {
if (strings == null || strings.size() == 0) {
-throw new IllegalArgumentException("[PARSE_JSON_STRING] incorrect
arguments. Usage: PARSE_JSON_STRING ");
+throw new IllegalArgumentException("[JSON_PARSE] incorrect
arguments. Usage: JSON_PARSE ");
}
--- End diff --
Added check
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362926
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
+return (HashMap) JSONUtils.INSTANCE.load((String)
strings.get(0), Object.class);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+ }
+ return new ParseException("Unable to parse JSON string");
+}
+ }
+
+ @Stellar(name = "JSON_TO_LIST"
+ , description = "Returns a List object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a List object containing the parsed JSON string"
+ )
+ public static class JsonToList extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_LIST] incorrect
arguments. Usage: JSON_TO_LIST ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
--- End diff --
Sure, done!
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362754
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
--- End diff --
Added check.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362605
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
--- End diff --
Yup, that makes sense @ottobackwards . Cleaned up now.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362883
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
--- End diff --
Sure, done!
---
[GitHub] metron issue #727: METRON-1146: Add ability to parse JSON string into JSONOb...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/727 Hi @nickwallen thank you. Sure, I see value in simplifying the naming. How does `TO_JSON_OBJECT`, `TO_JSON_MAP` and `TO_JSON_LIST` sound? ---
[GitHub] metron issue #727: METRON-1146: Add ability to parse JSON string into JSONOb...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/727 Thanks very much @ottobackwards ! @nickwallen , pushed latest commit with the functions renamed. Please have a look. ---
[GitHub] metron pull request #719: METRON-1137: Build RPM for Metron MaaS as a part o...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/719 METRON-1137: Build RPM for Metron MaaS as a part of rpm-docker packaging ## Contributor Comments Fixed metron.spec and the pom.xml to now copy and generate metron-maas-service RPM. **Testing Steps** 1. Go to folder `metron-deployment/packaging/docker/rpm-docker` 2. Run `mvn clean package -Pbuild-rpms` 3. Once the command completes, go to `metron-deployment/packaging/docker/rpm-docker/RPMS/noarch` and the metron-maas-service RPM should be present. **Testing Done** - Verified testing steps above are working fine. - Copied metron-maas-service RPM on an openstack cluster and validated that install is going through fine. - Validated installed RPM contents as follows: ``` [root@metron-12 ~]# rpm -ql metron-maas-0.4.1-201708290906 /usr/metron /usr/metron/0.4.1 /usr/metron/0.4.1/bin /usr/metron/0.4.1/bin/maas_deploy.sh /usr/metron/0.4.1/bin/maas_service.sh /usr/metron/0.4.1/lib/metron-maas-service-0.4.1-uber.jar ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [NA] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1137 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/719.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #719 commit b0a03fba303decbae1ab818c58e9eec02919ee4a Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-08-29T12:01:35Z Build metron-maas-service RPM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #714: METRON-1128: MAAS_GET_ENDPOINT - Unable to resolve...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/714#discussion_r135785927 --- Diff: metron-analytics/metron-maas-service/README.md --- @@ -126,16 +126,20 @@ Now let's install some prerequisites: Start Squid via `service squid start` Now that we have flask and jinja, we can create a mock DGA service to deploy with MaaS: -* Download the files in [this](https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a) gist into the `/root/mock_dga` directory -* Make `rest.sh` executable via `chmod +x /root/mock_dga/rest.sh` +* Download the files in [this](https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a) gist into the `$HOME/mock_dga` directory +* Make `rest.sh` executable via `chmod +x $HOME/mock_dga/rest.sh` This service will treat `yahoo.com` and `amazon.com` as legit and everything else as malicious. The contract is that the REST service exposes an endpoint `/apply` and returns back JSON maps with a single key `is_malicious` which can be `malicious` or `legit`. ## Deploy Mock DGA Service via MaaS +The following presumes that you are a logged in as a user who has a +home directory in HDFS under `/user/$USER`. If you do not, please create one +and ensure the permissions are set appropriate. --- End diff -- Would it be good to add the following example steps picking 'metron' as a user? ``` su - hdfs -c "hadoop fs -mkdir /user/metron" su - hdfs -c "hadoop fs -chown metron:metron /user/metron" ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #714: METRON-1128: MAAS_GET_ENDPOINT - Unable to resolve...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/714#discussion_r135786862 --- Diff: metron-analytics/metron-maas-service/README.md --- @@ -126,16 +126,20 @@ Now let's install some prerequisites: Start Squid via `service squid start` Now that we have flask and jinja, we can create a mock DGA service to deploy with MaaS: -* Download the files in [this](https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a) gist into the `/root/mock_dga` directory -* Make `rest.sh` executable via `chmod +x /root/mock_dga/rest.sh` +* Download the files in [this](https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a) gist into the `$HOME/mock_dga` directory +* Make `rest.sh` executable via `chmod +x $HOME/mock_dga/rest.sh` This service will treat `yahoo.com` and `amazon.com` as legit and everything else as malicious. The contract is that the REST service exposes an endpoint `/apply` and returns back JSON maps with a single key `is_malicious` which can be `malicious` or `legit`. ## Deploy Mock DGA Service via MaaS +The following presumes that you are a logged in as a user who has a +home directory in HDFS under `/user/$USER`. If you do not, please create one +and ensure the permissions are set appropriate. + Now let's start MaaS and deploy the Mock DGA Service: * Start MaaS via `$METRON_HOME/bin/maas_service.sh -zq node1:2181` -* Start one instance of the mock DGA model with 512M of memory via `$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga -v 1.0 -ni 1` +* Start one instance of the mock DGA model with 512M of memory via `$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp $HOME/mock_dga -hmp /user/$USER/models -mo ADD -m 512 -n dga -v 1.0 -ni 1` --- End diff -- I bumped into the same issue that @mmiklavc described. Turns out that the HDFS folder owner and group needs to be owned by the same user (e.g. metron) and then the LIST command worked fine. After this, I was able to follow the steps and everything worked fine. I was able to get the squid indices generated with the `is_alert` and `is_malicious` fields set correctly. +1 (non-binding) from my side. Thank you @cestella ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136954740
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/StringFunctionsTest.java
---
@@ -449,4 +450,68 @@ public void testCountMatches() throws Exception {
Assert.assertTrue(thrown);
}
+
--- End diff --
Thanks @ottobackwards . Have now used multiline strings.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136954647
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
Updated README
---
[GitHub] metron issue #723: METRON-1141: NameError: metronProfilerHost is not defined...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/723 Closing this pull request, since it is the same as https://github.com/apache/metron/pull/722 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #723: METRON-1141: NameError: metronProfilerHost is not ...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/723 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #807: METRON-1263: Start Alerts UI service after Metron ...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/807 METRON-1263: Start Alerts UI service after Metron REST ## Contributor Comments Currently the Alerts UI service starts up before any of the Metron services does. Ideally, it should be started after the Metron REST service. The fix is to update role_command_order.json in the Ambari Mpack so that Alerts UI starts after Metron REST service. This commit also adds the other Alerts UI dependency to other service checks. **Steps to verify** - Build mpack and start Ambari wizard based installation - Verify that the Alerts UI service starts after the Metron REST service has started. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1263 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/807.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #807 commit c2c54aa7a0bb47d7190e6e2e33489e33952d43d4 Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-10-19T08:04:08Z Update role_command_order to inlcude Alerts UI dependencies ---
[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/841 +1 (non-binding), btw ---
[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/841 As always, very crisp @nickwallen ! My +1 (non-binding) still holds. ---
[GitHub] metron pull request #846: METRON-1328: Enhance platform-info.sh script to ch...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/846 METRON-1328: Enhance platform-info.sh script to check if docker daemon is running ## Contributor Comments Run the `docker info` command along with checking the version. **Case A - Docker daemon running** - Same as existing behavior. **Case B - Docker daemon not running** - The `docker info` command would error out with the following message: ``` Cannot connect to the Docker daemon. Is the docker daemon running on this host? ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1328 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/846.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #846 commit ba71013009e36136d2dd5bd994d9dbaf11b84f71 Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-11-22T13:59:15Z Run docker info command to check for status ---
[GitHub] metron issue #846: METRON-1328: Enhance platform-info.sh script to check if ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/846 Hi @ottobackwards , thanks for your review. I prefixed a 'WARNING' to the message text so it would catch the eye of the user. Also, I removed the else bit for docker running scenario, since this will fail for the case where docker is not installed at all. If I retain the else block, it would report saying 'Docker daemon is running'. So, in summary here are the cases: 1. Docker installed and running would print: ```bash Docker version 1.12.1, build 23cf638, experimental ``` 2. Docker installed, but not running would print: ```bash Docker version 1.12.1, build 23cf638, experimental WARNING: Docker daemon is not running ``` 3. Docker not installed would print: ```bash -bash: docker: command not found ``` Let me know if this looks okay. ---
[GitHub] metron issue #852: METRON-1239 Drop extra dev environments
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/852 Hi @nickwallen , I found one more reference to quick dev - [here](https://github.com/nickwallen/metron/blob/7145b06fff9f4ecc8e63d9cdd56ed955a49e676d/metron-interface/metron-rest/src/test/resources/README.vm#L208). Also, I could see that the contents within [site/current-book](https://github.com/nickwallen/metron/tree/7145b06fff9f4ecc8e63d9cdd56ed955a49e676d/site/current-book) needs an update, since it does not have the changes. Does `mvn site` need to be run in order to fix this? ---
[GitHub] metron issue #852: METRON-1239 Drop extra dev environments
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/852 > I was confused about that README.vm file. Any idea what that is for? I am not entirely sure, @nickwallen . I could find a lot of similarities between this README.vm and the main Metron REST README.md though. https://github.com/nickwallen/metron/blob/7145b06fff9f4ecc8e63d9cdd56ed955a49e676d/metron-interface/metron-rest/README.md ---
[GitHub] metron issue #852: METRON-1239 Drop extra dev environments
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/852 +1 (non-binding) @nickwallen . This is a much needed fix since it is now straight-forward to anyone new and wanting to try Metron. ---
[GitHub] metron pull request #850: METRON-1335: Install metron-maas-service RPM as a ...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/850 METRON-1335: Install metron-maas-service RPM as a part of the full-dev deployment ## Contributor Comments Modified mpack metainfo.xml to install metron-maas-service RPM as well. **Testing Done** - [x] Deployed full dev and validated that the metron-maas-service RPM is shown as installed from the Ambari UI - [x] Did `vagrant ssh` to the full dev box. Confirmed that the RPM is installed. ``` [vagrant@node1 ~]$ rpm -qa | grep metron-maas metron-maas-service-0.4.2-201711271930.noarch ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1335 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/850.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #850 commit 313b6e611a9ed27fc54800c0c54a0ac6e93f959a Author: Anand Subramanian <asubraman...@hortonworks.com> Date: 2017-11-28T06:01:08Z Install metron-maas-service RPM ---
[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/850 bump. Any feedback on this pull request? ---
[GitHub] metron issue #839: METRON-1311 Service Check Should Check Elasticsearch Inde...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/839
+1 (non-binding) works fine, and a much needed fix as well ! If for some
reason, we were not able to install the templates during the startup, it would
be caught by the service check.
Validated this on a full dev environment. For a successful service check, I
see this now:
```
2017-11-10 15:48:06,150 - Checking Elasticsearch templates for Indexing
2017-11-10 15:48:06,151 - Execute['curl -s -XGET
"http://node1:9200/_template/yaf_index; | grep -o yaf_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
yaf_index
yaf_index
2017-11-10 15:48:06,343 - Execute['curl -s -XGET
"http://node1:9200/_template/error_index; | grep -o error_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
error_index
error_index
2017-11-10 15:48:06,385 - Execute['curl -s -XGET
"http://node1:9200/_template/snort_index; | grep -o snort_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
snort_index
snort_index
2017-11-10 15:48:06,433 - Execute['curl -s -XGET
"http://node1:9200/_template/bro_index; | grep -o bro_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
bro_index
bro_index
2017-11-10 15:48:06,491 - Execute['curl -s -XGET
"http://node1:9200/_template/metaalert_index; | grep -o metaalert_index']
{'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'],
'tries': 3, 'user': 'metron', 'try_sleep': 5}
metaalert_index
```
And the service check fails when I delete the ES template file.
```
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 78, in
ServiceCheck().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 51, in service_check
indexing_cmds.service_check(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 264, in service_check
self.check_elasticsearch_templates()
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 179, in check_elasticsearch_templates
metron_service.execute(cmd, user=self.__params.metron_user,
err_msg=err_msg)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/metron_service.py",
line 257, in execute
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Missing Elasticsearch index
template: name=yaf_index
```
@nickwallen - I see two conflicting files for the commit. Guess this would
be need to be resolved first?
---
[GitHub] metron issue #839: METRON-1311 Service Check Should Check Elasticsearch Inde...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/839 +1 (non-binding). Validated again on full-dev. Works fine, @nickwallen ! ---
[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/841
Hi @nickwallen
> I am not sure exactly what the problem is, but the same condition occurs
in master. I would call this a pre-existing condition that we can handle with a
separate PR.
>
> Of course that is just my opinion and you or others may feel differently.
Let me know what you think.
>
Sure, I am fine by it, since I did not see any anomalies due to this error.
Also, I was trying to test the same with CentOS 7.4 image and ran into some
challenges. I am constantly facing a 'Timed out' issue when attempting 'vagrant
up' on my local system.
```
➜ centos-7.4 git:(master) ✗ vagrant up
Bringing machine 'source' up with 'virtualbox' provider...
Bringing machine 'sink' up with 'virtualbox' provider...
==> source: Box 'bento/centos-7.4' could not be found. Attempting to find
and install...
source: Box Provider: virtualbox
source: Box Version: >= 0
==> source: Loading metadata for box 'bento/centos-7.4'
source: URL: https://atlas.hashicorp.com/bento/centos-7.4
==> source: Adding box 'bento/centos-7.4' (v201710.25.0) for provider:
virtualbox
source: Downloading:
https://vagrantcloud.com/bento/boxes/centos-7.4/versions/201710.25.0/providers/virtualbox.box
==> source: Box download is resuming from prior download progress
==> source: Successfully added box 'bento/centos-7.4' (v201710.25.0) for
'virtualbox'!
==> source: Importing base box 'bento/centos-7.4'...
==> source: Matching MAC address for NAT networking...
==> source: Checking if box 'bento/centos-7.4' is up to date...
==> source: Setting the name of the VM: centos-74_source_1511196258710_55009
==> source: Clearing any previously set network interfaces...
==> source: Preparing network interfaces based on configuration...
source: Adapter 1: nat
source: Adapter 2: hostonly
==> source: Forwarding ports...
source: 22 (guest) => (host) (adapter 1)
==> source: Running 'pre-boot' VM customizations...
==> source: Booting VM...
==> source: Waiting for machine to boot. This may take a few minutes...
source: SSH address: 127.0.0.1:
source: SSH username: vagrant
source: SSH auth method: private key
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.
If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
➜ centos-7.4 git:(master) ✗ vagrant status
Current machine states:
sourcerunning (virtualbox)
sink not created (virtualbox)
```
I did a fresh import of the 'bento/centos-7.4' box image multiple times
after cleaning up (`vagrant box remove bento/centos-7.4` followed by fresh
import), but I ran into the same timeout issue. Also tried bumping up
`config.vm.boot_timeout` in `Vagrantfile` to 600, but to no avail.
Further I am also not able to vagrant ssh into the 'source' VM despite it
showing as running.
Does it look like I missed something in my configuration?
---
[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/841
Hi @nickwallen , tested on the vagrant fastcapa deployment. Deployed fine,
and I was able to see this message as well:
```
TASK [debug]
***
ok: [source] => {
"msg": "Successfully received packets sent from pcap-replay!"
}
```
During the deployment, I noticed the following message. I was not sure if
this can be ignored or needs to be looked at.
```
TASK [fastcapa : Restart for modified kernel params]
***
fatal: [sink]: FAILED! => {"changed": false, "failed": true,
"module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE", "parsed":
false}
...ignoring
```
---
[GitHub] metron issue #829: METRON-1296 Full Dev Fails to Deploy Index Templates
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/829 @nickwallen , for the record full dev spun up fine with the fix. Thank you! ---
[GitHub] metron issue #829: METRON-1296 Full Dev Fails to Deploy Index Templates
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/829 Hi @nickwallen , thank you for the fix. It is my bad, I should have added the line `commands = IndexingCommands(params)` before the `try` block in my previous fix :(. I am under way validating and will post my observation. However, given that the indexing templates are a must have for the alerts UI to work properly, should we remove the try/expect block and allow the install to fail in the event it was not successfully installed? Or do you think we should WARN and move on? ---
[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/850 Sure, that would be great. ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 You're right @nickwallen , it wouldn't work. I reworked on the patch and here are the latest changes: - moved the ES template installation back to the Indexing "start" method - use a flag file to ensure the template installation runs only once. And so any start/restart of Indexing topology does not get affected. - enclosed the template installation in a try/except block so that we warn and move on in the event of ES service not running. I ran the following test scenarios to validate the changes: **Full dev deployment** - [x] Verified that the Indexing Start step also installs the ES templates. - [x] Verified that the flag files are created under $METRON_HOME/config - [x] Restarted Indexing service and verified that template install is not called. **12-node deployment** - [x] Verified that the Indexing Start step also installs the ES templates. - [x] Verified that the flag files are created under $METRON_HOME/config - [x] Restarted Indexing service and verified that template install is not called. - [x] Stopped the ES service manually before the Indexing service starts and confirmed that the Indexing service startup proceeds successfully with a warning message indicating the template could not be installed ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 If the install fails the first time due to ES being down, then we WARN and move on. If ES service comes up later, the admin needs to install the templates subsequently by using 'Ambari -> Metron -> Service Actions -> Elasticsearch Template Install' option. Alternatively s/he can also stop the Indexing service and start again so the template will be installed (Note that calling "restart" on the Indexing service will not trigger the template installation) ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 I have added this to the Elasticsearch README in my recent commit. Please see here: https://github.com/apache/metron/pull/817/files#diff-325aea0d364d12c8637eef347ebcfca6 The preceding section to the template installation talks about the update required to the templates. I hence found this to be a better option than hosting it in the Metron Indexing README. ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147681110
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Reworded per your suggestion.
I believe the README section covers the current behavior with my change. If
you are talking about the scenarios noted in the PR description, then these are
captured in the existing ES README under the section [Using Metron with
Elasticsearch
2.x](https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x).
Please let me know which specific scenario you are referring to and I would be
happy to include them.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147708307
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Okay, let me try to explain more using examples:
**Scenario 1 - Happy Path**
* Fresh install
* ES service up and running
* When the Indexing service comes up, it also installs the ES templates
* Admin can start ingesting into sensors, launch alerts UI and everything
works
**Scenario 2 - ES service down**
* Fresh install
* For some reason, the ES service is down when the Indexing service is
coming up
* We log a warning message in the Ambari install logs, and the Indexing
service starts up fine.
* Once the ES service issue is resolved, the Admin needs to install the ES
templates manually before s/he can start ingesting into sensors. This can be
done in two ways:
1) Using the Ambari UI -> Services -> Metron -> Service Actions ->
Elasticsearch Template Install
2) By stopping the Metron Indexing service from Ambari UI, and starting it
again so that it can trigger the piece of code to install the template.
Now, from a documentation perspective, point 2 above is counter intuitive
IMHO, since it would not make sense to ask the Admin to stop/start Indexing
service in order to have the ES template installed. I have hence documented
only the first option--which is also the same way it is done presently.
Let me know if this helps clarify.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147699187
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Ah okay, this comment and review is the same scenario described in the
README as:
> It is possible that Elasticsearch service is not available when the
Metron Indexing Service startup, in that case the Elasticsearch template will
not be installed. For such a scenario, an Admin can install the template
manually from the Ambari UI by following the below flow:
>
> Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch
Template Install
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147714561
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Fair enough, added both methods in the latest doc change.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147737598
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -86,8 +86,13 @@ rm ${SENSOR}.template
The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
-It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed. For such a scenario, an Admin can install the template manually from
the Ambari UI by following the below flow:
+It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed.
+For such a scenario, an Admin can have the template installed in two ways:
+
+_Method 1_ - Manually from the Ambari UI by following the flow:
Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch
Template Install
+_Method 2_ - Stop the Metron Indexing service, and start it again from
Ambari UI.
+
--- End diff --
Sure, added this as well @ottobackwards
---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 Hi @ottobackwards , let me know what you think. ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/817#discussion_r147587694 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py --- @@ -82,6 +82,13 @@ def start(self, env, upgrade_type=None): self.configure(env) commands = IndexingCommands(params) commands.start_indexing_topology(env) +# Install elasticsearch templates +try: +if not commands.is_elasticsearch_template_installed(): +self.elasticsearch_template_install(env) +commands.set_elasticsearch_template_installed() --- End diff -- Thank you @nickwallen . Sure, done! ---
[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/864 I did another 12-node deployment on Centos 7 with this PR (bypassed the kafka issue by installing Kafka broker on all nodes). The fix worked just perfect. Thanks much @nickwallen ! +1 (non-binding) ---
[GitHub] metron pull request #807: METRON-1263: Start Alerts UI service after Metron ...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/807#discussion_r145701351 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json --- @@ -13,16 +13,18 @@ "METRON_PROFILER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START", "METRON_ENRICHMENT-INSTALL"], "METRON_REST-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "METRON_PARSERS-INSTALL","METRON_INDEXING-INSTALL","METRON_ENRICHMENT-INSTALL"], "METRON_MANAGEMENT_UI-START" : ["METRON_REST-START"], +"METRON_ALERTS_UI-START" : ["METRON_REST-START"], -"STORM_REST_API-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP"], -"STORM_UI_SERVER-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP"], +"STORM_REST_API-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP", "METRON_ALERTS_UI-STOP"], +"STORM_UI_SERVER-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP", "METRON_ALERTS_UI-STOP"], -"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START", "METRON_ENRICHMENT_MASTER-START", "METRON_INDEXING-START", "METRON_PROFILER-START", "METRON_REST-START", "METRON_MANAGEMENT_UI-START"], +"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START", "METRON_ENRICHMENT_MASTER-START", "METRON_INDEXING-START", "METRON_PROFILER-START", "METRON_REST-START", "METRON_MANAGEMENT_UI-START", "METRON_ALERTS_UI-STOP"], --- End diff -- Ugh, mea culpa :/. Thanks @nickwallen ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 Restarting travis ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/817 METRON-1283: Install Elasticsearch template as a part of the mpack startup scripts ## Contributor Comments For a Metron multi-node deployment using mpack, the Elasticsearch template is required to be installed manually post-setup. These templates are required for the proper working of, for e.g. the Alerts UI. In the event that these templates are not installed, and if data is ingested, these would not be shown in the Alerts UI, since there would be missing fields without the template files (E.g. snort alert indices are not displayed in the Alerts UI, since it is missing the "alerts" field from the mapping). In such a case, one needs to install the templates, delete all indices for the given parser and re-ingest data again into the parser for it to appear in the Alerts UI. Further, the indices from all the parsers will have to be deleted and re-ingested again which could be a tedious job in the event that this step was missed out by chance. I have also seen other ill-effects from having stale indices for parsers that was created before template install. While documenting the template installation is a good practice, nothing would more failsafe than installing the template as a part of the mpack startup scripts itself. I have added the ES template install step just before starting the Indexing topology, since I found this to be the closest related step when this could be done. Do let me know if this needs to be moved to a different point in the startup, and I can make the change. **Testing Done** * Built mpack with the changes * Deployed a 12-node Metron setup using the mpack * Validated that the start Indexing topology step also installs the Elasticsearch template files. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1283 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/817.patch To close this pull request, make a commit to your master
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/817 Hi @ottobackwards , made the change to remove the ES template installation from 'load_web_template' role for vagrant deployments. Fired up full-dev with the change and it comes up fine. Was able to launch and view alerts in the UI. Hi @nickwallen - sure, what you are saying makes sense to me. To address point 1, I have moved the ES template installation step into the Indexing "Install" method, so this would be called only once. This would also not meddle with users making customization to the templates subsequently. Point 2 is partly addressed with the recent commit in that it takes care of ES service not running scenario. However I am not sure about how we need to proceed with a user _not_ using Elasticsearch at all. If I have understand right, this is something that would require changes and considerations to packaging and installation, not limited to this pull request alone. Please correct me if I am missing something. ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/817 ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/817 METRON-1283: Install Elasticsearch template as a part of the mpack startup scripts ## Contributor Comments For a Metron multi-node deployment using mpack, the Elasticsearch template is required to be installed manually post-setup. These templates are required for the proper working of, for e.g. the Alerts UI. In the event that these templates are not installed, and if data is ingested, these would not be shown in the Alerts UI, since there would be missing fields without the template files (E.g. snort alert indices are not displayed in the Alerts UI, since it is missing the "alerts" field from the mapping). In such a case, one needs to install the templates, delete all indices for the given parser and re-ingest data again into the parser for it to appear in the Alerts UI. Further, the indices from all the parsers will have to be deleted and re-ingested again which could be a tedious job in the event that this step was missed out by chance. I have also seen other ill-effects from having stale indices for parsers that was created before template install. While documenting the template installation is a good practice, nothing would more failsafe than installing the template as a part of the mpack startup scripts itself. I have added the ES template install step just before starting the Indexing topology, since I found this to be the closest related step when this could be done. Do let me know if this needs to be moved to a different point in the startup, and I can make the change. **Testing Done** * Built mpack with the changes * Deployed a 12-node Metron setup using the mpack * Validated that the start Indexing topology step also installs the Elasticsearch template files. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1283 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/817.patch To close this pull request, make a commit to your master
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/888 METRON-1389: Zeppelin notebook import does not work with Ambari 2.6 ## Contributor Comments In recent versions of Ambari, [Shiro Authentication](https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html) is enabled on the Zeppelin server, which requires a user/password authentication in order to interact with Zeppelin REST API. This pull request adds support for using the Shiro auth credentials and thus being able to import Zeppelin notebooks. Backward compatibility with non-authenticated Zeppelin has been maintained as well so that the full dev platform (which runs on an older Ambari version) is not broken. **Steps to test the PR** * In Ambari UI, go to Services -> Metron -> 'Service Actions' dropdown -> 'Zeppelin Notebook Import' * Validate that the Metron notebooks are imported by launching the Zeppelin UI (Service -> Zeppelin Notebook -> 'Quick Links' dropdown -> Zeppelin UI) **Testing Done** * Ran the changes against Ambari ver 2.6.0.0 (which uses Shiro authentication) and verified that notebooks are imported properly * Ran the changes against Ambari ver 2.4.2.0 (which does not uses Shiro authentication) and verified that notebooks are imported properly ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1389 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/888.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #888 commit e767022e6c87922f7a0c3ca143b7482548a0908c Author: Anand Subramanian <asubramanian@...> Date: 2018-01-05T06:10:04Z Add Shiro authentication support ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159430163 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,127 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at `/localrepo`. + +On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands. + +``` +yum install createrepo +mkdir /localrepo +cd /localrepo +createrepo +``` + +1. Metron will now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. --- End diff -- In the "Assign Masters" page, if the Kafka broker, Parsers host co-location or Alerts/Mgt UI co-location requirements are not met, then Ambari indicated this using an error balloon like:  If one of the above were not met and the user clicks "Next", then Ambari presents a validation issues dialog:  Now, the user can always hit "Continue Anyway" in the above dialog and move on with the wizard. This way, Ambari does not enforce the co-location. The other client related requirements are indicated as a warning icon (since the client installation selection happens in the next step of the wizard). (Thank you for reading through my long winding comment) ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159426522 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,127 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at `/localrepo`. + +On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands. + +``` +yum install createrepo +mkdir /localrepo +cd /localrepo +createrepo +``` + +1. Metron will now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + --- End diff -- Just to give some more context on the co-location requirements... The point no. 3 above is due to the way the [service_advisor](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py#L78) is designed. This checks for all of the Parsers, Enrichment, Indexing and Profiler to be on the same host. ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/869 +1 (non-binding). Looks good @nickwallen ! ---
[GitHub] metron issue #894: METRON-1326: Metron deploy with Kerberos fails on Ambari ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/894 +1 thank you @mmiklavc. Spun up a 12-node Centos 7 cluster with the latest fix and validated the following: * Did 'Stop All Services' -> no issues seen with ES and Kibana services stop any longer. * Kerberized the cluster -> the process completed successfully and all services came up fine. The issues reported in the JIRA ticket now stand resolved. ---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161735380
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in
{metron_config_zeppelin_path}'))
+
+# With Ambari 2.5+, Zeppelin server is enabled to work with Shiro
authentication, which requires user/password
+# for authentication (see
https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for
details).
+ses = requests.session()
+
+# Check if authentication is enabled on the Zeppelin server
+try:
+conn =
ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+# Establish connection if authentication is enabled
+try:
+# The following credentials are created at install time by
Ambari at /etc/zeppelin/conf/shiro.ini
+# when Shiro auth is enabled on the Zeppelin server
+zeppelin_payload = {'userName': 'admin', 'password' :
'admin'}
--- End diff --
Hi @cestella and @ottobackwards - I found a way to get the Shiro INI
content from Ambari and parse the admin credentials. Please have a look at my
latest commit.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/888 METRON-1389: Zeppelin notebook import does not work with Ambari 2.6 ## Contributor Comments In recent versions of Ambari, [Shiro Authentication](https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html) is enabled on the Zeppelin server, which requires a user/password authentication in order to interact with Zeppelin REST API. This pull request adds support for using the Shiro auth credentials and thus being able to import Zeppelin notebooks. Backward compatibility with non-authenticated Zeppelin has been maintained as well so that the full dev platform (which runs on an older Ambari version) is not broken. **Steps to test the PR** * In Ambari UI, go to Services -> Metron -> 'Service Actions' dropdown -> 'Zeppelin Notebook Import' * Validate that the Metron notebooks are imported by launching the Zeppelin UI (Service -> Zeppelin Notebook -> 'Quick Links' dropdown -> Zeppelin UI) **Testing Done** * Ran the changes against Ambari ver 2.6.0.0 (which uses Shiro authentication) and verified that notebooks are imported properly * Ran the changes against Ambari ver 2.4.2.0 (which does not uses Shiro authentication) and verified that notebooks are imported properly ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1389 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/888.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #888 commit e767022e6c87922f7a0c3ca143b7482548a0908c Author: Anand Subramanian <asubramanian@...> Date: 2018-01-05T06:10:04Z Add Shiro authentication support commit b8780df0a0f1231a18597302302a3bd247b7dfd6 Author: Anand Subramanian <asubramanian@...> Date: 2018-01-16T11:52:02Z Get admin user creds from Zeppelin shiro config in Ambari commit 3a799da66ad1c9568f5fcb85cdc8ea8de8c9872a Author: Anand Subramanian
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/888 ---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r163022818
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
---
@@ -150,12 +152,48 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in
{metron_config_zeppelin_path}'))
+
+# With Ambari 2.5+, Zeppelin server is enabled to work with Shiro
authentication, which requires user/password
+# for authentication (see
https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for
details).
+ses = requests.session()
--- End diff --
Sure thing @nickwallen . Please see latest commit where I have refactored
this into a separate method.
Further, I tested the latest changes on a full-dev environment (w/Ambari
2.4.2) as well as a 12-node cluster (w/Ambari 2.6.0) to confirm the notebooks
are imported fine.
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/888#discussion_r161955419 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py --- @@ -162,9 +163,21 @@ def zeppelin_notebook_import(self, env): --- End diff -- In Ambari, the Shiro INI content appears as a template like here: https://user-images.githubusercontent.com/20395490/35026106-0a038a28-fb6f-11e7-93e3-371e5fa353d0.png;> And this can be fetched using Ambari API as follows: http://:8080/api/v1/clusters/cl1/configurations?type=zeppelin-shiro-ini=version1 Which I am fetching in the `status_params.py` as ``` zeppelin_shiro_ini_content = config['configurations']['zeppelin-shiro-ini']['shiro_ini_content'] ``` I found it easy to parse this from the content. Could you please clarify how I can use `dict` to get this info? ---
[GitHub] metron issue #894: METRON-1326: Metron deploy with Kerberos fails on Ambari ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/894 > I'd like to treat this as distinct from the status check if that seems reasonable to you. I have to admit, I didn't even know the service check piece existed until now. Sure, it makes sense to address this as a separate issue. ---
[GitHub] metron pull request #894: METRON-1326: Metron deploy with Kerberos fails on ...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/894#discussion_r161185438
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/elastic_slave.py
---
@@ -56,8 +59,27 @@ def status(self, env):
import params
env.set_params(params)
Logger.info('Check status of Elasticsearch data node')
-status_cmd = "service elasticsearch status"
-Execute(status_cmd)
+
+# return codes defined by LSB
+#
http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html
+cmd = ('service', 'elasticsearch', 'status')
+rc, out = shell.call(cmd, sudo=True, quiet=False)
+
+if rc == 3:
--- End diff --
Same comment as Kibana
---
[GitHub] metron pull request #894: METRON-1326: Metron deploy with Kerberos fails on ...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/894#discussion_r161185471
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py
---
@@ -93,7 +96,26 @@ def status(self, env):
Logger.info("Status of the Master")
-Execute("service kibana status")
+# return codes defined by LSB
+#
http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html
+ cmd = ('service', 'kibana', 'status')
+ rc, out = shell.call(cmd, sudo=True, quiet=False)
+
+if rc == 3:
--- End diff --
Same comment as Kibana
---
[GitHub] metron pull request #894: METRON-1326: Metron deploy with Kerberos fails on ...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/894#discussion_r161184707
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/elastic_master.py
---
@@ -56,8 +59,28 @@ def status(self, env):
import params
env.set_params(params)
Logger.info('Check status of Elasticsearch master node')
-status_cmd = "service elasticsearch status"
-Execute(status_cmd)
+
+# return codes defined by LSB
+#
http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html
+cmd = ('service', 'elasticsearch', 'status')
+
+rc, out = shell.call(cmd, sudo=True, quiet=False)
+
+if rc == 3:
--- End diff --
On my 12-node CentOS 7, when I hit 'Stop Services', Kibana service stop
still failed with the following error:
```
stderr: /var/lib/ambari-agent/data/errors-753.txt
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py",
line 153, in
Kibana().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 332, in execute
self.execute_prefix_function(self.command_name, 'after', env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 350, in execute_prefix_function
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 398, in after_stop
status_method(env)
File
"/var/lib/ambari-agent/cache/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py",
line 118, in status
raise ExecutionFailed(err_msg, rc, out)
resource_management.core.exceptions.ExecutionFailed: Execution of 'service
kibana status' returned 2
stdout: /var/lib/ambari-agent/data/output-753.txt
2018-01-12 10:08:11,166 - Stop Kibana Master
2018-01-12 10:08:11,166 - Execute['service kibana stop'] {}
2018-01-12 10:08:12,251 - Waiting for actual component stop
2018-01-12 10:08:12,251 - Status of the Master
2018-01-12 10:08:12,251 - call[('service', 'kibana', 'status')] {'sudo':
True, 'quiet': False}
2018-01-12 10:08:12,285 - call returned (2, 'kibana is not running')
Command failed after 1 tries
```
This check only looks for exit status 3 alone.
I made a local chance to treat any non-zero exit status as
`ComponentIsNotRunning` and it worked fine.
---
[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/907 Hi @cestella , I did a 12-node deploy on CentOS 7 with this patch. Post-kerberization, I noticed the following errors in Metron REST. Is this a related issue or a different one? ``` 18/01/25 10:16:02 ERROR boot.SpringApplication: Application startup failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'alertController': Unsatisfied dependency expressed through field 'alertService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'alertServiceImpl' defined in URL [jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaServiceImpl' defined in URL [jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaProducer' defined in class path resourc e [org/apache/metron/rest/config/KafkaConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.kafka.clients.producer.KafkaProducer]: Factory method 'kafkaProducer' threw exception; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371) at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175) at org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:29) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'alertServiceImpl' defined in URL [jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaServiceImpl' defined in URL [jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean
[GitHub] metron issue #888: METRON-1389: Zeppelin notebook import does not work with ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/888 @ottobackwards @cestella and @nickwallen - any other feedback on this PR? The travis failures seems to be in an unrelated piece of code. ---
[GitHub] metron issue #904: METRON-1400: Elasticsearch service check fails in Ambari
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/904
+1 works fine @mmiklavc .
Verified on CentOS 7 multinode cluster. Simulated the problem first and
then ran with the fix.
Here is a sample output with the fix in place.
```
2018-01-24 14:17:58,928 - Running Elastic search service check against
metron-12.openstacklocal:9200
2018-01-24 14:17:58,929 - Checking cluster health
2018-01-24 14:17:58,929 - Execute['curl -sS -XGET
'http://metron-12.openstacklocal:9200/_cluster/health?wait_for_status=green=120s'
| grep '"status":"green"''] {'logoutput': True, 'tries': 5, 'try_sleep': 10}
{"cluster_name":"metron","status":"green","timed_out":false,"number_of_nodes":12,"number_of_data_nodes":11,"active_primary_shards":16,"active_shards":32,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
2018-01-24 14:17:58,967 - Elasticsearch service check successful
Command completed successfully!
```
---
[GitHub] metron pull request #907: METRON-1427: Add support for storm 1.1 and hdp 2.6
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/907#discussion_r163600738
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/service_check.py
---
@@ -32,7 +32,7 @@ def service_check(self, env):
doc = '{"name": "Ambari Smoke test"}'
index = "ambari_smoke_test"
-Logger.info("Running Elastic search service check",
file=sys.stdout)
+Logger.info("Running Elastic search service check")
--- End diff --
Hi @cestella , would it be a good idea to leave this unchanged since
@mmiklavc has PR #904 which addresses this issue?
---
[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r160591890
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
---
@@ -150,12 +151,36 @@ def zeppelin_notebook_import(self, env):
env.set_params(params)
Logger.info(ambari_format('Searching for Zeppelin Notebooks in
{metron_config_zeppelin_path}'))
+
+# With Ambari 2.5+, Zeppelin server is enabled to work with Shiro
authentication, which requires user/password
+# for authentication (see
https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html for
details).
+ses = requests.session()
+
+# Check if authentication is enabled on the Zeppelin server
+try:
+conn =
ses.get(ambari_format('http://{zeppelin_server_url}/api/login'))
+
+# Establish connection if authentication is enabled
+try:
+# The following credentials are created at install time by
Ambari at /etc/zeppelin/conf/shiro.ini
+# when Shiro auth is enabled on the Zeppelin server
+zeppelin_payload = {'userName': 'admin', 'password' :
'admin'}
--- End diff --
Hi @cestella , thanks for taking a look. The Zeppelin auth credentials are
not related to ambari credentials. They are as defined in
`/etc/zeppelin/conf/shiro.ini` on the Zeppelin host, which is set up by default
during stack deploy time.
Here's a snippet:
```
[root@hcpca-1 ~]# cat /etc/zeppelin/conf/shiro.ini
[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc
at http://shiro.apache.org/configuration.html#Configuration-INISections
admin = admin, admin
```
When I created the pull request, I considered the option of reading
`shiro.ini` file and parsing for the credentials. I did not pursue that further
since I wanted to address the immediate need of ability to import notebook on
an out-of-the-box Metron cluster. I understand that a thorough way would be to
handle the [different INI
settings](http://shiro.apache.org/configuration.html#Configuration-INISections)
provided by shiro, and could potentially be the scope of another pull request.
Please let me know if it makes sense.
---
[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/907 Hi @cestella , with the latest commit having the normalized producer, I was able to spin up a 12-node Centos 7 cluster, install with HDP 2.6 base and kerberize as well without issues. Ran the standard squid sensor test and was able to see events in the Alerts UI. +1 from my end. ---
[GitHub] metron pull request #935: METRON-1386: Fix Metron Website Required Links
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/935 METRON-1386: Fix Metron Website Required Links ## Contributor Comments Made the following changes to the Metron ASF site: * Fixed the missing entities and made corrections per the Apache TLP Website checks for [Metron](https://whimsy.apache.org/site/project/metron) * Added Raghu and Anand as committers in the Community page Please Note: This PR does not address the [graphics changes](https://issues.apache.org/jira/browse/METRON-1386?focusedCommentId=16309716=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16309716) required to conform to Apache standards, which is tracked by METRON-1456. **Testing Steps** Follow the below steps to test the changes 1. From `metron/site` execute ``` bundle install bundle exec jekyll serve ``` 2. Navigate to http://localhost:4000 to review the site Note that these steps are an excerpt from https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1386 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/935.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #935 commit 5622ac07b48b9f9ea862bb45ae396964343ccc59 Author: Anand Subramanian <asubramanian@...> Date: 2018-02-12T09:25:47Z First pass at fixing Apache TLP website checks ---
[GitHub] metron pull request #932: METRON-1451: On Centos full dev, Metron Indexing s...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/932 METRON-1451: On Centos full dev, Metron Indexing shows up as stopped ## Contributor Comments **Root Cause** The python-requests module introduced a VendorAlias import machinery in v2.5.2 and caused the sys.modules to break. This issue was later fixed in v2.6.1 of python-requests (see [here](https://pypi.python.org/pypi/requests)). The solution is to use this version of python-requests, which fixed the issue. **Testing Done** On CentOS 6 full dev - Validated python-requests v2.6.1 is installed - Checked for the Metron Indexing service state to be up. - Checked ambari-agent.log and the error was not seen. On Ubuntu 14 full dev - Repeated the same steps as on CentOS 6 **Verification Steps** - Spin up full dev. - Verify Metron Indexing service is shown as up and running. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1451 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #932 commit f3821ec78708c0122ba7d3ed8057907c8e7a224e Author: Anand Subramanian <asubramanian@...> Date: 2018-02-08T11:12:50Z Install the right version of python-requests ---
[GitHub] metron pull request #932: METRON-1451: On Centos full dev, Metron Indexing s...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/932 METRON-1451: On Centos full dev, Metron Indexing shows up as stopped ## Contributor Comments **Root Cause** The python-requests module introduced a VendorAlias import machinery in v2.5.2 and caused the sys.modules to break. This issue was later fixed in v2.6.1 of python-requests (see [here](https://pypi.python.org/pypi/requests)). The solution is to use this version of python-requests, which fixed the issue. **Testing Done** On CentOS 6 full dev - Validated python-requests v2.6.1 is installed - Checked for the Metron Indexing service state to be up. - Checked ambari-agent.log and the error was not seen. On Ubuntu 14 full dev - Repeated the same steps as on CentOS 6 **Verification Steps** - Spin up full dev. - Verify Metron Indexing service is shown as up and running. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1451 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #932 commit f3821ec78708c0122ba7d3ed8057907c8e7a224e Author: Anand Subramanian <asubramanian@...> Date: 2018-02-08T11:12:50Z Install the right version of python-requests ---
[GitHub] metron pull request #932: METRON-1451: On Centos full dev, Metron Indexing s...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/932 ---
[GitHub] metron pull request #938: METRON-1457: Move ASF links to main page in the Me...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/938 METRON-1457: Move ASF links to main page in the Metron website ## Contributor Comments Apache Whimsy checks for the site requisites in the main index.html and not inside sub-levels. This PR moves the content of the ASF and links to the main page. Here is a screenshot of how the new metron page will look like: https://imgur.com/3Y8ZLWL **Testing Steps** Follow the below steps to test the changes * From `metron/site` execute ``` bundle install bundle exec jekyll serve ``` * Navigate to http://localhost:4000 to review the site Note that these steps are an excerpt from https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge. Please follow this page to have the pre-requisites set up. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1457 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/938.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #938 commit dda05791a3667859be6159abc814a59af506ed09 Author: Anand Subramanian <asubramanian@...> Date: 2018-02-16T14:27:55Z Move ASF links to the main page ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159361598 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services ar
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159280034 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,132 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure you
[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/864
Hi @nickwallen , I tried this on a 12-node cluster. I validated that
`clusterHostInfo` is populated properly for the alerts_ui, management_ui and
rest_ui hosts.
However, in my case it failed on the parser service check since the 'Metron
Check' step landed on a host without Kafka broker installed.
Here's the error excerpt:
```
2017-12-20 18:42:54,285 - Performing Parser service check
2017-12-20 18:42:54,285 - Checking for grok patterns in HDFS for Parsers
2017-12-20 18:42:54,285 - Checking HDFS; directory=/apps/metron/patterns
user=metron
2017-12-20 18:42:54,285 - Execute['/usr/hdp/2.5.3.0-37/hadoop/bin/hdfs dfs
-test -d /apps/metron/patterns'] {'logoutput': True, 'path':
['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user': 'metron',
'try_sleep': 5}
2017-12-20 18:42:56,822 - Checking Kafka topics for Parsers
2017-12-20 18:42:56,822 - Checking existence of Kafka topic 'bro'
2017-12-20 18:42:56,823 -
Execute['/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
--list | awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit
0} else {exit 1}}''] {'logoutput': True, 'path':
['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user': 'kafka',
'try_sleep': 5}
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or
directory
2017-12-20 18:42:56,900 - Retrying after 5 seconds. Reason: Execution of
'/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
--list | awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit
0} else {exit 1}}'' returned 1. -bash:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or directory
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or
directory
2017-12-20 18:43:01,987 - Retrying after 5 seconds. Reason: Execution of
'/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
--list | awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit
0} else {exit 1}}'' returned 1. -bash:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or directory
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or
directory
Command failed after 1 tries
```
I noticed that the `clusterHostInfo` indeed has a list of the
`kafka_broker_hosts` (see attached
[clusterHostInfo-12-node.txt](https://github.com/apache/metron/files/1576752/clusterHostInfo-12-node.txt).
Would it be possible to either a) force Ambari to run metron service check on
one of the kafka broker hosts; or b) run
[check_kafka_topics](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py#L259)
on a `kafka_broker_host`
I am perfectly fine if you think the kafka_broker fix should be a different
PR than this.
---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/938 Thanks @justinleet for the pointer. I have made a change to use the Powered By logo instead of the foundation logo with feather, since I thought this looked better. I also added a direct link to the ASF press kit page, instead of storing the image on our local site. Let me know your thoughts. https://user-images.githubusercontent.com/20395490/36713710-ee7afc76-1bb4-11e8-9c06-a19bd20c17fd.png;> ---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/938 Thanks all. ---
[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/943 +1 Works as advertised. I used both mpacks to deploy a 12-node CentOS 7 cluster (Ambari 2.6.0.0 and HDP 2.6.3.0). Was able to kerberize the cluster and get bro indices into ES. ---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/938 Thank you @mmiklavc and @nickwallen for your reply on the dev list. Any feedback on this PR? ---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/938 Roger that. Here's the new look. Let me know if this looks ok @justinleet and @cestella .  ---
