[jira] [Commented] (METRON-1292) Metron should support Apache Nifi Storm Spout
[ https://issues.apache.org/jira/browse/METRON-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238713#comment-16238713 ] Josh Meyer commented on METRON-1292: Personally, I would not use the storm feature of NiFi (https://github.com/apache/nifi/tree/master/nifi-external). There seems to be little activity in this area where Kafka is used widely used. Also, are using Kafka quite a bit. To me the simplest and lowest risk would be NiFi <-> Kafka <-> Storm. > Metron should support Apache Nifi Storm Spout > - > > Key: METRON-1292 > URL: https://issues.apache.org/jira/browse/METRON-1292 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > https://issues.apache.org/jira/browse/NIFI-606 > Nifi has a storm spout for communication with nifi. > Metron should explore if there is any value to supporting a nifi storm spout > as opposed > to sending to kafka. > Sample : https://github.com/bbende/nifi-storm-topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates
[ https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238502#comment-16238502 ] ASF GitHub Bot commented on METRON-1296: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/829 Worked for me when I ran it up in Full Dev. I am able to see the templates and use the Alerts UI. ``` 2017-11-03 22:22:41,294 - File['/var/lib/ambari-agent/tmp/bro_index.template'] {'content': StaticFile('bro_index.template'), 'mode': 0755} 2017-11-03 22:22:41,298 - Writing File['/var/lib/ambari-agent/tmp/bro_index.template'] because it doesn't exist 2017-11-03 22:22:41,299 - Changing permission for /var/lib/ambari-agent/tmp/bro_index.template from 644 to 755 2017-11-03 22:22:41,299 - File['/var/lib/ambari-agent/tmp/snort_index.template'] {'content': StaticFile('snort_index.template'), 'mode': 0755} 2017-11-03 22:22:41,300 - Writing File['/var/lib/ambari-agent/tmp/snort_index.template'] because it doesn't exist 2017-11-03 22:22:41,301 - Changing permission for /var/lib/ambari-agent/tmp/snort_index.template from 644 to 755 2017-11-03 22:22:41,301 - File['/var/lib/ambari-agent/tmp/yaf_index.template'] {'content': StaticFile('yaf_index.template'), 'mode': 0755} 2017-11-03 22:22:41,302 - Writing File['/var/lib/ambari-agent/tmp/yaf_index.template'] because it doesn't exist 2017-11-03 22:22:41,302 - Changing permission for /var/lib/ambari-agent/tmp/yaf_index.template from 644 to 755 2017-11-03 22:22:41,302 - File['/var/lib/ambari-agent/tmp/error_index.template'] {'content': StaticFile('error_index.template'), 'mode': 0755} 2017-11-03 22:22:41,303 - Writing File['/var/lib/ambari-agent/tmp/error_index.template'] because it doesn't exist 2017-11-03 22:22:41,303 - Changing permission for /var/lib/ambari-agent/tmp/error_index.template from 644 to 755 2017-11-03 22:22:41,303 - File['/var/lib/ambari-agent/tmp/meta_index.mapping'] {'content': StaticFile('meta_index.mapping'), 'mode': 0755} 2017-11-03 22:22:41,304 - Writing File['/var/lib/ambari-agent/tmp/meta_index.mapping'] because it doesn't exist 2017-11-03 22:22:41,305 - Changing permission for /var/lib/ambari-agent/tmp/meta_index.mapping from 644 to 755 2017-11-03 22:22:41,305 - Execute['curl -s -XPOST http://node1:9200/_template/bro_index -d @/var/lib/ambari-agent/tmp/bro_index.template'] {'logoutput': True} {"acknowledged":true}2017-11-03 22:22:41,428 - Execute['curl -s -XPOST http://node1:9200/_template/snort_index -d @/var/lib/ambari-agent/tmp/snort_index.template'] {'logoutput': True} {"acknowledged":true}2017-11-03 22:22:41,448 - Execute['curl -s -XPOST http://node1:9200/_template/yaf_index -d @/var/lib/ambari-agent/tmp/yaf_index.template'] {'logoutput': True} {"acknowledged":true}2017-11-03 22:22:41,473 - Execute['curl -s -XPOST http://node1:9200/_template/error_index -d @/var/lib/ambari-agent/tmp/error_index.template'] {'logoutput': True} {"acknowledged":true}2017-11-03 22:22:41,497 - Execute['curl -s -XPOST http://node1:9200/metaalert_index -d @/var/lib/ambari-agent/tmp/meta_index.mapping'] {'logoutput': True} {"acknowledged":true}2017-11-03 22:22:41,832 - Setting Elasticsearch template installed to True 2017-11-03 22:22:41,832 - File['/usr/metron/0.4.2/config/zookeeper/../metron_elasticsearch_template_installed_flag_file'] {'owner': 'metron', 'content': 'This file created on: 2017-11-03 22:22:41', 'mode': 0755} 2017-11-03 22:22:41,833 - Writing File['/usr/metron/0.4.2/config/zookeeper/../metron_elasticsearch_template_installed_flag_file'] because it doesn't exist 2017-11-03 22:22:41,833 - Changing owner for /usr/metron/0.4.2/config/zookeeper/../metron_elasticsearch_template_installed_flag_file from 0 to metron 2017-11-03 22:22:41,833 - Changing permission for /usr/metron/0.4.2/config/zookeeper/../metron_elasticsearch_template_installed_flag_file from 644 to 755 2017-11-03 22:22:41,833 - Starting indexing ``` > Full Dev Fails to Deploy Index Templates > > > Key: METRON-1296 > URL: https://issues.apache.org/jira/browse/METRON-1296 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Normal > Fix For: 0.4.2 > > > There are cases where Full Dev is deployed, but the index templates are not > deployed. Without the index templates, the Alerts UI and Kibana dashboard do > not work as expected. This seems to occur sometimes and not others. > The index template install happens during startup of the Index topology. > Elasticsearch master start occurred well before the attempt to install the > templates. The following message is shown in the logs. > {code} > WARNING: Elasticsearch templates
[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates
[ https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238477#comment-16238477 ] ASF GitHub Bot commented on METRON-1296: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/829 METRON-1296 Full Dev Fails to Deploy Index Templates Fixed a bug that prevents the index templates from being installed. This also ensures that the cause of an exception is logged. Oops. This should not have passed testing. I reviewed the original PR and should have caught this. I think a late change in the review process mucked things up. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1296 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/829.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #829 commit d6a2ee52303e19a55194e0a0d1530b5ee9c27068 Author: Nick AllenDate: 2017-11-03T21:58:15Z METRON-1296 Full Dev Fails to Deploy Index Templates > Full Dev Fails to Deploy Index Templates > > > Key: METRON-1296 > URL: https://issues.apache.org/jira/browse/METRON-1296 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Normal > Fix For: 0.4.2 > > > There are cases where Full Dev is deployed, but the index templates are not > deployed. Without the index templates, the Alerts UI and Kibana dashboard do > not work as expected. This seems to occur sometimes and not others. > The index template install happens during startup of the Index topology. > Elasticsearch master start occurred well before the attempt to install the > templates. The following message is shown in the logs. > {code} > WARNING: Elasticsearch templates could not be installed. The Elasticsearch > service is probably down. > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (METRON-1296) Full Dev Fails to Deploy Index Templates
[ https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1296: -- Assignee: Nick Allen > Full Dev Fails to Deploy Index Templates > > > Key: METRON-1296 > URL: https://issues.apache.org/jira/browse/METRON-1296 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Normal > Fix For: 0.4.2 > > > There are cases where Full Dev is deployed, but the index templates are not > deployed. Without the index templates, the Alerts UI and Kibana dashboard do > not work as expected. This seems to occur sometimes and not others. > The index template install happens during startup of the Index topology. > Elasticsearch master start occurred well before the attempt to install the > templates. The following message is shown in the logs. > {code} > WARNING: Elasticsearch templates could not be installed. The Elasticsearch > service is probably down. > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates
[ https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238419#comment-16238419 ] Nick Allen commented on METRON-1296: Manually installing the templates using the Service Action; "Install Elasticsearch Templates" does successfully install the templates. > Full Dev Fails to Deploy Index Templates > > > Key: METRON-1296 > URL: https://issues.apache.org/jira/browse/METRON-1296 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Normal > Fix For: 0.4.2 > > > There are cases where Full Dev is deployed, but the index templates are not > deployed. Without the index templates, the Alerts UI and Kibana dashboard do > not work as expected. This seems to occur sometimes and not others. > The index template install happens during startup of the Index topology. > Elasticsearch master start occurred well before the attempt to install the > templates. The following message is shown in the logs. > {code} > WARNING: Elasticsearch templates could not be installed. The Elasticsearch > service is probably down. > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates
[ https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238414#comment-16238414 ] Nick Allen commented on METRON-1296: There was a recent PR that moved the logic that automates the template install from Ansible to the MPack. The goal was so that users never have to manually install the index templates. Prior to this it was only automated for Full Dev. https://github.com/apache/metron/pull/817 > Full Dev Fails to Deploy Index Templates > > > Key: METRON-1296 > URL: https://issues.apache.org/jira/browse/METRON-1296 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Normal > Fix For: 0.4.2 > > > There are cases where Full Dev is deployed, but the index templates are not > deployed. Without the index templates, the Alerts UI and Kibana dashboard do > not work as expected. This seems to occur sometimes and not others. > The index template install happens during startup of the Index topology. > Elasticsearch master start occurred well before the attempt to install the > templates. The following message is shown in the logs. > {code} > WARNING: Elasticsearch templates could not be installed. The Elasticsearch > service is probably down. > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1295) Unable to Configure Logging for REST API
[ https://issues.apache.org/jira/browse/METRON-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238294#comment-16238294 ] ASF GitHub Bot commented on METRON-1295: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/828 METRON-1295 Unable to Configure Logging for REST API When configuring the logging for the REST API, the following exception occurs. ``` log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable to a "org.apache.hadoop.hbase.shaded.org.apache.log4j.Appender" variable. ``` This change prevents the shaded log4j dependencies of metron-hbase from bleeding into metron-rest. ## Testing 1. Spin-up Full Dev 2. Create a log4j configuration file. In this example `/usr/metron/0.4.2/config/log4j.properties` 3. Add the following to "Metron JVM Flags" in Ambari > Metron > Config. ``` -Dlog4j.debug -Dlog4j.configuration=file:/usr/metron/0.4.2/config/log4j.properties ``` 4. Restart Metron REST and watch the logging output. Ensure that you can configure logging as you would expect. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1295 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/828.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #828 commit a7a7174d28f08cf95093a95bee257185c161 Author: Nick AllenDate: 2017-11-02T14:04:04Z METRON-1295 Unable to Configure Logging for REST API > Unable to Configure Logging for REST API > > > Key: METRON-1295 > URL: https://issues.apache.org/jira/browse/METRON-1295 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Normal > > I have not been able to configure logging for the REST API. To replicate, > create a log4j configuration file, then add the following to "Metron JVM > Flags" in Ambari > Metron > Config. > {code} > -Dlog4j.debug > -Dlog4j.configuration=file:/usr/metron/0.4.2/config/log4j.properties > {code} > This will result in the following exception when Log4j initializes. > {code} > log4j: Using URL [file:/usr/metron/0.4.2/config/log4j.properties] for > automatic log4j configuration. > log4j: Reading configuration from URL > file:/usr/metron/0.4.2/config/log4j.properties > log4j: Parsing for [root] with value=[INFO, file]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "file". > log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable > to a "org.apache.hadoop.hbase.shaded.org.apache.log4j.Appender" variable. > log4j:ERROR The class > "org.apache.hadoop.hbase.shaded.org.apache.log4j.Appender" was loaded by > log4j:ERROR [sun.misc.Launcher$AppClassLoader@5c647e05] whereas object of type > log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by > [sun.misc.Launcher$AppClassLoader@5c647e05]. > log4j:ERROR Could not instantiate appender named "file". > log4j: Finished configuring. > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1294) IP addresses are not formatted correctly in facet and group results
[ https://issues.apache.org/jira/browse/METRON-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238178#comment-16238178 ] ASF GitHub Bot commented on METRON-1294: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/827 METRON-1294: IP addresses are not formatted correctly in facet and group results ## Contributor Comments By default, values in facet and group by results are not formatted correctly in some cases (IP addresses being one) so the ElasticsearchDao must correctly format them instead. To do this it must know the type of each facet/group column to know how to correctly format them. The root cause of this issue is that the ElasticsearchDao was retrieving "common" column metadata, or columns that exist in all indices. The addition of the "metaalert" index broke this because it is missing most of the columns that are declared in alert indices. This PR corrects this by retrieving all column metadata across indices so that a missing column in one index does not affect the results. The http://localhost:8080/swagger-ui.html#!/search-controller/getColumnMetadataUsingPOST changed slightly in that it now just returns a single map with all columns/types included and no longer segments columns/types by index. This interface more closely matches how this function is used in the alerts UI and eliminates the need for the client side code to flatten the result. Column metadata can still be retrieved for an index by just passing in a single index to that endpoint. The case of columns existing in multiple indices with different types is solved by just excluding them from the results. This can be tested in full dev by executing a search (either with the Alerts UI or Swagger) and including an ip address field in the facet list. The results should now include correctly formatted IP addresses in the facetCounts section of the results. A group query can be verified by including an IP address in the list of group fields. I adjusted the SearchControllerIntegrationTest to account for the new interface and modified cases in the SearchIntegrationTest to catch these issues in the future. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built
[jira] [Created] (METRON-1294) IP addresses are not formatted correctly in facet and group results
Ryan Merriman created METRON-1294: - Summary: IP addresses are not formatted correctly in facet and group results Key: METRON-1294 URL: https://issues.apache.org/jira/browse/METRON-1294 Project: Metron Issue Type: Bug Reporter: Ryan Merriman Assignee: Ryan Merriman Priority: Major Fields that are of type IP address are being returned in numerical format whereas they should be returned in string format. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created
[ https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237667#comment-16237667 ] ASF GitHub Bot commented on METRON-1289: Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/824 I have tested the rest-api I am able to see the alert details for all alerts under meta alerts I see a couple of new issues: - When a meta alert is created the meta alert is returned as part of the group api which was not the case before. - When a meta alert is deleted (status set to inactive) the meta alert gets deleted but the details are missing for all alerts under meta alert. I feel this bug might be in there all along but never surfaced since the details were missing when the meta alert is created. It would be nice if you can handle this in this PR. > Alert fields are lost when a MetaAlert is created > - > > Key: METRON-1289 > URL: https://issues.apache.org/jira/browse/METRON-1289 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > When a MetaAlert is created, the included results are being updated > incorrectly with only the "metaalert" field. This causes subsequent findOne > operations to only return the "metaalert field for that alert. All fields > should continue to be present. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1292) Metron should support Apache Nifi Storm Spout
Otto Fowler created METRON-1292: --- Summary: Metron should support Apache Nifi Storm Spout Key: METRON-1292 URL: https://issues.apache.org/jira/browse/METRON-1292 Project: Metron Issue Type: New Feature Reporter: Otto Fowler Assignee: Otto Fowler Priority: Major https://issues.apache.org/jira/browse/NIFI-606 Nifi has a storm spout for communication with nifi. Metron should explore if there is any value to supporting a nifi storm spout as opposed to sending to kafka. Sample : https://github.com/bbende/nifi-storm-topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)