[jira] [Commented] (METRON-1764) Update version to 0.6.0
[ https://issues.apache.org/jira/browse/METRON-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603738#comment-16603738 ] ASF GitHub Bot commented on METRON-1764: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1183 > Update version to 0.6.0 > --- > > Key: METRON-1764 > URL: https://issues.apache.org/jira/browse/METRON-1764 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > In anticipation of release, we need to update the version since we agreed on > 0.6.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1183: METRON-1764: Update version to 0.6.0
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1183 ---
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603737#comment-16603737 ] ASF GitHub Bot commented on METRON-1476: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1096 I'd prefer this to not be included in the next release; 0.6.0. This is a rather large change and I'd like it to get more cycles in master before we include it in a release. Let's wait for @justinleet to kick-out at least the first release candidate of 0.6.0 before we merge this. Does that work for you @justinleet ? > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1096: METRON-1476: Update to Angular 6.1.3
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1096 I'd prefer this to not be included in the next release; 0.6.0. This is a rather large change and I'd like it to get more cycles in master before we include it in a release. Let's wait for @justinleet to kick-out at least the first release candidate of 0.6.0 before we merge this. Does that work for you @justinleet ? ---
[jira] [Commented] (METRON-1764) Update version to 0.6.0
[ https://issues.apache.org/jira/browse/METRON-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603729#comment-16603729 ] ASF GitHub Bot commented on METRON-1764: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1183 +1 Looks good. Thanks! > Update version to 0.6.0 > --- > > Key: METRON-1764 > URL: https://issues.apache.org/jira/browse/METRON-1764 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > In anticipation of release, we need to update the version since we agreed on > 0.6.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1183: METRON-1764: Update version to 0.6.0
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1183 +1 Looks good. Thanks! ---
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603673#comment-16603673 ] Jungtaek Lim commented on METRON-1588: -- [~nickwallen] Yeah thanks for explaining. I guess I haven't requested contributor role and that's why assigning to me doesn't work. Is it encouraged to request it from dev. mailing list, or someone can take forward in here? > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1588: -- Assignee: (was: Nick Allen) > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603668#comment-16603668 ] Nick Allen commented on METRON-1588: I was just going through old JIRAs and getting them ready for the next release. I tried to assign this PR to you and you do not have the proper permissions for it to be assigned to you. Feel free to correct that. > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Assignee: Nick Allen >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603667#comment-16603667 ] Jungtaek Lim commented on METRON-1588: -- [~nickwallen] I'm sorry but could I see the reason why you've just assigned this to you (and the issue is not assigned to me)? I've been contributing various Apache TLP and haven't seen any of project which issue is assigned to neither PR author nor PR committer. A bit strange. > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Assignee: Nick Allen >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1717) Relocate Storm Profiler Code
[ https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1717: --- Description: The Storm port of the Profiler currently lives in metron-analytics/metron-profiler. This should be moved to metron-analytics/metron-profiler-storm. This would mirror the project names for the Spark port (metron-profiler-spark) and the REPL port (metron-profiler-repl). The package name for the Storm port of the Profiler should be changed to org.apache.metron.profiler.storm. This would mimic the package name used for Spark; org.apache.metron.profiler.spark. was: The Storm port of the Profiler currently lives in metron-analytics/metron-profiler. This should be moved to metron-analytics/metron-profiler-storm. This would mirror the project name for the Spark port; matron-profiler-spark. The package name for the Storm port of the Profiler should be changed to org.apache.metron.profiler.storm. This would mimic the package name used for Spark; org.apache.metron.profiler.spark. > Relocate Storm Profiler Code > > > Key: METRON-1717 > URL: https://issues.apache.org/jira/browse/METRON-1717 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Priority: Major > > The Storm port of the Profiler currently lives in > metron-analytics/metron-profiler. This should be moved to > metron-analytics/metron-profiler-storm. This would mirror the project names > for the Spark port (metron-profiler-spark) and the REPL port > (metron-profiler-repl). > The package name for the Storm port of the Profiler should be changed to > org.apache.metron.profiler.storm. This would mimic the package name used for > Spark; org.apache.metron.profiler.spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1716) MPack Support for the Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1716: -- Assignee: Nick Allen > MPack Support for the Batch Profiler > > > Key: METRON-1716 > URL: https://issues.apache.org/jira/browse/METRON-1716 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1765) Merge Batch Profiler Feature Branch into Master
Nick Allen created METRON-1765: -- Summary: Merge Batch Profiler Feature Branch into Master Key: METRON-1765 URL: https://issues.apache.org/jira/browse/METRON-1765 Project: Metron Issue Type: Sub-task Reporter: Nick Allen -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1756) REST tests should use Embedded LDAP in metron-security
[ https://issues.apache.org/jira/browse/METRON-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603620#comment-16603620 ] ASF GitHub Bot commented on METRON-1756: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1186 METRON-1756: REST tests should use Embedded LDAP in metron-security ## Contributor Comments This PR switches authentication from JDBC-based to LDAP in metron-rest. ### Changes Included - Added test dependencies for metron-ui-security and Apache Directory to metron-rest - Removed JDBC authentication (the TestSecurityConfig class) - Updated the metron-rest integration test to set the `EMBEDDED_LDAP_PROFILE` - Added additional users to the LDAP schema as required by metron-rest tests - Suppressed Apache directory log warnings when running metron-rest tests ### Testing All metron-rest unit and integration tests should continue to pass. ### Other Considerations After switching the metron-rest tests to use LDAP through metron-ui-security, I noticed a significant number of LDAP-related log warnings when running the integration tests. As far as I can tell they are caused by missing LDAP attributes and similar issues that we probably don't care about. I can also see the argument for wanting these logs enabled to make it easier for future contributors to find and diagnose problems when working on these tests (although it's easy enough to update the log4j.properties file if needed). Thoughts? ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1756 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1186 commit 4bcfd594636bf8f8ec924b0b62dd0a08b3c315fb Author: merrimanr Date: 2018-08-29T17:05:38Z initial commit > REST tests should use Embedded LDAP in metron-security >
[GitHub] metron pull request #1186: METRON-1756: REST tests should use Embedded LDAP ...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1186 METRON-1756: REST tests should use Embedded LDAP in metron-security ## Contributor Comments This PR switches authentication from JDBC-based to LDAP in metron-rest. ### Changes Included - Added test dependencies for metron-ui-security and Apache Directory to metron-rest - Removed JDBC authentication (the TestSecurityConfig class) - Updated the metron-rest integration test to set the `EMBEDDED_LDAP_PROFILE` - Added additional users to the LDAP schema as required by metron-rest tests - Suppressed Apache directory log warnings when running metron-rest tests ### Testing All metron-rest unit and integration tests should continue to pass. ### Other Considerations After switching the metron-rest tests to use LDAP through metron-ui-security, I noticed a significant number of LDAP-related log warnings when running the integration tests. As far as I can tell they are caused by missing LDAP attributes and similar issues that we probably don't care about. I can also see the argument for wanting these logs enabled to make it easier for future contributors to find and diagnose problems when working on these tests (although it's easy enough to update the log4j.properties file if needed). Thoughts? ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1756 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1186 commit 4bcfd594636bf8f8ec924b0b62dd0a08b3c315fb Author: merrimanr Date: 2018-08-29T17:05:38Z initial commit ---
[GitHub] metron pull request #1185: METRON-1508 In Ubuntu14 Dev Indexing Fails to Wri...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1185
METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
When spinning up the Ubuntu 14 development environment, the indexing
topology fails to write to Elasticsearch. The indexing topology reports the
following error.
```
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3
3] [ERROR]
org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes are available:
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
~[stormjar.jar:?]
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
~[stormjar.jar:?]
at
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
~[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
```
In Ubuntu, Elasticsearch is not setup to bind to `node1:9300`. When the
index topology spins-up and attempts to connect, it fails with this exception.
Simply binding to `localhost:9300` fixes this problem in the Ubuntu development
environment.
I extracted the `es_hosts` variable from `single_node_vm.yml` in a way that
allows each development environment (Ubuntu and CentOS) to define this value
independently. This allows CentOS to continue to use `node1:9300` as it always
has and Ubuntu can use `localhost:9300` as it needs to.
This change only impacts the way the development environments are deployed.
This does not impact any of the production deployment tooling.
## Testing
1. Spin-up the CentOS development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/centos6
vagrant up
```
1. Spin-up the Ubuntu development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/ubuntu14
vagrant up
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603618#comment-16603618
]
ASF GitHub Bot commented on METRON-1508:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1185
METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
When spinning up the Ubuntu 14 development environment, the indexing
topology fails to write to Elasticsearch. The indexing topology reports the
following error.
```
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3
3] [ERROR]
org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes are available:
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
~[stormjar.jar:?]
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
~[stormjar.jar:?]
at
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
~[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
```
In Ubuntu, Elasticsearch is not setup to bind to `node1:9300`. When the
index topology spins-up and attempts to connect, it fails with this exception.
Simply binding to `localhost:9300` fixes this problem in the Ubuntu development
environment.
I extracted the `es_hosts` variable from `single_node_vm.yml` in a way that
allows each development environment (Ubuntu and CentOS) to define this value
independently. This allows CentOS to continue to use `node1:9300` as it always
has and Ubuntu can use `localhost:9300` as it needs to.
This change only impacts the way the development environments are deployed.
This does not impact any of the production deployment tooling.
## Testing
1. Spin-up the CentOS development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/centos6
vagrant up
```
1. Spin-up the Ubuntu development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/ubuntu14
vagrant up
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
[jira] [Commented] (METRON-1741) Move REPL Port of Profiler to Separate Project
[ https://issues.apache.org/jira/browse/METRON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603610#comment-16603610 ] ASF GitHub Bot commented on METRON-1741: Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1170 > Move REPL Port of Profiler to Separate Project > -- > > Key: METRON-1741 > URL: https://issues.apache.org/jira/browse/METRON-1741 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1741) Move REPL Port of Profiler to Separate Project
[ https://issues.apache.org/jira/browse/METRON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603609#comment-16603609 ] ASF GitHub Bot commented on METRON-1741: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1170 Thanks for the review! This has been merged into the feature branch. > Move REPL Port of Profiler to Separate Project > -- > > Key: METRON-1741 > URL: https://issues.apache.org/jira/browse/METRON-1741 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1170: METRON-1741 Move REPL Port of Profiler to Separate Proje...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1170 Thanks for the review! This has been merged into the feature branch. ---
[GitHub] metron pull request #1170: METRON-1741 Move REPL Port of Profiler to Separat...
Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1170 ---
[jira] [Assigned] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-1508:
--
Assignee: Nick Allen
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1741) Move REPL Port of Profiler to Separate Project
[ https://issues.apache.org/jira/browse/METRON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603499#comment-16603499 ] ASF GitHub Bot commented on METRON-1741: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1170 I ran this up in full dev and went through the testing instructions. Everything worked great. +1 > Move REPL Port of Profiler to Separate Project > -- > > Key: METRON-1741 > URL: https://issues.apache.org/jira/browse/METRON-1741 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1170: METRON-1741 Move REPL Port of Profiler to Separate Proje...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1170 I ran this up in full dev and went through the testing instructions. Everything worked great. +1 ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603411#comment-16603411
]
ASF GitHub Bot commented on METRON-1761:
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/1184
METRON-1761, allow application of grok statement multiple times
This PR adds support for incoming messages to grok parsers that have
multiple log lines.
Instead of having to split the logs before sending to metron/kafka, you
could just send the logs in batches.
# todo testing
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [-] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron grok-split
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1184.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1184
commit 5cb7cb2b869694ceff37c7e07e16358e4b918b2c
Author: Otto Fowler
Date: 2018-09-04T10:50:38Z
METRON-1761, allow application of grok statement multiple times
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1184 METRON-1761, allow application of grok statement multiple times This PR adds support for incoming messages to grok parsers that have multiple log lines. Instead of having to split the logs before sending to metron/kafka, you could just send the logs in batches. # todo testing ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron grok-split Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 5cb7cb2b869694ceff37c7e07e16358e4b918b2c Author: Otto Fowler Date: 2018-09-04T10:50:38Z METRON-1761, allow application of grok statement multiple times ---
[jira] [Updated] (METRON-1670) Stellar WEEK_OF_YEAR test is locale sensitive
[ https://issues.apache.org/jira/browse/METRON-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1670: --- Fix Version/s: 0.6.0 > Stellar WEEK_OF_YEAR test is locale sensitive > - > > Key: METRON-1670 > URL: https://issues.apache.org/jira/browse/METRON-1670 > Project: Metron > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Elliston Ball >Assignee: Simon Elliston Ball >Priority: Trivial > Fix For: 0.6.0 > > > The Stellar WEEK_OF_YEAR(epoch) function is sensitive to the locale of the > machine it is running on. The tests in DateFunctionsTest are not, this leads > to test failures on machine locales that differ in their first day of week > definition or days in first week definition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1670) Stellar WEEK_OF_YEAR test is locale sensitive
[ https://issues.apache.org/jira/browse/METRON-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1670: -- Assignee: Simon Elliston Ball > Stellar WEEK_OF_YEAR test is locale sensitive > - > > Key: METRON-1670 > URL: https://issues.apache.org/jira/browse/METRON-1670 > Project: Metron > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Simon Elliston Ball >Assignee: Simon Elliston Ball >Priority: Trivial > Fix For: 0.6.0 > > > The Stellar WEEK_OF_YEAR(epoch) function is sensitive to the locale of the > machine it is running on. The tests in DateFunctionsTest are not, this leads > to test failures on machine locales that differ in their first day of week > definition or days in first week definition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1621) Add sorting by triage score to the end to end tests for the UI
[ https://issues.apache.org/jira/browse/METRON-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603358#comment-16603358 ] Nick Allen commented on METRON-1621: https://github.com/apache/metron/pull/1088 > Add sorting by triage score to the end to end tests for the UI > -- > > Key: METRON-1621 > URL: https://issues.apache.org/jira/browse/METRON-1621 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1621) Add sorting by triage score to the end to end tests for the UI
[ https://issues.apache.org/jira/browse/METRON-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1621: -- Assignee: Tibor Meller > Add sorting by triage score to the end to end tests for the UI > -- > > Key: METRON-1621 > URL: https://issues.apache.org/jira/browse/METRON-1621 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1621) Add sorting by triage score to the end to end tests for the UI
[ https://issues.apache.org/jira/browse/METRON-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1621: --- Fix Version/s: 0.6.0 > Add sorting by triage score to the end to end tests for the UI > -- > > Key: METRON-1621 > URL: https://issues.apache.org/jira/browse/METRON-1621 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1236) Add mpack support for ambari-agent running as non-root user
[ https://issues.apache.org/jira/browse/METRON-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1236: --- Fix Version/s: 0.6.0 > Add mpack support for ambari-agent running as non-root user > --- > > Key: METRON-1236 > URL: https://issues.apache.org/jira/browse/METRON-1236 > Project: Metron > Issue Type: Improvement >Reporter: Kyle Richardson >Priority: Minor > Labels: mpack > Fix For: 0.6.0 > > > The current service start/stop/status/restart commands do not utilize `sudo` > and therefore the ambari-agent must be running as root on each node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1613) Metaalerts status update broken in Alerts UI
[
https://issues.apache.org/jira/browse/METRON-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-1613:
--
Assignee: Ryan Merriman
> Metaalerts status update broken in Alerts UI
>
>
> Key: METRON-1613
> URL: https://issues.apache.org/jira/browse/METRON-1613
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> Steps to reproduce:
> 1. Launch Alerts UI
> 2. Create a metaalert (E.g. based on source type)
> 3. Click on the Metaalert and attempt to change the status using the right
> side pane.
> 4. The Metaalert status is not changed.
> Following error is seen in the metron-rest.log
> {code:java}
> 18/05/23 16:58:32 ERROR controller.RestExceptionHandler: Encountered error:
> Collection not found: metaalert_index org.apache.metron.rest.RestException:
> Collection not found: metaalert_index at
> org.apache.metron.rest.service.impl.UpdateServiceImpl.patch(UpdateServiceImpl.java:50)
> at
> org.apache.metron.rest.controller.UpdateController.patch(UpdateController.java:52)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:848)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at
>
[jira] [Updated] (METRON-1613) Metaalerts status update broken in Alerts UI
[
https://issues.apache.org/jira/browse/METRON-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1613:
---
Fix Version/s: 0.6.0
> Metaalerts status update broken in Alerts UI
>
>
> Key: METRON-1613
> URL: https://issues.apache.org/jira/browse/METRON-1613
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> Steps to reproduce:
> 1. Launch Alerts UI
> 2. Create a metaalert (E.g. based on source type)
> 3. Click on the Metaalert and attempt to change the status using the right
> side pane.
> 4. The Metaalert status is not changed.
> Following error is seen in the metron-rest.log
> {code:java}
> 18/05/23 16:58:32 ERROR controller.RestExceptionHandler: Encountered error:
> Collection not found: metaalert_index org.apache.metron.rest.RestException:
> Collection not found: metaalert_index at
> org.apache.metron.rest.service.impl.UpdateServiceImpl.patch(UpdateServiceImpl.java:50)
> at
> org.apache.metron.rest.controller.UpdateController.patch(UpdateController.java:52)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:848)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
> at
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at
>
[jira] [Assigned] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1588: -- Assignee: Nick Allen > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Assignee: Nick Allen >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1588: --- Fix Version/s: 0.6.0 > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Priority: Critical > Fix For: 0.6.0 > > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1532) Getting started documentation improvements
[ https://issues.apache.org/jira/browse/METRON-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1532: -- Assignee: Shane Ardell > Getting started documentation improvements > -- > > Key: METRON-1532 > URL: https://issues.apache.org/jira/browse/METRON-1532 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Trivial > Fix For: 0.6.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > After running through a fresh install of full dev on my local machine, I > think the documentation could improve by including some essential credential > information in the full dev README. The credentials exist elsewhere in the > Metron documentation, but for someone spinning up full dev for the first > time, it's important to have credentials for apps like the Metron UI and > Ambari easy to find. > I also think it's worth documenting how to make full dev more stable on your > machine by turning off various services via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1448) Update SolrWriter to conform to new collection strategy
[ https://issues.apache.org/jira/browse/METRON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1448: --- Fix Version/s: 0.6.0 > Update SolrWriter to conform to new collection strategy > --- > > Key: METRON-1448 > URL: https://issues.apache.org/jira/browse/METRON-1448 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently the SolrWriter presumes a single collection to be written to. The > new collection strategy for Solr implies a collection per sensor. Also, > there are a few rough edges in the writer which could stand smoothing: > * By default, we use solr's implicit commit mechanism, rather than > committing at the batch granularity. This may result in lost data on worker > failure. > * We do not use the the batch add api, but rather message-by-message add -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1448) Update SolrWriter to conform to new collection strategy
[ https://issues.apache.org/jira/browse/METRON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1448: -- Assignee: Nick Allen (was: Casey Stella) > Update SolrWriter to conform to new collection strategy > --- > > Key: METRON-1448 > URL: https://issues.apache.org/jira/browse/METRON-1448 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > Currently the SolrWriter presumes a single collection to be written to. The > new collection strategy for Solr implies a collection per sensor. Also, > there are a few rough edges in the writer which could stand smoothing: > * By default, we use solr's implicit commit mechanism, rather than > committing at the batch granularity. This may result in lost data on worker > failure. > * We do not use the the batch add api, but rather message-by-message add -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1532) Getting started documentation improvements
[ https://issues.apache.org/jira/browse/METRON-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1532: --- Fix Version/s: (was: 0.5.0) 0.6.0 > Getting started documentation improvements > -- > > Key: METRON-1532 > URL: https://issues.apache.org/jira/browse/METRON-1532 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Trivial > Fix For: 0.6.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > After running through a fresh install of full dev on my local machine, I > think the documentation could improve by including some essential credential > information in the full dev README. The credentials exist elsewhere in the > Metron documentation, but for someone spinning up full dev for the first > time, it's important to have credentials for apps like the Metron UI and > Ambari easy to find. > I also think it's worth documenting how to make full dev more stable on your > machine by turning off various services via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1448) Update SolrWriter to conform to new collection strategy
[ https://issues.apache.org/jira/browse/METRON-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1448: -- Assignee: Casey Stella > Update SolrWriter to conform to new collection strategy > --- > > Key: METRON-1448 > URL: https://issues.apache.org/jira/browse/METRON-1448 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > > Currently the SolrWriter presumes a single collection to be written to. The > new collection strategy for Solr implies a collection per sensor. Also, > there are a few rough edges in the writer which could stand smoothing: > * By default, we use solr's implicit commit mechanism, rather than > committing at the batch granularity. This may result in lost data on worker > failure. > * We do not use the the batch add api, but rather message-by-message add -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1416) Upgrade Solr
[ https://issues.apache.org/jira/browse/METRON-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1416: --- Fix Version/s: 0.6.0 > Upgrade Solr > > > Key: METRON-1416 > URL: https://issues.apache.org/jira/browse/METRON-1416 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > From the discussion thread: > > Now that we have ES at a modern version, we should consider bringing Solr to > a modern version as well. > > The focus of this work would be to get us in a place where Solr is upgraded, > along with the related work of building out the Solr functionality to parity > with Elasticsearch. The goal would not be to add net new functionality, just > to get Solr and ES in the same place for the alerts UI and REST interface. > Additionally, it would include the various supporting necessities such as > ensuring associated DAOs are testable, and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1416) Upgrade Solr
[ https://issues.apache.org/jira/browse/METRON-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1416: -- Assignee: Justin Leet > Upgrade Solr > > > Key: METRON-1416 > URL: https://issues.apache.org/jira/browse/METRON-1416 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > From the discussion thread: > > Now that we have ES at a modern version, we should consider bringing Solr to > a modern version as well. > > The focus of this work would be to get us in a place where Solr is upgraded, > along with the related work of building out the Solr functionality to parity > with Elasticsearch. The goal would not be to add net new functionality, just > to get Solr and ES in the same place for the alerts UI and REST interface. > Additionally, it would include the various supporting necessities such as > ensuring associated DAOs are testable, and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1651) Fixing failing protractor e2e test
[ https://issues.apache.org/jira/browse/METRON-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1651: --- Fix Version/s: 0.6.0 > Fixing failing protractor e2e test > -- > > Key: METRON-1651 > URL: https://issues.apache.org/jira/browse/METRON-1651 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > Attachments: e2e-assertion-errors.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1751) Storm Profiler dies when consuming null message
[
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1751:
---
Fix Version/s: 0.6.0
> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> When You publish a null message to the profiler input kafka topic which is
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Async loop died! java.lang.RuntimeException:
> java.lang.NullPointerException at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by:
> java.lang.NullPointerException at java.lang.String.(String.java:491)
> ~[?:1.8.0_181] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
> ~[stormjar.jar:?] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
> ~[stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even if you start up the profiler again and run it
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1554) Pcap Query Panel
[ https://issues.apache.org/jira/browse/METRON-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1554: -- Assignee: Ryan Merriman > Pcap Query Panel > > > Key: METRON-1554 > URL: https://issues.apache.org/jira/browse/METRON-1554 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Legacy OpenSOC included a panel in Kibana that allowed users to query for > pcap data. We would like to add this feature back into Metron. There are 2 > discussions happening on the dev list where we are gathering user > requirements: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E] > and working through the backend architecture: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E] > Forthcoming sub tasks will be based on the outcome of these discussions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1592) Unable to use third party parser with Storm versions >= 1.1.0
[
https://issues.apache.org/jira/browse/METRON-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1592:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Unable to use third party parser with Storm versions >= 1.1.0
> -
>
> Key: METRON-1592
> URL: https://issues.apache.org/jira/browse/METRON-1592
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> When launching a third party parser using
> $METRON_HOME/bin/start_parser_topology.sh in HDP 2.6, Storm 1.1.0 the
> following exception is thrown.
>
> {code:java}
> ClassNotFoundException:
> org.apache.metron.parsers.topology.MergeAndShadeTransformer{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1544:
---
Fix Version/s: (was: 0.5.0)
0.6.0
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1569: --- Fix Version/s: (was: 0.5.0) 0.6.0 > Allow user to change field name conversion when indexing to Elasticsearch > - > > Key: METRON-1569 > URL: https://issues.apache.org/jira/browse/METRON-1569 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > The `ElasticsearchWriter` has a mechanism to transform the field names of a > message before it is written to Elasticsearch. Right now this mechanism is > hard-coded to replace all '.' dots with ':' colons. > This mechanism was needed for Elasticsearch 2.x which did not allow dots in > field names. Now that Metron supports Elasticsearch 5.x this is no longer a > problem. > A user should be able to configure the field name transformation when writing > to Elasticsearch, as needed. > While it might have been simpler to just remove the de-dotting mechanism, > this would break backwards compatibility. Providing users with a means to > configure this mechanism provides them with an upgrade path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
[
https://issues.apache.org/jira/browse/METRON-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1598:
---
Fix Version/s: (was: Next + 1)
0.6.0
> NoClassDefFoundError when running with Elasticsearch X-Pack
> ---
>
> Key: METRON-1598
> URL: https://issues.apache.org/jira/browse/METRON-1598
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> When running on Elasticsearch with X-Pack enabled, the following exception is
> thrown in the Metron REST logs.
> {code}
> 2018-05-23T07:43:16.000 ERROR
> [org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to
> convert search request to JSON java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/dataformat/smile/SmileParser at
> org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
> at
> org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
> {code}
> This does not happen with all queries, just some queries. For example, the
> following query completes successfully.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : null
> }
> {code}
> On the other hand, this query fails with the exception shown above.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : []
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1533:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
> Fix For: 0.6.0
>
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1553) Validate JIRA Script Error
[
https://issues.apache.org/jira/browse/METRON-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1553:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Validate JIRA Script Error
> --
>
> Key: METRON-1553
> URL: https://issues.apache.org/jira/browse/METRON-1553
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> {code:java}
> $ ./validate-jira-for-release --version="0.4.3"
> --start=tags/apache-metron-0.4.2-release
> Cloning into 'metron-0.4.3'...
> remote: Counting objects: 39871, done.
> remote: Compressing objects: 100% (13140/13140), done.
> remote: Total 39871 (delta 18078), reused 37550 (delta 17019)
> Receiving objects: 100% (39871/39871), 54.34 MiB | 6.23 MiB/s, done.
> Resolving deltas: 100% (18078/18078), done.
> fatal: Not a git repository (or any of the parent directories): .git
> Fetching origin
> JIRA STATUS FIX VERSION ASSIGNEE FIX
> METRON-1549 Done Michael Miklavcic
> https://issues.apache.org/jira/browse/METRON-1549
> METRON-1541 To Do Unassigned https://issues.apache.org/jira/browse/METRON-1541
> ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1584:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
> random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Terminating messaging
> context
> 2018-05-24 09:21:22.238 o.a.s.d.worker Thread-20 [INFO] Shutting down
> executors
> 2018-05-24 09:21:22.238 o.a.s.d.executor Thread-20 [INFO] Shutting down
> executor
> __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
> 2018-05-24 09:21:22.239 o.a.s.util Thread-6-disruptor-executor[2
> 2]-send-queue [INFO] Async loop interrupted!
> 2018-05-24 09:21:22.239 o.a.s.util
>
[jira] [Updated] (METRON-1572) Enhance KAFKA_PUT function
[ https://issues.apache.org/jira/browse/METRON-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1572: --- Fix Version/s: (was: Next + 1) 0.6.0 > Enhance KAFKA_PUT function > -- > > Key: METRON-1572 > URL: https://issues.apache.org/jira/browse/METRON-1572 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > Enhance the KAFKA_PUT function to accept either a List of String or a String. > This makes it simpler to send either 1 message or a bunch of messages. > KAFKA_PUT should queue up all messages to be sent before waiting on a > response. This improves response when sending a large number of messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
[ https://issues.apache.org/jira/browse/METRON-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1609: --- Fix Version/s: (was: Next + 1) 0.6.0 > Elasticsearch settings in Ambari should not be required if Solr is the indexer > -- > > Key: METRON-1609 > URL: https://issues.apache.org/jira/browse/METRON-1609 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-06 at 12.34.13 PM.png, Screen Shot > 2018-06-06 at 12.37.08 PM.png > > > When a user deploys Metron with Solr, the Mpack requires that the user > populate the Index Settings > Elasticsearch Hosts field to continue, even > when deploying a Metron cluster that indexes to Solr. > The other Elasticsearch specific fields on the Index Settings page are also > required, although defaults are provided and thus the user is not required to > enter anything here. If you remove the provided defaults, you can see that > each of the fields are required, even when Elasticsearch will not be used. > * The Elasticsearch related properties under the Ambari 'Index settings' tab > should only be required if Elasticsearch is chosen as the indexer. > * The Solr related properties under the 'Index Settings' tab should only be > required if Solr is chosen as the indexer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1599) Allow user to define global property 'source.type.field' in Ambari
[ https://issues.apache.org/jira/browse/METRON-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1599: --- Fix Version/s: (was: Next + 1) 0.6.0 > Allow user to define global property 'source.type.field' in Ambari > --- > > Key: METRON-1599 > URL: https://issues.apache.org/jira/browse/METRON-1599 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > The user can specify the name of the message field containing the source type > in the global configuration. This is used by the Alerts UI. > Currently this necessitates a user using the CLI to add the field. This > property should be exposed to the user via Ambari. The user should be able > to define this property directly in Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1622) Allow user to define global property 'threat.triage.score.field' in Ambari
[ https://issues.apache.org/jira/browse/METRON-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1622: --- Fix Version/s: (was: Next + 1) 0.6.0 > Allow user to define global property 'threat.triage.score.field' in Ambari > --- > > Key: METRON-1622 > URL: https://issues.apache.org/jira/browse/METRON-1622 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > Based on METRON-1608 and METRON-1617, the user can specify the name of the > field containing the threat triage score in the global configuration. This is > used by the Alerts UI. > Currently a user can only change this value using the CLI. This property > should be exposed to the user via Ambari. The user should be able to define > this property directly in Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1573) Enhance KAFKA_* functions to return partition and offset details
[ https://issues.apache.org/jira/browse/METRON-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1573: --- Fix Version/s: (was: Next + 1) 0.6.0 > Enhance KAFKA_* functions to return partition and offset details > > > Key: METRON-1573 > URL: https://issues.apache.org/jira/browse/METRON-1573 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > The KAFKA_* functions currently simply return the value of the message. > There are often times when it would be useful to get more detailed > information including the message partition, offset, key, etc. > The functions should be enhanced to allow a user to optionally get a more > detailed view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1649) Intermittent Test Failure ProfileBuilderBoltTest#testFlushExpiredProfiles
[
https://issues.apache.org/jira/browse/METRON-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1649:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Intermittent Test Failure ProfileBuilderBoltTest#testFlushExpiredProfiles
> -
>
> Key: METRON-1649
> URL: https://issues.apache.org/jira/browse/METRON-1649
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> {code}
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.412 sec <<<
> FAILURE! - in org.apache.metron.profiler.bolt.ProfileBuilderBoltTest
> testFlushExpiredProfiles(org.apache.metron.profiler.bolt.ProfileBuilderBoltTest)
> Time elapsed: 0.026 sec <<< FAILURE!
> org.mockito.exceptions.verification.TooManyActualInvocations:
> outputCollector.emit(
> "hbase",
>
> );
> Wanted 1 time:
> -> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.getProfileMeasurements(ProfileBuilderBoltTest.java:265)
> But was 2 times. Undesired invocation:
> -> at org.apache.metron.profiler.bolt.HBaseEmitter.emit(HBaseEmitter.java:54)
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.getProfileMeasurements(ProfileBuilderBoltTest.java:265)
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.testFlushExpiredProfiles(ProfileBuilderBoltTest.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1633) Incorrect instructions when merging PR into feature branch
[
https://issues.apache.org/jira/browse/METRON-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1633:
---
Fix Version/s: (was: Next + 1)
0.6.0
> Incorrect instructions when merging PR into feature branch
> --
>
> Key: METRON-1633
> URL: https://issues.apache.org/jira/browse/METRON-1633
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
> Fix For: 0.6.0
>
>
> The 'dev-utilities/committer-utils/prepare-commit' script can be used to
> merge PRs against a feature branch. The script automatically determines
> which branch the PR should be merged into based on how the contributor
> submitted the PR.
> Instructions are offered at the end of the script, after the local merge has
> completed. These instructions are incorrect when merging into a feature
> branch.
> When merging into a feature branch, the instructions say...
> {code}
> Review commit carefully then run...
> cd /Users/nallen/tmp/metron-pr1056
> git push upstream master
> {code}
> But they should say...
> {code}
> Review commit carefully then run...
> cd /Users/nallen/tmp/metron-pr1056
> git push upstream feature/METRON-1416-upgrade-solr
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1646) Sensor Stubs should work when kerberized
[ https://issues.apache.org/jira/browse/METRON-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1646: --- Fix Version/s: (was: Next + 1) 0.6.0 > Sensor Stubs should work when kerberized > > > Key: METRON-1646 > URL: https://issues.apache.org/jira/browse/METRON-1646 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > The sensor-stubs do not work in a kerberized development environment. This > makes it difficult to test issues related to kerberization. Enhancing > sensor-stubs to work in a kerberized environment will make testing in the > development environment under kerberization much simpler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1652) Document X-Pack Common Problem
[ https://issues.apache.org/jira/browse/METRON-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1652: --- Fix Version/s: (was: Next + 1) 0.6.0 > Document X-Pack Common Problem > -- > > Key: METRON-1652 > URL: https://issues.apache.org/jira/browse/METRON-1652 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.6.0 > > > Improvements to the Elasticsearch X-Pack documentation to document a common > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1656) Create KAKFA_SEEK function
[ https://issues.apache.org/jira/browse/METRON-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1656: --- Fix Version/s: (was: Next + 1) 0.6.0 > Create KAKFA_SEEK function > -- > > Key: METRON-1656 > URL: https://issues.apache.org/jira/browse/METRON-1656 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > A Stellar function to be used in the REPL that allows you to seek to a > specific offset within a Kafka topic. This complements the existing Kafka > functions like KAFKA_GET, KAFKA_TAIL, and KAFKA_FIND. Like those functions > this supports the "rich" message view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1757) Storm Profiler Serialization Exception
[
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1757:
---
Affects Version/s: 0.5.0
> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> When running the Storm Profiler with multiple workers this serialization
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the
> time. It may depend on how the individual bolts are executed across the
> workers in a cluster. This likely occurs only when the HBaseBolt is executed
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing
> no-arg constructor):
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
> at
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.messaging.netty.Server.received(Server.java:254)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>
[jira] [Updated] (METRON-1586) Defaulting for the source type field in alerts UI does not work
[
https://issues.apache.org/jira/browse/METRON-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1586:
---
Fix Version/s: (was: 0.5.0)
0.6.0
> Defaulting for the source type field in alerts UI does not work
> ---
>
> Key: METRON-1586
> URL: https://issues.apache.org/jira/browse/METRON-1586
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> The alerts UI does not allow you to display an individual alert. The error is
> a 404 against the findOne endpoint becuase the sensorType is set to
> "undefined". We should be defaulting this to source:type in the UI.
> The POST data is:
> {guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
> The core issue is everywhere in
> [https://github.com/apache/metron/pull/1010/files] that we're calling
> this.globalConfig['source.type.field'] should fall-back to source:type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1752) Prevent package.lock from changing during build
[ https://issues.apache.org/jira/browse/METRON-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1752: -- Assignee: Shane Ardell > Prevent package.lock from changing during build > --- > > Key: METRON-1752 > URL: https://issues.apache.org/jira/browse/METRON-1752 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Minor > > As referenced in this mail list discussion, the package-lock.json file in > metron-alerts updates whenever we run a maven build: > https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E > The fix for this was originally included in [PR #1096, which upgrades the > Angular version used in the alerts > ui|https://github.com/apache/metron/pull/1096], but as mentioned in the > comments, it would be best to fix this bug in a separate issue in order to > simplify its inclusion in the next release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1724) Date/time validation missing in PCAP query
[ https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1724: -- Assignee: Tibor Meller > Date/time validation missing in PCAP query > -- > > Key: METRON-1724 > URL: https://issues.apache.org/jira/browse/METRON-1724 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > Validation formula should be the following: > From < To < current date/time > > Validation messages: > Selected date range is invalid. The "To" date must be later than the "From" > date and the "To" date cannot be in the future. > Source IP address format is invalid. Use valid v4IP format, for example, > [192.168.0.1|http://192.168.0.1/]. > Source port is invalid. Port number must be within the range of 0-65535. > Destination IP address format is invalid. Use valid v4IP format, for example, > [192.168.0.1|http://192.168.0.1/]. > Destination port is invalid. Port number must be within the range of 0-65535. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603322#comment-16603322 ] ASF GitHub Bot commented on METRON-1476: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1096 +1 Looks good. Thanks for all the hard work on this. I validated full dev by spinning it up, running service check, manually validated the Alerts UI, manually validated the Management UI, and successfully ran the e2e tests. > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1096: METRON-1476: Update to Angular 6.1.3
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1096 +1 Looks good. Thanks for all the hard work on this. I validated full dev by spinning it up, running service check, manually validated the Alerts UI, manually validated the Management UI, and successfully ran the e2e tests. ---
[jira] [Commented] (METRON-1615) Fill in some endpoint details from Ambari info
[ https://issues.apache.org/jira/browse/METRON-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603253#comment-16603253 ] ASF GitHub Bot commented on METRON-1615: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1060 @simonellistonball - I'm happy to test this out if you can resolve the conflicts. > Fill in some endpoint details from Ambari info > -- > > Key: METRON-1615 > URL: https://issues.apache.org/jira/browse/METRON-1615 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Simon Elliston Ball >Priority: Trivial > > When installing a metron cluster, users are required to fill in values that > Ambari should already know: > * location of storm ui > * location of elastic search > * location of zeppelin notebook server > While it is useful to be able to override the ES location if you're using a > non-metron managed ES, the others are absolutely a function of the Metron > cluster, whose locations are known to Ambari, so users should not be expected > to fill them in. > If the ES hosts settings is left blank, it should be calculated from info in > the ES mpack. > The kibana service in the elastic mpack should also automatically set the url > for ES masters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1060: METRON-1615 Default endpoint locations based on Ambari s...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1060 @simonellistonball - I'm happy to test this out if you can resolve the conflicts. ---
[jira] [Commented] (METRON-1764) Update version to 0.6.0
[ https://issues.apache.org/jira/browse/METRON-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603249#comment-16603249 ] ASF GitHub Bot commented on METRON-1764: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1183 METRON-1764: Update version to 0.6.0 ## Contributor Comments Updating version in prep for upcoming release. Reviewers should check a couple things * All necessary versions are changed to 0.6.0 * Nothing is left over as 0.5.1 ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron version060 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1183 commit a5ee2761e613d5d9548d55fc91b1651cab516e0b Author: Justin Leet Date: 2018-09-04T16:04:23Z Updating version to 0.6.0 > Update version to 0.6.0 > --- > > Key: METRON-1764 > URL: https://issues.apache.org/jira/browse/METRON-1764 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > In anticipation of release, we need to update the version since we agreed on > 0.6.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1183: METRON-1764: Update version to 0.6.0
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1183 METRON-1764: Update version to 0.6.0 ## Contributor Comments Updating version in prep for upcoming release. Reviewers should check a couple things * All necessary versions are changed to 0.6.0 * Nothing is left over as 0.5.1 ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron version060 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1183.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1183 commit a5ee2761e613d5d9548d55fc91b1651cab516e0b Author: Justin Leet Date: 2018-09-04T16:04:23Z Updating version to 0.6.0 ---
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603190#comment-16603190 ] ASF GitHub Bot commented on METRON-1476: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214959210 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- Actually, I think this was added as part of #1177 . I will update some of these instructions in a separate follow-on PR to help clarify things for new users. No need to address this here. > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1096: METRON-1476: Update to Angular 6.1.3
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214959210 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- Actually, I think this was added as part of #1177 . I will update some of these instructions in a separate follow-on PR to help clarify things for new users. No need to address this here. ---
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603133#comment-16603133 ] ASF GitHub Bot commented on METRON-1476: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214945499 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- However we update the instructions here, we should also do the same in `metron-interface/metron-config/README.md`. > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1096: METRON-1476: Update to Angular 6.1.3
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214945499 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- However we update the instructions here, we should also do the same in `metron-interface/metron-config/README.md`. ---
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603131#comment-16603131 ] ASF GitHub Bot commented on METRON-1476: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214944338 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- Do you have any recommendations for how we should install NVM? I am assuming a `brew install nvm` will do the trick on a Mac? Or do you prefer what is mentioned [here](https://github.com/creationix/nvm#install-script)? Maybe we should embed a link to [this](https://github.com/creationix/nvm#install-script) if you think this is the right way to install. > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1096: METRON-1476: Update to Angular 6.1.3
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1096#discussion_r214944338 --- Diff: metron-interface/metron-alerts/README.md --- @@ -45,7 +45,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r * The Management UI should be installed (which includes [Express](https://expressjs.com/)) * The alerts can be populated using Full Dev or any other setup * UI is developed using angular4 and uses angular-cli -* node.JS >= 7.8.0 +* nvm (or a similar node verison manager) should be installed. The node version required for this project is listed in the [.nvmrc](https://github.com/creationix/nvm#nvmrc) file. --- End diff -- Do you have any recommendations for how we should install NVM? I am assuming a `brew install nvm` will do the trick on a Mac? Or do you prefer what is mentioned [here](https://github.com/creationix/nvm#install-script)? Maybe we should embed a link to [this](https://github.com/creationix/nvm#install-script) if you think this is the right way to install. ---
[jira] [Created] (METRON-1763) Some Management UI endpoints returning 404
Shane Ardell created METRON-1763: Summary: Some Management UI endpoints returning 404 Key: METRON-1763 URL: https://issues.apache.org/jira/browse/METRON-1763 Project: Metron Issue Type: Bug Reporter: Shane Ardell Attachments: Screen Shot 2018-09-03 at 2.16.47 PM.png While testing on full-dev, I've noticed many 404 errors in the browser console when I click to open the details panel for different parsers in the management UI. Is this by design, or are these endpoints broken? I tested these endpoints in swagger and had the same results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1476: --- Summary: Update to Angular 6.1.3 (was: Update angular) > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages
[ https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602913#comment-16602913 ] ASF GitHub Bot commented on METRON-1750: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 New upstream integrated now. > Create Parser for Syslog RFC 5424 Messages > -- > > Key: METRON-1750 > URL: https://issues.apache.org/jira/browse/METRON-1750 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > > Create a Metron parser for working with valid RFC 5424 syslog messages, > including support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1175: METRON-1750 Metron Parser for valid RFC 5424 Syslog mess...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1175 New upstream integrated now. ---
[GitHub] metron issue #1180: METRON-1759: PCAP UI: Removing wrong Input annotations f...
Github user ruffle1986 commented on the issue: https://github.com/apache/metron/pull/1180 +1. Nice catch! ð» ---
