[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/803 @merrimanr METRON-1272 has broken few of our assertions and these are cascaded failures bcoz of them. I am working on fixing them. ---
[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/814 the issue with MAP() is resolved as well ---
[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/814
OK, the validation issues are a big big deal. I have changed the match
functionality so that they work in that context, supporting guard expressions
and requiring the default operation.
1. Require default statement
2. change the action separator from : to => so that...
3. we can use conditional_exp and logical_exp ( can check for EXISTS() with
IF etc )
I still have to do the hack for the match{ var1 => 'ok' , default =>
'notOk' }
since the lone null as logical doesn't work as stated earlier.
So the new syntax is
match { logical_or_conditional_expr => transform_exp , default =>
transform_exp }
---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r147853082
--- Diff:
metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
@@ -159,4 +159,34 @@ export class TreeViewPage {
return column.getText();
});
}
+
+ clickOnMergeAlerts(groupName: string) {
+return element(by.css('[data-name="' + groupName + '"]
.fa-link')).click();
+ }
+
+ getConfirmationText() {
+browser.sleep(1000);
+let dialogElement = element(by.css('.metron-dialog .modal-header
.close'));
+return waitForElementVisibility(dialogElement).then(() =>
element(by.css('.metron-dialog .modal-body')).getText());
+ }
+
+ clickNoForConfirmation() {
+browser.sleep(1000);
+let dialogElement = element(by.css('.metron-dialog .modal-header
.close'));
+let maskElement = element(by.css('.modal-backdrop.fade'));
+waitForElementVisibility(dialogElement).then(() =>
element(by.css('.metron-dialog')).element(by.buttonText('Cancel')).click())
+.then(() => waitForElementInVisibility(maskElement));
+ }
+
+ clickYesForConfirmation() {
+browser.sleep(1000);
--- End diff --
Is this sleep statement absolutely necessary?
---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r147853051
--- Diff:
metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
@@ -159,4 +159,34 @@ export class TreeViewPage {
return column.getText();
});
}
+
+ clickOnMergeAlerts(groupName: string) {
+return element(by.css('[data-name="' + groupName + '"]
.fa-link')).click();
+ }
+
+ getConfirmationText() {
+browser.sleep(1000);
--- End diff --
Is this sleep statement absolutely necessary?
---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r147853067
--- Diff:
metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
@@ -159,4 +159,34 @@ export class TreeViewPage {
return column.getText();
});
}
+
+ clickOnMergeAlerts(groupName: string) {
+return element(by.css('[data-name="' + groupName + '"]
.fa-link')).click();
+ }
+
+ getConfirmationText() {
+browser.sleep(1000);
+let dialogElement = element(by.css('.metron-dialog .modal-header
.close'));
+return waitForElementVisibility(dialogElement).then(() =>
element(by.css('.metron-dialog .modal-body')).getText());
+ }
+
+ clickNoForConfirmation() {
+browser.sleep(1000);
--- End diff --
Is this sleep statement absolutely necessary?
---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r147852935
--- Diff:
metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.po.ts ---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {browser, element, by} from 'protractor';
+
+export class MetaAlertPage {
+
+ getPageTitle() {
+return element(by.css('app-meta-alerts .form-title')).getText();
+ }
+
+ getMetaAlertsTitle() {
+return element(by.css('app-meta-alerts .title')).getText();
+ }
+
+ getAvailableMetaAlerts() {
+return element(by.css('app-meta-alerts .guid-name-container
div')).getText();
+ }
+
+ selectRadio() {
+return element.all(by.css('app-meta-alerts .checkmark')).click();
+ }
+
+ addToMetaAlert() {
+
element.all(by.css('app-meta-alerts')).get(0).element(by.buttonText('ADD')).click();
+browser.sleep(2000);
--- End diff --
Is this sleep statement absolutely necessary?
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/803 I am consistently getting these e2e test failures: ``` 1) meta-alerts workflow should have all the steps for meta alerts workflow - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. 2) metron-alerts alert status should change alert statuses - Expected 'RESOLVE' to equal 'OPEN'. - Expected 'NEW' to equal 'OPEN'. - Expected 'RESOLVE' to equal 'DISMISS'. - Expected 'NEW' to equal 'DISMISS'. - Expected 'RESOLVE' to equal 'ESCALATE'. - Expected 'NEW' to equal 'ESCALATE'. - Expected 'NEW' to equal 'RESOLVE'. 3) metron-alerts alert status should add comments for table view - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. 4) metron-alerts alert status should add comments for tree view - Failed: unknown error: Element is not clickable at point (2485, 311). Other element would receive the click: (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.11.6 x86_64) ``` Not sure if 3) and 4) are cascading errors caused by previous errors. I suspect 2) was introduced by https://github.com/apache/metron/pull/796. ---
[GitHub] metron issue #820: METRON-1287 Full Dev Fails When Installing EPEL Repositor...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/820 I was able to successfully build full dev with this change. +1 ---
Re: [DISCUSS] Using Yarn package manager for metron-alerts
Would love to revive this - I think this could help drastically reduce our build times for metron-interface, which locally just took me 9 minutes in non-parallel mode with -DskipTests set. This is a really good suggestion even just for the offline install and version locking, as pointed out by Nick. Best, Mike On Thu, Aug 17, 2017 at 8:12 AM, Ryan Merriman wrote: > Thanks for this Raghu. You make a pretty compelling argument. I'm +1 on > moving to yarn. > > Ryan > > On Wed, Aug 16, 2017 at 3:51 PM, Nick Allen wrote: > > > It is also my understanding that > > there is no hard cut-over to yarn > > . > > After we > > introduce the yarn.lock > > > > , > > as a developer you can choose to continue to use npm or switch to yarn. > > > > Other developers on the project can keep using npm, so you don’t need to > > > get everyone on your project to convert at the same time. The > developers > > > using yarn will all get exactly the same configuration as each other, > and > > > the developers using npm may get slightly different configurations, > which > > > is the intended behavior of npm. > > > > > > https://yarnpkg.com/lang/en/docs/migrating-from-npm/ > > > > > > Oh, and I just switched metron-alerts projects to yarn (as a test) and > > performed an offline install. It was stupid simple. > > > > > > > > > > On Wed, Aug 16, 2017 at 4:12 PM Nick Allen wrote: > > > > > Thanks for laying this all out for us, Raghu. Based on the built-in > > > support for offline installs and version locking, I think this is a > great > > > suggestion. (However unfortunate the namespace collision might be.) > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 16, 2017 at 8:51 AM RaghuMitra Kandikonda < > > > raghumitra@gmail.com> wrote: > > > > > >> I would like to start a discussion around using 'yarn' for managing > > >> dependencies for metron-alerts instead of 'npm'. > > >> > > >> This article beautifully summarizes the need of yarn and npm. > > >> (https://code.facebook.com/posts/1840075619545360) > > >> > > >> If you have read the above article you can skip the next two sections > > >> and jump to 'Additional advantages of Yarn' > > >> > > >> > > >> > > >> === > > >> Why do we need a new package manager ?. > > >> > > >> While 'npm' does a good job for downloading all the required > > >> dependencies. npm always tries to download the latest and greatest > > >> versions of all these dependencies. This would create a problem in > > >> replicating the same build every time we build. Having hard coded > > >> versions in the package.json seems like a possible solution but this > > >> will prevent us from knowing that a library has been updated. In JS > > >> world the version updates are very frequent and we might be missing on > > >> some of the latest updates and some of these updates might be related > > >> to security or a cool feature we would like to have in our code base. > > >> Ex: Angular made 10 releases in last two months, bootstrap made 2 > > >> releases in last two months. > > >> > > >> > > >> > > >> === > > >> What is Yarn ?. > > >> > > >> Yarn is a new age package manager that can (needs to) be installed > > >> over npm (or bower). Yarn resolves issues around versioning and > > >> non-determinism of JS dependencies by using lock files and an install > > >> algorithm that is deterministic and reliable. These lock files lock > > >> the installed dependencies to a specific version and ensure that every > > >> install results in the exact same file structure in node_modules > > >> across all machines. This kind of a locking mechanism is not available > > >> with vanilla node. > > >> > > >> > > >> > > >> === > > >> Additional advantages of Yarn ?. > > >> > > >> 1.Yarn helps us to check licenses of all the frameworks we are using. > > >> (This feature is built in) > > >> 2.It will reduce the build time of UI for dev as well as in Travis as > > >> all the dependencies are cached inside '~/.config/yarn/global' > > >> 3.We can do an offline install of UI as we can zip the dependencies > > >> and supply it to Yarn instead of downloading from the internet > > >> 4.Yarn is already integrated with Travis > > >> (https://blog.travis-ci.com/2016-11-21-travis-ci-now-supports-yarn) > > >> > > >> > > >> > > >> === > > >> How to migrate ?. > > >> > > >> A yarn.lock file can be created from existing package.json file and > > >> this file would be checked in. > > >> > > >>
[GitHub] metron pull request #819: METRON-1267: Alerts UI returns a 404 when refreshi...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/819 ---
[GitHub] metron issue #819: METRON-1267: Alerts UI returns a 404 when refreshing the ...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/819 Tested this in full dev and it resolved the issue. Thanks for fixing this. +1 ---
[GitHub] metron pull request #821: METRON-1281: Remove hard-coded indices from the Al...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/821 METRON-1281: Remove hard-coded indices from the Alerts UI ## Contributor Comments This PR adjusts the Alerts UI and REST app to remove the need to hard-code the indices in the Alerts UI source code. I felt like this feature could be kept simple by having the REST SearchService supply the appropriate indices by default because: - the front end code has no knowledge of when the list of available indices changes and would have to make an extra call on every search - the underlying ElasticsearchDao lives in a different module and does not have access to the SensorIndexingConfigService which is required to look up the available indices This resulted in an additional "index.writer.name" property because the indices lookup requires this information. The "error" index doesn't make sense in an alert search context so it is excluded. The "metaalert" index is added by default in anticipation of https://github.com/apache/metron/pull/803. A SearchServiceTest had not been created yet so it has been added with coverage for this PR. An integration test for a query with default indices was also added. While working on the integration tests I noticed an intermittent failure happening frequently so I fixed that in this PR to get the build to pass consistently (GlobalConfigControllerIntegrationTest.java). I can revert that if desired but my thinking was it's probably not worth the overhead of a separate PR. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron remove_hardcoded_indices Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/821.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #821 commit 0570ba18d773688bfc2d71e2f081155583c21a2e Author: cstella Date: 2017-10-09T19:03:32Z Abstracting zookeeper substantially. commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b Author: cstella Date: 2017-10-09T20:10:57Z U
[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/814#discussion_r147785631 --- Diff: metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4 --- @@ -272,4 +278,30 @@ lambda_variable: IDENTIFIER ; +match_expr : + MATCH LBRACE match_clauses RBRACE #MatchClauses + ; + +match_clauses : + match_clause (COMMA match_clause)* + ; + +match_clause : + match_clause_check COLON match_clause_action + ; + +match_clause_action : + match_clause_action_expr #MatchClauseAction + ; +match_clause_action_expr : + transformation_expr + ; + +match_clause_check : --- End diff -- Done ---
[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/814#discussion_r147784210
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/StellarCompiler.java
---
@@ -105,59 +107,77 @@ public Object apply(ExpressionState state) {
Deque> instanceDeque = new ArrayDeque<>();
{
boolean skipElse = false;
+boolean skipMatchClauses = false;
Token token = null;
for (Iterator> it = getTokenDeque().descendingIterator();
it.hasNext(); ) {
token = it.next();
//if we've skipped an else previously, then we need to skip the
deferred tokens associated with the else.
- if(skipElse && token.getUnderlyingType() == ElseExpr.class) {
-while(it.hasNext()) {
+ if (skipElse && token.getUnderlyingType() == ElseExpr.class) {
+while (it.hasNext()) {
token = it.next();
- if(token.getUnderlyingType() == EndConditional.class) {
+ if (token.getUnderlyingType() == EndConditional.class) {
break;
}
}
skipElse = false;
}
+ if (skipMatchClauses && (token.getUnderlyingType() ==
MatchClauseEnd.class
+ || token.getUnderlyingType() == MatchClauseCheckExpr.class))
{
+while (it.hasNext()) {
--- End diff --
This is another issue with validate
---
[GitHub] metron pull request #820: METRON-1287 Full Dev Fails When Installing EPEL Re...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/820
METRON-1287 Full Dev Fails When Installing EPEL Repository
This fixes a change in the EPEL repository that is causing deployments of
Full Dev to fail.
```
TASK [epel : Get epel-repo rpm]
fatal: [node1]: FAILED! => {"changed": false, "dest":
"/tmp/epel-release.rpm", "failed": true, "msg": "Request failed", "response":
"HTTP Error 404: Not Found", "state": "absent", "status_code": 404, "url":
"http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm"}
to retry, use: --limit
@/Users/nallen/Development/metron/metron-deployment/playbooks/metron_full_install.retry
```
This has fixed the immediate issue for me, but I still want to run a full
deployment before this gets merged.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1287
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/820.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #820
commit e11bb430a5960f7f87c0aebce0dbe90cb05f2e45
Author: Nick Allen
Date: 2017-10-30T16:50:47Z
METRON-1287 Full Dev Fails When Installing EPEL Repository
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/817 ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/817 +1, thanks! ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/801 ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147737598
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -86,8 +86,13 @@ rm ${SENSOR}.template
The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
-It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed. For such a scenario, an Admin can install the template manually from
the Ambari UI by following the below flow:
+It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed.
+For such a scenario, an Admin can have the template installed in two ways:
+
+_Method 1_ - Manually from the Ambari UI by following the flow:
Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch
Template Install
+_Method 2_ - Stop the Metron Indexing service, and start it again from
Ambari UI.
+
--- End diff --
Sure, added this as well @ottobackwards
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147725937
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -86,8 +86,13 @@ rm ${SENSOR}.template
The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
-It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed. For such a scenario, an Admin can install the template manually from
the Ambari UI by following the below flow:
+It is possible that Elasticsearch service is not available when the Metron
Indexing Service startup, in that case the Elasticsearch template will not be
installed.
+For such a scenario, an Admin can have the template installed in two ways:
+
+_Method 1_ - Manually from the Ambari UI by following the flow:
Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch
Template Install
+_Method 2_ - Stop the Metron Indexing service, and start it again from
Ambari UI.
+
--- End diff --
I am not trying to nit, but this should expressly call out that we will
attempt to install the templates each START until successful.
---
_Method 2_ - Stop the Metron Indexing service, and start it again from
Ambari UI.
> Note: The Metron Indexing service tracks if it has successfully
installed the Elasticsearch templates, and will attempt to do so each time it
is Started until successful.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147714561
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Fair enough, added both methods in the latest doc change.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147711822
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
But, if that is the way it works, and is what will happen, you should
documented it. If it doesn't make sense enough to document, then change the
behavior.
We have people all over the user's list with issues understanding what is
going on with things.
"Oh, the doc doesn't say it but we *do* try to install automatically next
time or if you " isn't right.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147708307
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Okay, let me try to explain more using examples:
**Scenario 1 - Happy Path**
* Fresh install
* ES service up and running
* When the Indexing service comes up, it also installs the ES templates
* Admin can start ingesting into sensors, launch alerts UI and everything
works
**Scenario 2 - ES service down**
* Fresh install
* For some reason, the ES service is down when the Indexing service is
coming up
* We log a warning message in the Ambari install logs, and the Indexing
service starts up fine.
* Once the ES service issue is resolved, the Admin needs to install the ES
templates manually before s/he can start ingesting into sensors. This can be
done in two ways:
1) Using the Ambari UI -> Services -> Metron -> Service Actions ->
Elasticsearch Template Install
2) By stopping the Metron Indexing service from Ambari UI, and starting it
again so that it can trigger the piece of code to install the template.
Now, from a documentation perspective, point 2 above is counter intuitive
IMHO, since it would not make sense to ask the Admin to stop/start Indexing
service in order to have the ES template installed. I have hence documented
only the first option--which is also the same way it is done presently.
Let me know if this helps clarify.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147702906
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
So I am def. confused. I don't see how "It will try again next time X
happens" as the same as Go and do it manually in ambari
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147699187
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Ah okay, this comment and review is the same scenario described in the
README as:
> It is possible that Elasticsearch service is not available when the
Metron Indexing Service startup, in that case the Elasticsearch template will
not be installed. For such a scenario, an Admin can install the template
manually from the Ambari UI by following the below flow:
>
> Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch
Template Install
---
[GitHub] metron pull request #819: METRON-1267: Alerts UI returns a 404 when refreshi...
GitHub user iraghumitra opened a pull request: https://github.com/apache/metron/pull/819 METRON-1267: Alerts UI returns a 404 when refreshing the alerts-list page Fixed the 404 error when refreshing alerts-ui(added the missing base-dir path). Looks like the last fix didn't completely fix the issue still refresh on alerts-ui in full dev throws 404 with the following error in /var/log/metron/metron-alerts-ui.log TypeError: path must be absolute or specify root to res.sendFile at ServerResponse.sendFile (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/response.js:410:11) at /usr/metron/0.4.2/web/expressjs/alerts-server.js:71:7 at Layer.handle [as handle_request] (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/layer.js:95:5) at next (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/route.js:137:13) at Route.dispatch (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/route.js:112:3) at Layer.handle [as handle_request] (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/layer.js:95:5) at /usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/index.js:281:22 at param (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/index.js:354:14) at param (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/index.js:365:14) at Function.process_params (/usr/metron/0.4.2/web/expressjs/node_modules/express/lib/router/index.js:410:3)` ## Testing - Launch alerts-ui from full dev and refresh the page, we get a 404 on the page - Copy the file (or copy the fix) to $METRON_HOME/web/expressjs/alerts-server.js on full dev - Restart Alerts UI from Ambari - Launch alerts-ui from full dev and refresh the page it should not throws 404 ## Contributor Comments [Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.] ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/iraghumitra/incubator-metron METRON-1267 Alternatively you can revi
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147686518
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
https://github.com/apache/metron/pull/817#issuecomment-340269215
https://github.com/apache/metron/pull/817#pullrequestreview-72703358
My impression from these comments is that automatic installation could
still possibly happen. That is what I'm referring to as being missing. I may
be confused though.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147681110
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Reworded per your suggestion.
I believe the README section covers the current behavior with my change. If
you are talking about the scenarios noted in the PR description, then these are
captured in the existing ES README under the section [Using Metron with
Elasticsearch
2.x](https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x).
Please let me know which specific scenario you are referring to and I would be
happy to include them.
---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/817#discussion_r147670198
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -81,3 +81,13 @@ curl -XPUT
"http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc
'
rm ${SENSOR}.template
```
+
+## Installing Elasticsearch Templates
+
+The stock set of Elasticsearch templates for bro, snort, yaf, error index
and meta index are installed automatically during the first time install and
startup of Metron Indexing service.
+
--- End diff --
Instead of saying 'in the rare case', it might be better to say:
"It is possible that Elasticsearch is not available when the Metron
Indexing Service starts the Elasticsearch topology, in that case the
Elasticsearch templates will not be installed."
Also, this doesn't document the scenarios you have been describing fully.
---
