[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243316#comment-16243316
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/774
  
@mattf-horton I have added some more documentation, and I think the 
bundles-testing project provides a simple, integrated example.  @nickwallen and 
I talked it over, and he is going to try that.


> Bundles and Maven Plugin
> 
>
> Key: METRON-1212
> URL: https://issues.apache.org/jira/browse/METRON-1212
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The first effort will be to land the bundle system and supporting maven 
> plugin on master



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1288) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243310#comment-16243310
 ] 

ASF GitHub Bot commented on METRON-1288:


Github user CUGCR closed the pull request at:

https://github.com/apache/metron/pull/822


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1288
> URL: https://issues.apache.org/jira/browse/METRON-1288
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> -brew cask install vagrant virtualbox java docker
> +brew cask install vagrant virtualbox docker
> +brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-295) Script parsing bolt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243302#comment-16243302
 ] 

ASF GitHub Bot commented on METRON-295:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/338
  
What is the status of this effort?


> Script parsing bolt 
> 
>
> Key: METRON-295
> URL: https://issues.apache.org/jira/browse/METRON-295
> Project: Metron
>  Issue Type: New Feature
>Affects Versions: 0.2.2BETA
>Reporter: James Sirota
>Assignee: Karthik Narayanan
>Priority: Minor
>  Labels: newbie, platform
>
> In addition to having a Grok parsing bolt we need a bolt that can execute a 
> script in order to parse a telemetry.  This way you can still script the 
> parsing for telemetries for which Grok expressions are too complex, but still 
> don't have to define a java parser 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1290) Only first 10 alerts are update when a MetaAlert status is changed to inactive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241759#comment-16241759
 ] 

ASF GitHub Bot commented on METRON-1290:


Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/825
  
+1 for functionality, tested on full dev when status is changed on a 
meta-alert all the alerts in the meta-alert are updated with the correct status 


> Only first 10 alerts are update when a MetaAlert status is changed to inactive
> --
>
> Key: METRON-1290
> URL: https://issues.apache.org/jira/browse/METRON-1290
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241771#comment-16241771
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/832
  
@nickwallen really liked the way you annotated the PR with your comments. 
You made it really easy to understand and saved a lot of time for me. As you 
rightly pointed out I will wait till all other dependent PR's to come in before 
giving this a  spin.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242868#comment-16242868
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149501925
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

The latest has the change that makes the ignore indices configurable.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242729#comment-16242729
 ] 

ASF GitHub Bot commented on METRON-1307:


GitHub user brianhurley opened a pull request:

https://github.com/apache/metron/pull/835

METRON-1307   Force install of java8 since java9 does not appear to work

Force install of java8 since java9 does not appear to work with the current 
scripts in the README.md file

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brianhurley/metron patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #835






> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242739#comment-16242739
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/834
  
  My comment is more 'big picture'.  I don't think it nec. applies to how 
this PR is resolved.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-813) Migrate bro-plugin-kafka to be a bro package

[Jon Zeolla (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241993#comment-16241993
 ] 

Jon Zeolla commented on METRON-813:
---

Note to self:  Consider adding a memleak test.  See 
https://github.com/grigorescu/bro-1/blob/master/testing/btest/core/leaks/dataseries.bro
 and https://github.com/bro/bro/tree/master/testing/btest/core/leaks

> Migrate bro-plugin-kafka to be a bro package
> 
>
> Key: METRON-813
> URL: https://issues.apache.org/jira/browse/METRON-813
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Per a 
> [discussion](https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E)
>  on the dev mailing list, the kafka plugin should be hosted as a [bro 
> package](https://github.com/bro/packages) and mirrored to 
> https://github.com/apache/incubator-metron-bro-plugin-kafka.  The bro kafka 
> plugin should be installed, when necessary, using the bro package.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241914#comment-16241914
 ] 

ASF GitHub Bot commented on METRON-1303:


GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron-bro-plugin-kafka/pull/1

METRON-1303:  Reorganize the metron-bro-plugin-kafka

This should rename Bro to Apache everywhere that's necessary for this 
plugin.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1303

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron-bro-plugin-kafka/pull/1.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1


commit f21e51f4f91452d66b644b1c041e9a3ae3b39bd7
Author: Jon Zeolla 
Date:   2017-11-07T12:12:53Z

METRON-1303:  Reorganize the metron-bro-plugin-kafka




> Reorganize the metron-bro-plugin-kafka
> --
>
> Key: METRON-1303
> URL: https://issues.apache.org/jira/browse/METRON-1303
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>
> Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
> which is somewhat confusing based on the way bro packages are currently 
> named/sorted[1].  Based on this, we should align the plugin namespace to run 
> under the owner of the repo, which is apache.
> 1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[Jon Zeolla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla reassigned METRON-1303:
--

Assignee: Jon Zeolla

> Reorganize the metron-bro-plugin-kafka
> --
>
> Key: METRON-1303
> URL: https://issues.apache.org/jira/browse/METRON-1303
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
> which is somewhat confusing based on the way bro packages are currently 
> named/sorted[1].  Based on this, we should align the plugin namespace to run 
> under the owner of the repo, which is apache.
> 1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-1304:
--

 Summary: Allow metron-bro-plugin-kafka to include or exclude logs
 Key: METRON-1304
 URL: https://issues.apache.org/jira/browse/METRON-1304
 Project: Metron
  Issue Type: Bug
Reporter: Jon Zeolla


Right now, you must specify which logs you want to send to kafka via 
metron-bro-plugin-kafka.  This would allow the additional feature of excluding 
certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241961#comment-16241961
 ] 

ASF GitHub Bot commented on METRON-1303:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/1
  
or if it is building can we link that to github?


> Reorganize the metron-bro-plugin-kafka
> --
>
> Key: METRON-1303
> URL: https://issues.apache.org/jira/browse/METRON-1303
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
> which is somewhat confusing based on the way bro packages are currently 
> named/sorted[1].  Based on this, we should align the plugin namespace to run 
> under the owner of the repo, which is apache.
> 1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-1303:
--

 Summary: Reorganize the metron-bro-plugin-kafka
 Key: METRON-1303
 URL: https://issues.apache.org/jira/browse/METRON-1303
 Project: Metron
  Issue Type: Sub-task
Reporter: Jon Zeolla


Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
which is somewhat confusing based on the way bro packages are currently 
named/sorted[1].  Based on this, we should align the plugin namespace to run 
under the owner of the repo, which is apache.

1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241982#comment-16241982
 ] 

ASF GitHub Bot commented on METRON-1304:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/2
  
logs_to_exclude should be in the readme, and not just the bro file


> Allow metron-bro-plugin-kafka to include or exclude logs
> 
>
> Key: METRON-1304
> URL: https://issues.apache.org/jira/browse/METRON-1304
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Right now, you must specify which logs you want to send to kafka via 
> metron-bro-plugin-kafka.  This would allow the additional feature of 
> excluding certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241987#comment-16241987
 ] 

ASF GitHub Bot commented on METRON-1304:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/2
  
Yup, I'm typing that up as we speak.


> Allow metron-bro-plugin-kafka to include or exclude logs
> 
>
> Key: METRON-1304
> URL: https://issues.apache.org/jira/browse/METRON-1304
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Right now, you must specify which logs you want to send to kafka via 
> metron-bro-plugin-kafka.  This would allow the additional feature of 
> excluding certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-1305) Add metron-bro-plugin-kafka to travis and expand on btests

[Jon Zeolla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla reassigned METRON-1305:
--

Assignee: Jon Zeolla

> Add metron-bro-plugin-kafka to travis and expand on btests
> --
>
> Key: METRON-1305
> URL: https://issues.apache.org/jira/browse/METRON-1305
> Project: Metron
>  Issue Type: Test
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> After metron-bro-plugin-kafka is turned into a plugin, we should expand on 
> its btests and get it into travis.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242100#comment-16242100
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/829
  
Hi @nickwallen , thank you for the fix. It is my bad, I should have added 
the line `commands = IndexingCommands(params)` before the `try` block in my 
previous fix :(.

I am under way validating and will post my observation. However, given that 
the indexing templates are a must have for the alerts UI to work properly, 
should we remove the try/expect block and allow the install to fail in the 
event it was not successfully installed? Or do you think we should WARN and 
move on?


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242129#comment-16242129
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/829


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242130#comment-16242130
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/829
  
+1.  I agree with Casey that this should fail instead of WARN.  If 
templates are not installed it puts us in a bad state and requires manual 
intervention that may not be obvious.


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242452#comment-16242452
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/829
  
@nickwallen , for the record full dev spun up fine with the fix. Thank you!


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1288) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243304#comment-16243304
 ] 

ASF GitHub Bot commented on METRON-1288:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/822
  
Since this has been landed in the other PR, can you close this one please?


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1288
> URL: https://issues.apache.org/jira/browse/METRON-1288
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> -brew cask install vagrant virtualbox java docker
> +brew cask install vagrant virtualbox docker
> +brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242741#comment-16242741
 ] 

ASF GitHub Bot commented on METRON-1307:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/835
  
thank you @brianhurley.

+1, I will commit


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242104#comment-16242104
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/829
  
> However, given that the indexing templates are a must have for the alerts 
UI to work properly, should we remove the try/expect block and allow the 
install to fail in the event it was not successfully installed? Or do you think 
we should WARN and move on?

I think we should leave that as a separate discussion.  It's definitely 
worthy of a discussion, but I think we should just fix what we have in this PR. 
  Thanks for validating!


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242134#comment-16242134
 ] 

Nick Allen commented on METRON-1306:


Index templates can only be installed when starting the Indexing topology, not 
during install.  During install Elasticsearch is most likely not running yet.

That being said, I think the ask here is that starting the indexing topology 
should fail, if the index templates fail to install.  Correct me if I am wrong.

> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242143#comment-16242143
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/834
  
Just a comment on the description of the JIRA, not the code change itself.  
Index templates can only be installed when starting the Indexing topology, not 
during install. During install, Elasticsearch is most likely not running yet.  
That being said, I think the ask here is that starting the indexing topology 
should fail, if the index templates fail to install.  And this is what you have 
with your PR.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242213#comment-16242213
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/834
  
Nick's points are spot-on, I think.  I think the core abstraction issue is 
that we're doing ES-specific things in the indexing section.  We probably want 
to decouple indexing from ES a bit and have a separate component for ES 
configuration installation.  This also might not be something we want to do as 
part of ambari, but rather the installation of templates might should be done 
upon parser start time.  This might bear rethinking is what I'm saying.

That being said, I think I'm still more uncomfortable with a masked failure 
like we have it now.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242252#comment-16242252
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149411916
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

While I agree with the inclusion of `.kibana` here, what are the 
consequences for those users with other, non-metron, indices?  Should we allow 
users to specify this list in the REST application.yml?


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1275) Fix Metron Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242272#comment-16242272
 ] 

ASF GitHub Bot commented on METRON-1275:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/833
  
Yes @lvets your +1 is definitely relevant.  I wish more community members 
would review PRs.


> Fix Metron Documentation
> 
>
> Key: METRON-1275
> URL: https://issues.apache.org/jira/browse/METRON-1275
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> I am combing through Metron documentation and I will have various updates for 
> minor fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1290) Only first 10 alerts are update when a MetaAlert status is changed to inactive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242288#comment-16242288
 ] 

ASF GitHub Bot commented on METRON-1290:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/825#discussion_r149421326
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDaoTest.java
 ---
@@ -489,4 +505,54 @@ public void testHandleMetaUpdateAlert() throws 
IOException {
 verify(mockEsDao, times(1)).getLatest(guidAlert, null);
 verify(mockEsDao, times(1)).batchUpdate(updates);
   }
+
+  @Test
+  public void getAllAlertsForMetaAlertShouldReturnAllAlerts() throws 
Exception {
--- End diff --

Agreed


> Only first 10 alerts are update when a MetaAlert status is changed to inactive
> --
>
> Key: METRON-1290
> URL: https://issues.apache.org/jira/browse/METRON-1290
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1275) Fix Metron Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242281#comment-16242281
 ] 

ASF GitHub Bot commented on METRON-1275:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/833
  
@JonZeolla I am assuming you just added additional options that are now 
available.  That being the case, +1 from me.


> Fix Metron Documentation
> 
>
> Key: METRON-1275
> URL: https://issues.apache.org/jira/browse/METRON-1275
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> I am combing through Metron documentation and I will have various updates for 
> minor fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242250#comment-16242250
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149410922
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java
 ---
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.elasticsearch.utils.ElasticsearchUtils;
+import org.apache.metron.indexing.dao.search.FieldType;
+import 
org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest;
+import org.elasticsearch.client.AdminClient;
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.elasticsearch.common.collect.ImmutableOpenMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static 
org.apache.metron.elasticsearch.utils.ElasticsearchUtils.INDEX_NAME_DELIMITER;
+
+/**
+ * Responsible for retrieving column-level metadata for Elasticsearch 
search indices.
+ */
+public class ElasticsearchColumnMetadataDao implements ColumnMetadataDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private static Map elasticsearchTypeMap;
+  static {
+Map fieldTypeMap = new HashMap<>();
+fieldTypeMap.put("string", FieldType.STRING);
+fieldTypeMap.put("ip", FieldType.IP);
+fieldTypeMap.put("integer", FieldType.INTEGER);
+fieldTypeMap.put("long", FieldType.LONG);
+fieldTypeMap.put("date", FieldType.DATE);
+fieldTypeMap.put("float", FieldType.FLOAT);
+fieldTypeMap.put("double", FieldType.DOUBLE);
+fieldTypeMap.put("boolean", FieldType.BOOLEAN);
+elasticsearchTypeMap = Collections.unmodifiableMap(fieldTypeMap);
+  }
+
+  private transient AdminClient adminClient;
+  private List ignoredIndices;
+
+  public ElasticsearchColumnMetadataDao(AdminClient adminClient, 
List ignoredIndices) {
--- End diff --

Small nit, can we make `ignoredIndices` a `Set`


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242253#comment-16242253
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149409051
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
 ---
@@ -98,7 +98,7 @@
   "mapping": {
 "type": "float"
   },
-  "match": "threat.triage.rules:*:score",
+  "match": "threat:triage:*score",
--- End diff --

great catch.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242251#comment-16242251
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149413486
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -179,4 +186,46 @@ else if(ipObj instanceof List) {
 }
 throw new IllegalStateException("Unable to read the elasticsearch ips, 
expected es.ip to be either a list of strings, a string hostname or a host:port 
string");
   }
+
+  /**
+   * Converts an Elasticsearch SearchRequest to JSON.
+   * @param esRequest The search request.
+   * @return The JSON representation of the SearchRequest.
+   */
+  public static String 
toJSON(org.elasticsearch.action.search.SearchRequest esRequest) {
+String json = "null";
--- End diff --

Should we instead return back an `Optional` rather than the string 
representation of null on a failure here?  Alternatively, maybe throw an 
exception so the calling function can handle the exception how it likes?


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1295) Unable to Configure Logging for REST API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242259#comment-16242259
 ] 

ASF GitHub Bot commented on METRON-1295:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/828
  
Dig. +1 by inspection


> Unable to Configure Logging for REST API
> 
>
> Key: METRON-1295
> URL: https://issues.apache.org/jira/browse/METRON-1295
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> I have not been able to configure logging for the REST API.  To replicate, 
> create a log4j configuration file, then add the following to "Metron JVM 
> Flags" in Ambari > Metron > Config.
> {code}
> -Dlog4j.debug 
> -Dlog4j.configuration=file:/usr/metron/0.4.2/config/log4j.properties
> {code}
> This will result in the following exception when Log4j initializes.
> {code}
> log4j: Using URL [file:/usr/metron/0.4.2/config/log4j.properties] for 
> automatic log4j configuration.
> log4j: Reading configuration from URL 
> file:/usr/metron/0.4.2/config/log4j.properties
> log4j: Parsing for [root] with value=[INFO, file].
> log4j: Level token is [INFO].
> log4j: Category root set to INFO
> log4j: Parsing appender named "file".
> log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable 
> to a "org.apache.hadoop.hbase.shaded.org.apache.log4j.Appender" variable.
> log4j:ERROR The class 
> "org.apache.hadoop.hbase.shaded.org.apache.log4j.Appender" was loaded by
> log4j:ERROR [sun.misc.Launcher$AppClassLoader@5c647e05] whereas object of type
> log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by 
> [sun.misc.Launcher$AppClassLoader@5c647e05].
> log4j:ERROR Could not instantiate appender named "file".
> log4j: Finished configuring.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1294) IP addresses are not formatted correctly in facet and group results

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242265#comment-16242265
 ] 

ASF GitHub Bot commented on METRON-1294:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/827#discussion_r149418025
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SearchControllerIntegrationTest.java
 ---
@@ -236,17 +236,14 @@ public void test() throws Exception {
 
.andExpect(jsonPath("$.groupResults[0].groupResults[0].score").value(50));
 
 this.mockMvc.perform(post(searchUrl + 
"/column/metadata").with(httpBasic(user, 
password)).with(csrf()).contentType(MediaType.parseMediaType("application/json;charset=UTF-8")).content("[\"bro\",\"snort\"]"))
-.andExpect(status().isOk())
-
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
-.andExpect(jsonPath("$.*", hasSize(2)))
-
.andExpect(jsonPath("$.bro.common_string_field").value("string"))
-
.andExpect(jsonPath("$.bro.common_integer_field").value("integer"))
-.andExpect(jsonPath("$.bro.bro_field").value("boolean"))
-.andExpect(jsonPath("$.bro.duplicate_field").value("date"))
-
.andExpect(jsonPath("$.snort.common_string_field").value("string"))
-
.andExpect(jsonPath("$.snort.common_integer_field").value("integer"))
-.andExpect(jsonPath("$.snort.snort_field").value("double"))
-.andExpect(jsonPath("$.snort.duplicate_field").value("long"));
+.andExpect(status().isOk())
--- End diff --

Just some historical color to this discussion.  metron-rest's tests do not 
assume ES, rather the ES Dao tests are in metron-elasticsearch.  The idea was 
that metron-rest should have a runtime dependency on the index being used, 
rather than a compile-time dependency.  The thought was that the testing for 
the controller and such should assume an implementation of the DAO that 
conforms to the interface allowing the ES DAO tests to ensure that it's 
conforming to the expectations of the interface independently.


> IP addresses are not formatted correctly in facet and group results
> ---
>
> Key: METRON-1294
> URL: https://issues.apache.org/jira/browse/METRON-1294
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> Fields that are of type IP address are being returned in numerical format 
> whereas they should be returned in string format.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1275) Fix Metron Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242266#comment-16242266
 ] 

ASF GitHub Bot commented on METRON-1275:


Github user lvets commented on the issue:

https://github.com/apache/metron/pull/833
  
I don't know whether my +1 is actually relevant, but here it goes: +1. 
Looks good to me :)


> Fix Metron Documentation
> 
>
> Key: METRON-1275
> URL: https://issues.apache.org/jira/browse/METRON-1275
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> I am combing through Metron documentation and I will have various updates for 
> minor fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1288) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242267#comment-16242267
 ] 

ASF GitHub Bot commented on METRON-1288:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/822
  
Thank you again for the contribution.  We would like for you to re-submit 
this pr as an individual so that we can maintain the correct author and 
attribution.

Feel free to tag me in the new PR


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1288
> URL: https://issues.apache.org/jira/browse/METRON-1288
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> -brew cask install vagrant virtualbox java docker
> +brew cask install vagrant virtualbox docker
> +brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1291) Kafka produce REST endpoint does not work in a Kerberized cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242280#comment-16242280
 ] 

ASF GitHub Bot commented on METRON-1291:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/826#discussion_r149420627
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
 ---
@@ -108,6 +108,9 @@ public ZkUtils zkUtils() {
 producerConfig.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
 producerConfig.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
 producerConfig.put("request.required.acks", 1);
+if 
(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, 
Boolean.class, false)) {
+  producerConfig.put("security.protocol", "SASL_PLAINTEXT");
--- End diff --

Shouldn't we pull this from the `KAFKA_SECURITY_PROTOCOL` environment 
variable?  Looks like it's being set in 
`metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2`


> Kafka produce REST endpoint does not work in a Kerberized cluster
> -
>
> Key: METRON-1291
> URL: https://issues.apache.org/jira/browse/METRON-1291
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1275) Fix Metron Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242277#comment-16242277
 ] 

ASF GitHub Bot commented on METRON-1275:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/833#discussion_r149420478
  
--- Diff: metron-platform/Performance-tuning-guide.md ---
@@ -219,38 +219,49 @@ And we ran our bro parser topology with the following 
options. We did not need t
 though you could certainly do so if necessary. Notice that we only needed 
1 worker.
 
 ```
-/usr/metron/0.4.0/bin/start_parser_topology.sh -k $BROKERLIST -z 
$ZOOKEEPER -s bro -ksp SASL_PLAINTEXT
--ot enrichments
+/usr/metron/0.4.2/bin/start_parser_topology.sh \
 -e ~metron/.storm/storm-bro.config \
 -esc ~/.storm/spout-bro.config \
--sp 24 \
--snt 24 \
+-k $BROKERLIST \
+-ksp SASL_PLAINTEXT \
--- End diff --

Oops, sorry that's already how the docs were.  I wasn't following what you 
were doing here.  All you're doing is a reformat.  LGTM.


> Fix Metron Documentation
> 
>
> Key: METRON-1275
> URL: https://issues.apache.org/jira/browse/METRON-1275
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> I am combing through Metron documentation and I will have various updates for 
> minor fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242300#comment-16242300
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r149422318
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
 ---
@@ -29,6 +29,7 @@
   public static final String ZOOKEEPER_TOPOLOGY_ROOT = ZOOKEEPER_ROOT + 
"/topology";
   public static final long DEFAULT_CONFIGURED_BOLT_TIMEOUT = 5000;
   public static final String SENSOR_TYPE = "source.type";
+  public static final String SOURCE_TYPE = SENSOR_TYPE.replace('.', ':');
--- End diff --

This doesn't feel like it should exist in Constants.java as it's very ES 
specific (solr would not work this way, for instance).  Maybe we could have an 
ElasticsearchConstants or just move this to a constant in the ESDao?


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242299#comment-16242299
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r149423305
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java
 ---
@@ -65,6 +65,8 @@
*/
   Document getLatest(String guid, String sensorType) throws IOException;
 
+  Iterable getAllLatest(Map guidToIndices) 
throws IOException;
--- End diff --

Can we add a javadoc similar to getLatest?


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242302#comment-16242302
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r149423194
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/HBaseDao.java
 ---
@@ -121,13 +121,16 @@ public synchronized Document getLatest(String guid, 
String sensorType) throws IO
   }
 
   @Override
+  public Iterable getAllLatest(Map 
guidToIndices) throws IOException {
+return null;
--- End diff --

Why is there no implementation here?  HBase certainly should be able to 
have a multiget for the documents, no?


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242301#comment-16242301
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r149423571
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java
 ---
@@ -65,6 +65,8 @@
*/
   Document getLatest(String guid, String sensorType) throws IOException;
 
+  Iterable getAllLatest(Map guidToIndices) 
throws IOException;
--- End diff --

Actually, even a default implementation here that calls getLatest might be 
useful.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242648#comment-16242648
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149471327
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java
 ---
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.elasticsearch.utils.ElasticsearchUtils;
+import org.apache.metron.indexing.dao.search.FieldType;
+import 
org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest;
+import org.elasticsearch.client.AdminClient;
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.elasticsearch.common.collect.ImmutableOpenMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static 
org.apache.metron.elasticsearch.utils.ElasticsearchUtils.INDEX_NAME_DELIMITER;
+
+/**
+ * Responsible for retrieving column-level metadata for Elasticsearch 
search indices.
+ */
+public class ElasticsearchColumnMetadataDao implements ColumnMetadataDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private static Map elasticsearchTypeMap;
+  static {
+Map fieldTypeMap = new HashMap<>();
+fieldTypeMap.put("string", FieldType.STRING);
+fieldTypeMap.put("ip", FieldType.IP);
+fieldTypeMap.put("integer", FieldType.INTEGER);
+fieldTypeMap.put("long", FieldType.LONG);
+fieldTypeMap.put("date", FieldType.DATE);
+fieldTypeMap.put("float", FieldType.FLOAT);
+fieldTypeMap.put("double", FieldType.DOUBLE);
+fieldTypeMap.put("boolean", FieldType.BOOLEAN);
+elasticsearchTypeMap = Collections.unmodifiableMap(fieldTypeMap);
+  }
+
+  private transient AdminClient adminClient;
+  private List ignoredIndices;
+
+  public ElasticsearchColumnMetadataDao(AdminClient adminClient, 
List ignoredIndices) {
--- End diff --

Done.  See latest.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242646#comment-16242646
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149471276
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -179,4 +186,46 @@ else if(ipObj instanceof List) {
 }
 throw new IllegalStateException("Unable to read the elasticsearch ips, 
expected es.ip to be either a list of strings, a string hostname or a host:port 
string");
   }
+
+  /**
+   * Converts an Elasticsearch SearchRequest to JSON.
+   * @param esRequest The search request.
+   * @return The JSON representation of the SearchRequest.
+   */
+  public static String 
toJSON(org.elasticsearch.action.search.SearchRequest esRequest) {
+String json = "null";
--- End diff --

Yes, definitely this is kind of weird.  This is taken care of in the latest 
commit.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242021#comment-16242021
 ] 

ASF GitHub Bot commented on METRON-1304:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/2
  
Yeah, I did send an initial email out [last 
night](https://lists.apache.org/thread.html/621cbeca432fef0170836e07036e309d943068c5d6a544c1ef2136f9@%3Cdev.metron.apache.org%3E),
 and in a previous email in that thread I outlined a suggested roadmap, but I 
will bring it up and clarify things so we can discuss on the list instead of 
here.  Thanks


> Allow metron-bro-plugin-kafka to include or exclude logs
> 
>
> Key: METRON-1304
> URL: https://issues.apache.org/jira/browse/METRON-1304
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Right now, you must specify which logs you want to send to kafka via 
> metron-bro-plugin-kafka.  This would allow the additional feature of 
> excluding certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242026#comment-16242026
 ] 

ASF GitHub Bot commented on METRON-1304:


Github user nickwallen commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/2
  
My bad, you did.  You're totally on top of it.  Ignore me.  :)


> Allow metron-bro-plugin-kafka to include or exclude logs
> 
>
> Key: METRON-1304
> URL: https://issues.apache.org/jira/browse/METRON-1304
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Right now, you must specify which logs you want to send to kafka via 
> metron-bro-plugin-kafka.  This would allow the additional feature of 
> excluding certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242124#comment-16242124
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/829
  
I'm +1 on the PR, but I strongly agree with Anand, this should not be a 
warn.  We should throw an exception here and fail the build.  That being said, 
despite it being in the vicinity of this PR, it's not the core value of the PR, 
so I've created 
[METRON-1306](https://issues.apache.org/jira/browse/METRON-1306) to track it.


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242165#comment-16242165
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/834
  
This is good. +1 by inspection.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1296) Full Dev Fails to Deploy Index Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242159#comment-16242159
 ] 

ASF GitHub Bot commented on METRON-1296:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/829
  
@anandsubbu agreed with your comments on the exception


> Full Dev Fails to Deploy Index Templates
> 
>
> Key: METRON-1296
> URL: https://issues.apache.org/jira/browse/METRON-1296
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.2
>
>
> There are cases where Full Dev is deployed, but the index templates are not 
> deployed.  Without the index templates, the Alerts UI and Kibana dashboard do 
> not work as expected.  This seems to occur sometimes and not others.
> The index template install happens during startup of the Index topology.  
> Elasticsearch master start occurred well before the attempt to install the 
> templates.  The following message is shown in the logs.
> {code}
> WARNING: Elasticsearch templates could not be installed. The Elasticsearch 
> service is probably down.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242194#comment-16242194
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/834
  
I clearly see the advantages of failing fast here, but I do have a 
hesitation.   I haven't even totally convinced myself of this.   Maybe you guys 
can help me think down this line of thought and make sure it is a non-issue.

This seems like we are introducing a hard dependency between Indexing and 
Elasticsearch.  If Elasticsearch is not running, then I cannot start the 
Indexing topology.  What if I am not using ES, but just want to index into HDFS 
(or Solr, ha)?  Or what if I am need to customize my templates?  Does this 
introduce any pain points that we are not anticipating?



> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242200#comment-16242200
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/834
  
And for the record @anandsubbu did implement this as "fail fast" in his 
original PR.  So I think he is all for this. I was the one who talked him out 
of failing fast here.  I was wary of introducing a hard dependency between 
Indexing and ES and didn't think that the "Install Index Templates" step would 
fail often enough for this to be a problem.  


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242202#comment-16242202
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/834
  
Ok, so a couple of things here.  I wanted to have a vigorous discussion, 
because this change has some pros and cons.

At the moment, as Nick alluded, we do template install upon index topology 
start.  The problem is that we cannot depend on ES starting prior to the 
indexing topology due to us not having a hard dependency listed there in the 
[role_command_order.json](https://github.com/cestella/incubator-metron/blob/e83390a39910903ccba313a7a1b00433bf347058/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json).
  The reason why, I believe, is because we don't assume that users are using ES 
managed via ambari necessarily.

All this to say that there may be a failure to start the indexing topology 
upon startup, which would necessitate a restart of the indexing topology to 
retry template install.  The downside to what we are doing currently, though, 
is that a warning is hidden in logs that users probably wont' see.  They will 
see green lights and the UI may or may not work (probably not).  This will get 
worse in ES 5, because without those templates, tuples will fail.

So, my question to you guys, is there sufficient value in failing fast here 
given the broader context.  I think so, but I want everyone to realize that we 
may end up in a situation where indexing fails to start becuase of an ordering 
problem with ES.

I will not push this through without discussion, so I'd like at least 2 
+1's from committers and a week of discussion before a commit happens.  
Thoughts?


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1304) Allow metron-bro-plugin-kafka to include or exclude logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242017#comment-16242017
 ] 

ASF GitHub Bot commented on METRON-1304:


Github user nickwallen commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/2
  
@JonZeolla Can you send an update to the Dev list on where we are at in 
migrating the Bro plugin?  I had not even realized that you had added the code 
to this repo.  

We also still have the Bro plugin code in the main Metron repository.  
Don't we have some clean-up to do before we start merging new changes?


> Allow metron-bro-plugin-kafka to include or exclude logs
> 
>
> Key: METRON-1304
> URL: https://issues.apache.org/jira/browse/METRON-1304
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Right now, you must specify which logs you want to send to kafka via 
> metron-bro-plugin-kafka.  This would allow the additional feature of 
> excluding certain logs, and sending everything else.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1303) Reorganize the metron-bro-plugin-kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242015#comment-16242015
 ] 

ASF GitHub Bot commented on METRON-1303:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/1
  
I'm willing to bet we could get this tested using travis.  There is a 
built-in btest which will be automatically run on install when this is turned 
into a package (although it is *very* rudamentary), and I'm planning to expand 
the btest to include a memtest, but it is missing a lot of tests at this point 
and that probably needs to be a future focus.  I threw together 
[METRON-1305](https://issues.apache.org/jira/browse/METRON-1305) to track.


> Reorganize the metron-bro-plugin-kafka
> --
>
> Key: METRON-1303
> URL: https://issues.apache.org/jira/browse/METRON-1303
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, 
> which is somewhat confusing based on the way bro packages are currently 
> named/sorted[1].  Based on this, we should align the plugin namespace to run 
> under the owner of the repo, which is apache.
> 1:  https://github.com/bro/packages



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1306) When index template install fails, we should fail the install

[Casey Stella (JIRA)]

Casey Stella created METRON-1306:


 Summary: When index template install fails, we should fail the 
install
 Key: METRON-1306
 URL: https://issues.apache.org/jira/browse/METRON-1306
 Project: Metron
  Issue Type: Bug
Reporter: Casey Stella


Right now we warn when an index template install fails.  In my opinion, this 
should be an install failure, not a warning.  It's disorienting for the user to 
have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242138#comment-16242138
 ] 

ASF GitHub Bot commented on METRON-1306:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/834

METRON-1306: When index template install fails, we should fail the install

## Contributor Comments
Right now we warn when an index template install fails. In my opinion, this 
should be an install failure, not a warning. It's disorienting for the user to 
have a green install and have the UIs fail to work due to template issues.  I 
think we should warn in the logs, but also fail fast.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-1306

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/834.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #834


commit e83390a39910903ccba313a7a1b00433bf347058
Author: cstella 
Date:   2017-11-07T15:00:38Z

Failing fast.




> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242147#comment-16242147
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/834
  
Ah, yes, sorry, I should be more clear here.  Starting the indexing 
topology should fail if the indexing templates are not installed.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242476#comment-16242476
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/834
  
This is why I put the templates with the parsers in 777.  Although I did 
not have the templates installing from there because list discussion was tough, 
mainly because talking about this in the abstract is hard.  The idea was that 
in the end we would install with the parsers.  Possibly after the ambari flow.

- we should be able to install metron without installing and configuring 
any 'demo' system or parsers at all
- we should be able to manage and install templates 
- templates are part of any parser configuration and should be managed and 
deployed with the parsers ( ie. 'create an deploy an es template is on the list 
of what you need to do for a production parser so it goes with the parser 
workflow')
- we shouldn't lay down default configurations for the 'demo' parsers
- the demo system should be managed outside main code ( ie. the bro.json 
files configured with the geo enrichments etc should be separate than what is 
installed by default)
- the demo system should be a separate installable option
- all the ES/KIBANA/SOLR issues with mpack etc





> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242315#comment-16242315
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149426981
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

Yes, I agree with your feedback.  I was just trying to refactor the column 
metadata logic and minimize other changes.  So in this case `.kibana` was 
already hard-coded in ElasticsearchDao.

I would be totally open to making this improvement though.  I was just 
trying to walk the line of how much should I change when refactoring?  
Considering that, what do you think?




> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242332#comment-16242332
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149428796
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

I'm almost always in favor of having hard coded things passed in via config 
files.  It gets us out of jams and almost every time I've convinced myself that 
it's not necessary, it totally is.  How convinced are you that we are only ever 
going to need one ignored index?

If you have doubts, then I'd probably add something to the 
`application.yml` which gets set in the AccessConfig object when we set up the 
index in the IndexConfig from `metron-rest`.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242334#comment-16242334
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149429061
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

Given that it's a pretty simple change and requires no new machinery to 
added, I think it is probably a good fit to add to this PR, but I might be on 
the wrong side of that line more than I'm on the right side, so take my word 
with a grain of salt. ;)


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1275) Fix Metron Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242345#comment-16242345
 ] 

ASF GitHub Bot commented on METRON-1275:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/833
  
Piling on with a +1, great job @JonZeolla 


> Fix Metron Documentation
> 
>
> Key: METRON-1275
> URL: https://issues.apache.org/jira/browse/METRON-1275
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> I am combing through Metron documentation and I will have various updates for 
> minor fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242322#comment-16242322
 ] 

ASF GitHub Bot commented on METRON-1301:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/832#discussion_r149428276
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -234,26 +366,43 @@ public synchronized void init(AccessConfig config) {
 if(this.client == null) {
   this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
   this.accessConfig = config;
+  this.columnMetadataDao = new 
ElasticsearchColumnMetadataDao(this.client.admin(), 
Collections.singletonList(".kibana"));
--- End diff --

Ok, man I have been really bad about reading and comprehension today.  I 
see in your review comment feedback that you already addressed this point. 
Sorry.

I think having this hardcoded is just not good.  The change seems simple 
enough.  I can probably take care of this no problem.


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> -
>
> Key: METRON-1301
> URL: https://issues.apache.org/jira/browse/METRON-1301
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
> Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any 
> records that do not have a threat triage score defined from the search 
> results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a 
> timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 
> records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242340#comment-16242340
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/834
  
I ran this up in full-dev and index start functions fine and data is 
written to the indices.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242405#comment-16242405
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/834
  
@nickwallen and @cestella you both make some good points here. The upshot 
is that this part of our architecture is inextricably tied to ES and in reality 
we should have another abstraction here. It appears to me that we cannot add 
new templates via the current Ambari mechanism without modifying a 
release/MPack. It's fine for managed sensors (bro, yaf, snort, other potential 
future default sensors), but if a user wants to add new templates, you'd have 
to manage them outside of Ambari. It's clear to me now that the back and forth 
on the right way to manage the error handling is because we have a code smell 
that needs some additional work.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1306) When index template install fails, we should fail the install

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242703#comment-16242703
 ] 

ASF GitHub Bot commented on METRON-1306:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/834
  
@ottobackwards I need to think through the use cases and implications more 
(e.g. what about a non-ES situation?), but I think your points are valid.


> When index template install fails, we should fail the install
> -
>
> Key: METRON-1306
> URL: https://issues.apache.org/jira/browse/METRON-1306
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>
> Right now we warn when an index template install fails.  In my opinion, this 
> should be an install failure, not a warning.  It's disorienting for the user 
> to have a green install and have the UIs fail to work due to template issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242738#comment-16242738
 ] 

ASF GitHub Bot commented on METRON-1307:


Github user CUGCR commented on the issue:

https://github.com/apache/metron/pull/835
  
@ottobackwards back to you!


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[Brian Hurley (JIRA)]

Brian Hurley created METRON-1307:


 Summary: Force install of java8 since java9 does not appear to 
work with the scripts
 Key: METRON-1307
 URL: https://issues.apache.org/jira/browse/METRON-1307
 Project: Metron
  Issue Type: Bug
Reporter: Brian Hurley
Priority: Blocker


Force install of java8 since java9 does not appear to work with the scripts



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1307) Force install of java8 since java9 does not appear to work with the scripts

[Brian Hurley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Hurley updated METRON-1307:
-
Description: 
Install script for dev instance does not work with Java9 (released in Sept 2017 
and pulled in by default), the change will force install of Java8 which it does 
work with.

- brew cask install vagrant virtualbox java docker
+ brew cask install vagrant virtualbox docker
+ brew cask install caskroom/versions/java8

  was:Force install of java8 since java9 does not appear to work with the 
scripts


> Force install of java8 since java9 does not appear to work with the scripts
> ---
>
> Key: METRON-1307
> URL: https://issues.apache.org/jira/browse/METRON-1307
> Project: Metron
>  Issue Type: Bug
>Reporter: Brian Hurley
>Priority: Blocker
>
> Install script for dev instance does not work with Java9 (released in Sept 
> 2017 and pulled in by default), the change will force install of Java8 which 
> it does work with.
> - brew cask install vagrant virtualbox java docker
> + brew cask install vagrant virtualbox docker
> + brew cask install caskroom/versions/java8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)