from:"merrimanr"

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/957


---

[GitHub] metron pull request #945: METRON-1464: Convert schemas to be compatible with...

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/945


---

[GitHub] metron issue #949: METRON-1471: Migrate shuffle connections to local or shuf...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/949
  
+1 thanks!


---

[GitHub] metron issue #957: METRON-1482: Update REST to work with Solr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/957
  
I've been maintaining this branch for a while and have been merging in 
other branches before they were committed to master.  I think that's why you 
see all these commits.  I merged in feature/METRON-1416-upgrade-solr right 
before I submitted this PR.  It's the commit right before "fixed metron 
version".


---

[GitHub] metron pull request #957: METRON-1482: Update REST to work with Solr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/957

METRON-1482: Update REST to work with Solr

## Contributor Comments
This PR updates the REST start script to properly include the correct 
indexing jar on the classpath depending on the configured indexing writer 
(Elasticsearch or Solr).  To test this:

1. Spin up full dev and follow the instructions for manually installing 
Solr:  
https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr#installing.
 
2. Change the Index Writer to "Solr" in Ambari (Metron > Indexing tab) and 
restart the Indexing component.
3. Verify data is landing in the Solr collections
4. Restart the REST component in Ambari
5. You should be able to execute search queries in Swagger

I had to include a stubbed SolrMetaAlertDao class that doesn't do anything 
to get everything to start up.  This will eventually be replaced by the work 
being done in https://issues.apache.org/jira/browse/METRON-1421.  I also fixed 
a bug in the SolrDao that was causing in issue in the Alerts UI.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/merrimanr/incubator-metron solr-rest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/957.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #957


commit 6bb30af9d2005414e3ee44c0bdb0ea14540ce13c
Author: cstella <cestella@...>
Date:   2018-02-01T21:33:56Z

METRON-1441: Create complementary Solr schemas for the main sensors

commit f4ff0c401eff23d9c1b2ca3b264bd9b0d4e8f381
Author: cstella <cestella@...>
Date:   2018-02-01T21:47:12Z

Updating dao

commit 7e2ecb0f2f55ea16529128fec14920bc2a546b07
Author: cstella <cestella@...>
Date:   2018-02-02T21:43:38Z

Migrated data to files, renamed test and added yaf and error.

commit 2aacd202ff1a2ebcbeb30300b30d080391cfe1cf
Author: cstella <cestella@...>
Date:   2018-02-02T21:45:08Z

Merge branch 'feature/

[GitHub] metron issue #949: METRON-1471: Migrate shuffle connections to local or shuf...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/949
  
I still see a SHUFFLE grouping (spout -> enrichmentSplit) in the legacy 
enrichment topology flux file.  Maybe we end up switching to the new enrichment 
topology so not important in that case.


---

[GitHub] metron issue #949: METRON-1471: Migrate shuffle connections to local or shuf...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/949
  
There are also shuffle groupings in the enrichment and profiler topologies. 
 Do we want to update those too?


---

[GitHub] metron issue #952: METRON-1480 Add yarn as default build tool for the fronte...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/952
  
I think we are on board with switching to yarn so no concerns there.  There 
was a discussion on it but if anyone now feels differently, speak up.

Sounds like we still have a lot of work to do.  I would suggest reviewing 
the metron-config and metron-alerts READMEs and update them appropriately.  In 
fact, I would just search the whole project for npm commands and switch to yarn 
as needed.  You will definitely need to run this up in full dev and verify 
everything works.

Our maven front end plugin will also need to be reviewed and updated:  
https://github.com/eirslett/frontend-maven-plugin.  We use that in 
metron-config and metron-alerts.


---

[GitHub] metron pull request #952: METRON-1480 Add yarn as default build tool for the...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172937150
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

No I was just curious.  If Yarn did this automatically we might as well 
leave it that way.  No need to revert.


---

[GitHub] metron pull request #952: METRON-1480 Add yarn as default build tool for the...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172934046
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

What was the reason for rearranging the dependencies?  It's not a big deal 
but it's distracting and makes reviewing harder.


---

[GitHub] metron issue #952: METRON-1480 Add yarn as default build tool for the fronte...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/952
  
I'm having trouble understanding what this PR actually does.  I see that 
package-lock.json was removed and yarn.lock was added.  Is that all that's 
needed to switch to yarn?  Do we need to also update our build commands to 
match those [here](https://yarnpkg.com/en/docs/usage)?

What considerations are there for development?  Are we supposed to be using 
yarn commands now instead of npm commands?  Do we need to install yarn locally? 
 We probably need to review our documentation and rpm build processes too.  We 
use npm all over the place.


---

[GitHub] metron pull request #952: METRON-1480 Add yarn as default build tool for the...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172927751
  
--- Diff: metron-interface/metron-alerts/package.json ---
@@ -27,8 +27,8 @@
 "core-js": "^2.4.1",
 "font-awesome": "^4.7.0",
 "moment": "^2.18.1",
-"pikaday-time": "^1.6.1",
 "ng2-dragula": "^1.5.0",
+"pikaday-time": "^1.6.1",
--- End diff --

See comment for metron-config/package.json.


---

[GitHub] metron pull request #952: METRON-1480 Add yarn as default build tool for the...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172927486
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

Were there any real changes made to this file?  It looks like some 
dependencies were simply rearranged.  I do see the addition of `"main": 
"index.js"`.  What is the purpose of that?


---

[GitHub] metron issue #950: METRON-1470: Update jquery to version 3+

2018-03-06 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/950
  
I ran this up in full dev and tested several different features.  I could 
not find any regressions.  Thanks @xyztdanid4! +1 


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

2018-03-06 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/943
  
I spun this up in full dev and everything worked as expected.  The 
organization looks good to me and I can't find anything wrong with it.  +1


---

[GitHub] metron issue #941: METRON-1355: Convert metron-elasticsearch to new infrastr...

2018-03-06 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/941
  
Sorry I should have documented this better in the PR description.  The 
docker-machine ip address needs to be substituted in ElasticsearchTestUtils 
temporarily if you want to run this on your local Mac OS.  There is a Jira for 
this work here:  https://issues.apache.org/jira/browse/METRON-1356.  Everything 
you've described including the TODO is intended to be handled there.  Do you 
think that needs to come first before you review this PR?  


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-03-01 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Any other feedback @ottobackwards or is this ready to go?


---

[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...

2018-03-01 Thread merrimanr

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/934


---

[GitHub] metron pull request #945: METRON-1464: Convert schemas to be compatible with...

2018-02-28 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/945

METRON-1464: Convert schemas to be compatible with Solr 5.5.2

## Contributor Comments
This PR makes the Solr schema compatible with Solr 5.5.2.  The client code 
worked as is (compiled with version 6.6.2), no changes necessary.

To test this in full dev:

1. Spin up full dev
2. Install HDP Search ("Chapter 3:  Installing HDP Search Using Ambari" in 
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_solr-search-installation/bk_solr-search-installation.pdf)
3. Create the bro and snort collections manually:
```
/opt/lucidworks-hdpsearch/solr/bin/solr create -c bro -d 
/usr/metron/0.4.3/config/schema/bro
/opt/lucidworks-hdpsearch/solr/bin/solr create -c snort -d 
/usr/metron/0.4.3/config/schema/snort
```
4. Create a file named `metron_solr_template_installed_flag_file` in 
$METRON_HOME/config
5. Navigate to the Metron service in Ambari and change the "Solr Zookeeper 
Urls" setting in the Index Settings tab to `node1:2181/solr`
6. Change the "Random Access Search Engine" setting in the Indexing tab to 
`Solr`
7. Restart the Indexing component in Ambari

Navigate to node1:8983 and inspect the bro and snort collections.  They 
should contain data.  There will likely be additional Ambari work should we 
decide to move to HDP Search.  This PR still assumes Solr 6.6.2 will be 
installed manually.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1464

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/945.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #945


commit ac1de170af8fa769683f72783f6258cf12663f94
Author: merrimanr <merrimanr@...>
Date:   2018-02-28T22:49:37Z

initial commit




---

[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-02-26 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/940
  
I tested this in full dev and worked as expected.  +1


---

[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...

2018-02-23 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/934#discussion_r170365518
  
--- Diff: metron-platform/metron-solr/src/main/scripts/create_collection.sh 
---
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+SOLR_VERSION=6.6.2
--- End diff --

Latest commit should address this.


---

[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...

2018-02-23 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/934#discussion_r170272344
  
--- Diff: metron-platform/metron-solr/src/main/scripts/create_collection.sh 
---
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+SOLR_VERSION=6.6.2
--- End diff --

Good catch and I agree they should match.  I think the mistake is actually 
the maven version.  We initially explored using HDP Search (which is version 
6.6.2) but it's not quite ready yet so we went ahead with a manual approach in 
the meantime.  The thinking was this would make it easy to switch to HDP Search 
in the future.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r170120909
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/AlertServiceImpl.java
 ---
@@ -37,15 +47,21 @@
 @Service
 public class AlertServiceImpl implements AlertService {
 
--- End diff --

Name is changed in the latest commit.  Let me know what you think.


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/858


---

[GitHub] metron pull request #941: METRON-1355: Convert metron-elasticsearch to new i...

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/941

METRON-1355: Convert metron-elasticsearch to new infrastructure

## Contributor Comments
This PR switches metron-elasticsearch integration tests from using 
in-memory components to the e2e Docker infrastructure.  A high-level summary of 
the changes:

- Updated travis to only run the metron-elasticsearch tests
- Updated the Elasticsearch Docker image to the same version as Metron
- Removed the requirement to build a base "metron-centos" image.  Removes 
an extra Docker build step and I'm not sure caching this helps much anyways.
- Updated each integration test to use index names that are namespaced with 
the test class name
- Replaced the in-memory component setup with an Elasticsearch client 
configured for the Elasticsearch Docker container
- Added steps to setup/delete indices before/after each test
- Moved Elasticsearch in-memory helper methods and common integration tests 
methods to a utils class
- Fixed a minor bug in the ElasticsearchMetaAlertDao class where the index 
is always hardcoded to "metaalert"
- Added some initial documentation for the metron-docker-e2e module

The scope of this PR are 3 of the 4 metron-elasticsearch integration tests:

- 
org.apache.metron.elasticsearch.integration.ElasticsearchMetaAlertIntegrationTest
- 
org.apache.metron.elasticsearch.integration.ElasticsearchSearchIntegrationTest
- 
org.apache.metron.elasticsearch.integration.ElasticsearchUpdateIntegrationTest

Most of the test logic in 
org.apache.metron.elasticsearch.integration.ElasticsearchIndexingIntegrationTest
 is actually in metron-indexing so that test will be updated when we convert 
that module.

At this point only the metron-elasticsearch tests are run in travis.  My 
plan is to slowly add tests for each module until we reach feature parity with 
master.  At that point the "install" section of .travis.yml will be the same.  
The alerts ui e2e tests were removed for now until the work being done to 
stabilize them is complete.

The ES in-memory component starts up pretty fast so performance improved 
for the 3 tests by about 7 seconds.  Curious to hear what people think.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set

[GitHub] metron issue #924: METRON-1299 In MetronError tests, don't test for HostName...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/924
  
+1 pending @cestella's approval.  Thanks @ottobackwards.


---

[GitHub] metron issue #895: METRON-1394:Create Rest endpoint to add the ACL for curre...

2018-02-21 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/895
  
+1 by inspection.  Thanks @MohanDV!


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

2018-02-20 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r169455383
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/AlertServiceImpl.java
 ---
@@ -37,15 +47,21 @@
 @Service
 public class AlertServiceImpl implements AlertService {
 
--- End diff --

Sure I can do that.  Since these settings are specific to the Alerts UI, 
should we include "ui" too?  Should the REST service also follow a similar 
pattern?


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-02-20 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
I took @ottobackwards's suggestion and added a "type" parameter that can be 
used to independently manage user settings for different types of clients.  I 
moved the users settings for the Alerts UI (the original intention of this PR) 
back to the AlertController.  I also went ahead and moved the low level client 
code to the metron-hbase module so that this user settings abstraction can be 
used anywhere in Metron.  For examples of how to setup and use the client see 
HBaseConfig and AlertServiceImpl in the metron-rest module.

Testing in full dev is slightly different now.  Instead of the endpoints 
being in the UserController they should now be tested in the AlertController.  
The Alerts UI should also be tested.  Navigate to the Alerts UI and you should 
see the default facets on the left.  Update the "facetFields" user setting with 
Swagger and the new list should appear in the Alerts UI after refreshing.  For 
example, if you set facetFields = ip_src_addr, only that facet field should 
appear in the Alerts UI.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168817020
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HBaseConfig.java
 ---
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.hbase.HTableProvider;
+import org.apache.metron.rest.MetronRestConstants;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.GlobalConfigService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Profile;
+import org.springframework.core.env.Environment;
+
+import java.io.IOException;
+
+import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
+import static 
org.apache.metron.rest.repository.UserSettingsRepository.USER_SETTINGS_HBASE_TABLE;
+
+@Configuration
+@Profile("!" + TEST_PROFILE)
--- End diff --

We don't want to use this Configuration if tests are running.  There is a 
separate TestConfig that setups up mock, inmemory components, etc.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168816712
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchRequest.java
 ---
@@ -101,16 +99,16 @@ public void setSort(List sort) {
 this.sort = sort;
   }
 
-  public Optional<List> getFields() {
-return fields == null || fields.size() == 0 ? Optional.empty() : 
Optional.of(fields);
+  public List getFields() {
+return fields;
   }
 
   public void setFields(List fields) {
 this.fields = fields;
   }
 
-  public Optional<List> getFacetFields() {
-return facetFields == null || facetFields.size() == 0 ? 
Optional.empty() : Optional.of(facetFields);
+  public List getFacetFields() {
+return facetFields;
--- End diff --

Because null and an empty array mean 2 different things.  See point 6 here: 
 https://github.com/apache/metron/pull/853#issuecomment-356350839.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168816212
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
 ---
@@ -179,6 +203,35 @@ def status_rest_application(self, env):
 self.__params.metron_rest_port,
 self.__params.metron_user)
 
+def create_hbase_tables(self):
+Logger.info("Creating HBase Tables")
+metron_service.create_hbase_table(self.__params,
+  
self.__params.user_settings_hbase_table,
+  
self.__params.user_settings_hbase_cf)
+Logger.info("Done creating HBase Tables")
+self.set_hbase_configured()
+
+def set_hbase_acls(self):
+Logger.info("Setting HBase ACLs")
+if self.__params.security_enabled:
+kinit(self.__params.kinit_path_local,
+  self.__params.hbase_keytab_path,
+  self.__params.hbase_principal_name,
+  execute_user=self.__params.hbase_user)
+
+cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
--- End diff --

Good catch.


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
I think it's a fair question @ottobackwards.  Anything that might affect 
how the HBase table is laid out should be worked out now or we're back to 
altering tables during upgrades.  Currently the row key is the user name, the 
column family is hardcoded (although configurable), and the column qualifier is 
hardcoded.  Would you add the type to the row key or store different client 
settings in different columns?

We could add versioning but that is not trivial and will increase the scope 
of this PR.  I can take it on here if we decide we want it but this PR is 
already large.  Follow on maybe?  Either way works for me.  If we decide a 
follow on is better I would make an effort to do it right away before we do a 
release.

Jackson is the standard in Metron for serialization/deserialization.  I 
think we should have a very compelling reason to introduce something new.  
Happy to discuss.




---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
The latest commit switches the persistence for storing user settings to 
HBase rather than a RDBMS as discussed on the dev list.  Instead of fields 
being stored in RDBMS columns, the user settings object is now serialized with 
Jackson and stored in HBase as a byte[].  This required several changes 
including:

- Added code to setup HBase in the MPack, following the update table 
pattern in the indexing MPack scripts as an example
- Added default table/cf values to REST properties
- Added a Config class to the REST app to setup an HBase client
- Added a Service class to the REST app that manages user settings in HBase
- Added tests for all new classes and updated existing tests
- Updated the REST README

I also refactored some areas to make things clearer and easier to 
understand:

- Renamed AlertProfile to UserSettings and moved the endpoints to the 
UserController.  The thinking is that we may not want this to be limited to 
just the Alerts UI.
- Refactored the REST MPack scripts to more closely align with other 
components in regards to setting up Kafka, HBase, etc.
- Removed the Optional fields from a couple model classes.  This was 
causing issues with Jackson and I don't believe it benefits us enough to have 
to deal with that complexity.

There are a couple of design issues to consider when reviewing this.  The 
implementation in the REST app is specific to this use case.  I considered 
trying to make it more generic for future HBase use cases but decided to keep 
it simple for now.   Instead of trying to predict what those use cases look 
like and choose a pattern that works, I decided to leave that to whoever 
implements a new use case in the future.  I also considered making this generic 
to all of Metron but again, decided to keep it simple.  Should these settings 
be limited to just REST/UIs?  Any thoughts on this?

This has been tested in full dev and all tests are passing.  In full dev, 
navigate to the UserController and use the various endpoints to save and 
retrieve settings for the currently logged in user.  You should also be able to 
log in as the admin user and see all user settings and delete an individual 
user's settings.  I also tested evolving the user settings model by adding new 
fields and it worked without issue.  Users with existing settings just return 
null for new fields. 


---

[GitHub] metron issue #937: METRON-1455: Patch and Replace methods in the REST Update...

2018-02-15 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/937
  
I verified this works in full dev.  Thanks for fixing this!  +1


---

[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration

2018-02-12 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/934
  
I agree with @justinleet.  I would prefer we use Solr 6+.  At some point 
HDP Search will move to 6+ and we can easily switch to the Mpack since 
installing Solr is a manual step now.  Happy to document that once we are all 
in agreement.

As for the parameters we expose in Ambari, I'm referring to the parameters 
stored in Zookeeper that the SolrWriter reads.  They are:

- solr.zookeeper
- solr.commitPerBatch
- solr.commit.waitSearcher
- solr.commit.waitFlush
- solr.commit.soft
- solr.collection
- solr.http.config

Currently Ambari only exposes the "solr.zookeeper" parameter which is 
necessary to get the indexing topology writing to Solr.  I think a reasonable 
solution would be to include all the others except "solr.http.config" since 
it's an advanced config.  Storm tuning parameters are independent of the writer 
implementation and already exposed in Ambari.  

For the collection service actions, it would mirror what we expose for ES:  
creating and deleting templates (schemas in Solr terms).


---

[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...

2018-02-09 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/934

METRON-1423: Ambari work to handle Solr configuration

## Contributor Comments
This PR allows support for Metron-related Solr configuration and management
through Ambari. That does NOT include managing the actual Solr service. For
now this is done outside of Ambari.

High level changes include:

- Solr is now installed automatically in full dev, although not in a
running state. This implementation was inspired by the `install_solr.sh`
script.
- Additional scripts for managing the Solr installation provided in full
dev and installed with `install_solr.sh`.
- Exposes a single Zookeeper Solr configuration (for now) parameter through
Ambari.
- Replaced the random access index writer class Ambari parameter with a
random access search engine parameter that is a drop down of "Elasticsearch"
and "Solr".
- Replaced "Elasticsearch" with "Random Access" in all the Ambari parameter
descriptions.
- Added the Solr schema creation to Ambari on Indexing start. This matches
the ES template loading implementation with the exception of exposing it
through Service actions.

Testing instructions are as follows:
1. Spin up full dev
2. Shutdown ES and Kibana
3. Start Solr from the command line with $METRON_HOME/bin/start_solr.sh
4. Navigate to the Indexing tab in Ambari > Metron > Configs and change the
"Random Access Search Engine" setting from "Elasticsearch" to "Solr"
5. Now you can start ingesting data into Solr in 2 ways. After you make
the change in Ambari it will prompt you to restart affected services. A
restart does not automatically create collections so you will need to do this
outside of Ambari for bro, snort and error with
$METRON_HOME/bin/create_collection.sh (first argument is sensor name) before
you restart.
6. Or you can simply stop Metron Indexing and then start it. This will
trigger Ambari to automatically load the collections.
7. Navigate to http://node1:8983/solr/#/snort/query or
http://node1:8983/solr/#/bro/query and hit the "Execute Search" button. You
should see data.

There is a lot to digest here and still more questions to answer but this
should get us started. Outstanding items include:

- What Solr parameters do we expose in Ambari? All of them?
- Do we want to add Solr collection functions to Service actions? I don't
see a way to make this conditional on the selected search engine so both
options would always be shown
- Need to review and add documentation
- Support for REST configuration (probably a separate PR)

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```

- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output i

[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/929
  
Yes I pushed the change to zookeeper with Swagger.  Probably just a minor 
detail he missed in his instructions.  

You can use either "localhost:9983" or "node1:9983".  That address is the 
embedded zookeeper that Solr starts up in this case but in general it is 
whatever zookeeper you configure Solr to use.


---

[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/929
  
+1 from me pending approval from others.  Tested this in full dev and 
worked fine.


---

[GitHub] metron issue #932: METRON-1451: On Centos full dev, Metron Indexing shows up...

2018-02-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/932
  
I spun this up and I know longer see the Indexing component as red.  +1 as 
soon as @nickwallen's build finishes and it's working there too.


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

2018-02-06 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
Looks good to me.  +1


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

2018-02-05 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
I would say collection for each parser and an error collection.  It's 
similar to ES templates where we define fields, types, etc.


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

2018-02-05 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
The errors I found happened when I tried to create collections with each 
schema.  The error related to the `guid` field happens because it is defined as 
the unique key but not included in the list of fields (for error schema).


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

2018-02-02 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
I tested this in full dev using the install script in 
https://github.com/apache/metron/pull/918.  I was able to create collections 
for each schema except for "error".  For that to work properly, I had to: 

- remove `docValues="true"` from the "bytes" field type
- add the "guid" field used in other schemas

Still working on indexing data into these collections but so far so good.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-02-01 Thread merrimanr

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


---

[GitHub] metron issue #918: METRON-1436: Manually Install Solr Cloud in Full Dev

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/918
  
I tested it out and worked fine.  I think it's a good start.  +1 as long as 
other commenters are satisfied.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142091
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/components/SolrComponent.java
 ---
@@ -158,4 +162,16 @@ public boolean hasCollection(String collection) {
 }
 return docs;
   }
+
+  public void addDocs(String collection, List<Map<String, Object>> docs)
+  throws IOException, SolrServerException {
+CloudSolrClient solr = miniSolrCloudCluster.getSolrClient();
+solr.setDefaultCollection(collection);
+Collection solrInputDocuments = 
docs.stream().map(doc -> {
+  SolrInputDocument solrInputDocument = new SolrInputDocument();
+  doc.entrySet().forEach(entry -> 
solrInputDocument.addField(entry.getKey(), entry.getValue()));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141979
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+Strin

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142049
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.integration;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.SearchIntegrationTest;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.integration.InMemoryComponent;
+import org.apache.metron.solr.dao.SolrDao;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.json.simple.JSONArray;
+import org.json.simple.parser.JSONParser;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SolrSearchIntegrationTest extends SearchIntegrationTest {
+
+  private SolrComponent solrComponent;
+
+  @Override
+  protected IndexDao createDao() throws Exception {
+AccessConfig config = new AccessConfig();
+config.setMaxSearchResults(100);
+config.setMaxSearchGroups(100);
+config.setGlobalConfigSupplier( () ->
+new HashMap<String, Object>() {{
+  put("solr.zookeeper", solrComponent.getZookeeperUrl());
+}}
+);
+
+IndexDao dao = new SolrDao();
+dao.init(config);
+return dao;
+  }
+
+  @Override
+  protected InMemoryComponent startIndex() throws Exception {
+solrComponent = new SolrComponent.Builder()
+.addCollection("bro", 
"../metron-solr/src/test/resources/config/bro/conf")
+.addCollection("snort", 
"../metron-solr/src/test/resources/config/snort/conf")
+.build();
+solrComponent.start();
+return solrComponent;
+  }
+
+  @Override
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141915
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+Strin

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141548
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+Strin

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141296
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -655,83 +699,54 @@ public void disabled_facet_query_returns_null_count() 
throws Exception {
   }
 
   @Test
-  public void exceeding_max_resulsts_throws_exception() throws Exception {
+  public void missing_type_facet_query() throws Exception {
+SearchRequest request = JSONUtils.INSTANCE.load(missingTypeFacetQuery, 
SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+
+Map<String, Map<String, Long>> facetCounts = response.getFacetCounts();
+Assert.assertEquals(1, facetCounts.size());
+Map<String, Long> snortFieldCounts = facetCounts.get("snort_field");
+Assert.assertEquals(5, snortFieldCounts.size());
+Assert.assertEquals(new Long(1), snortFieldCounts.get("50"));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141322
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java
 ---
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.ColumnMetadataDao;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrDao implements IndexDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final String ID_FIELD = "guid";
--- End diff --

Done


---

[GitHub] metron pull request #917: METRON-1435: Management UI cannot save json object...

2018-01-30 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/917

METRON-1435: Management UI cannot save json objects in advanced config

## Contributor Comments
This PR fixes a bug in the Management UI that limits a user to only 
entering strings in the Advanced Config section.  This can be verified in full 
dev by navigating to the Management UI and editing the bro sensor.  Expand the 
Advanced Config by clicking on the "Advanced" link at the bottom of the main 
panel.  In the "PARSER CONFIG" section, enter a field name and `["value"]` as 
the value.  Use Swagger to get the bro config and the parserConfig should look 
like:
```
{
  ...
  "parserConfig": {
"field": ["value"]
  }
}
```
Before this PR the parserConfig would have been incorrectly converted to a 
string:
```
{
  ...
  "parserConfig": {
"field": "[\"value\"]"
  }
}
```
This will also work for json objects as well as arrays.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1435

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #917


commit 1a58b1a109ca44f4c44ab0a62d29f323cb1cf969
Author: merrimanr <merrimanr@...>
Date:   2018-01-30T22:12:53Z

initial commit




---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-30 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164881484
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -443,11 +495,11 @@ public void all_query_returns_all_results() throws 
Exception {
 Assert.assertEquals(10, results.size());
 for(int i = 0;i < 5;++i) {
   Assert.assertEquals("snort", 
results.get(i).getSource().get("source:type"));
-  Assert.assertEquals(10 - i, 
results.get(i).getSource().get("timestamp"));
+  Assert.assertEquals(10 - i + "", 
results.get(i).getSource().get("timestamp").toString());
--- End diff --

The ES and Solr clients return different types (Int vs Longs) so I am 
converting to string to account for that.  I'm not sure this is the correct 
approach and open to ideas.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-30 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164881237
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchDao.java
 ---
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.metron.indexing.dao.update.Document;
+
+public interface SearchDao {
--- End diff --

I was undecided as to whether we should do it in this PR or a future one 
and also wanted to make sure people liked it.  Will make this change.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-29 Thread merrimanr

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-29 Thread merrimanr

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes
all tests in SearchIntegrationTest and UpdateIntegrationTest.

A high level summary of the changes include:

- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields
with different types, faceting on fields with different types, and faceting on
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand
classes. The ElasticsearchDao class has become very large so I attempted to
make the SolrDao more readable.

There were a couple areas where Elasticsearch and Solr behave slightly
different. I attempted to accommodated for that through the SolrDao
implementation, by adjusting existing tests, and by splitting out specific
tests:

- Column metadata is different between the 2 search engines so each
implementation has their own tests
- There are cases where the clients will return different types in search
results. I am handling this in SearchIntegrationTest by first converting the
types to strings and then comparing (other ideas?). For example, the ES client
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while
Solr does not (and vice-versa). These were moved to either ES or Solr
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting
them client-side instead.

At this point the scope is limited to tests passing, meaning
metron-elasticsearch and metron-solr pass all tests. There are other PRs in
progress that are needed before automated testing with full dev can be done. I
am still actively working on manually testing in full dev and adding
documentation but this should get us started.

This PR is intended to be merged into the upgrade-solr feature branch but I
have it set to master temporarily so review is easier. We will need to merge
in master to the feature branch to get rid of the extra commits since this PR
is up to date with master.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running lo

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-26 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

A high level summary of the changes include:

At this point the scope is limited to tests passing. There are other PRs
in progress that are needed before automated testing with full dev can be done.
I am still actively working on manually testing in full dev and adding
documentation but this should get us started.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

[GitHub] metron issue #892: METRON-1392 Fix a test case to expect an Exception when r...

2018-01-26 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/892
  
+1 by inspection.  Thanks for fixing this @MohanDV.  The original test 
didn't even make sense.


---

[GitHub] metron issue #898: METRON-1398:Exclude the basic-error-controller from being...

2018-01-26 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/898
  
+1 by inspection.  Thanks @MohanDV!


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

2018-01-25 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/909
  
I think this is useful outside of any Solr work and I intended for it to go 
into master.


---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

2018-01-25 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/909#discussion_r163859481
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -724,6 +503,52 @@ public void sort_query_sorts_results_ascending() 
throws Exception {
 }
   }
 
+  @Test
+  public void sort_ascending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortAscendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(0).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(1).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(2).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(3).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(4).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(5).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(6).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(7).getSource().containsKey("threat:triage:score"));
+
+// validate sorted order - there are only 2 with a 
'threat:triage:score'
+Assert.assertEquals("10", 
results.get(8).getSource().get("threat:triage:score"));
+Assert.assertEquals("20", 
results.get(9).getSource().get("threat:triage:score"));
+  }
+
+  @Test
+  public void sort_descending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortDescendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// validate sorted order - there are only 2 with a 
'threat:triage:score'
+Assert.assertEquals("20", 
results.get(0).getSource().get("threat:triage:score"));
+Assert.assertEquals("10", 
results.get(1).getSource().get("threat:triage:score"));
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(2).getSource().containsKey("threat:triage:score"));
--- End diff --

Done


---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

2018-01-25 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/909#discussion_r163859462
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -724,6 +503,52 @@ public void sort_query_sorts_results_ascending() 
throws Exception {
 }
   }
 
+  @Test
+  public void sort_ascending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortAscendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(0).getSource().containsKey("threat:triage:score"));
--- End diff --

Done


---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

2018-01-24 Thread merrimanr

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/909

METRON-1429: SearchIntegrationTest refactor

## Contributor Comments
This PR cleans up SearchIntegrationTest which will make it easier to create 
a Solr implementation.  Changes include:

- Updated the "all_query_returns_all_results" method in 
SearchIntegrationTest to only contain a single test rather than also including 
duplicate tests that were moved to their own methods during the ES 5 upgrade 
work.  Tests that were not duplicated were moved to their own method.
- Removed logic from ElasticsearchSearchIntegrationTest that rewrites guids 
without the `-` character.  This made the tests confusing because different ids 
were being used for validation and is no longer needed.
- Metaalert data was removed and tests involving metaalerts were updated to 
use other sensors instead.  There is no reason for metaalerts to be included 
here as there are other dedicated metaalert integration tests. 
- Improved the "get_all_latest_guid" test by using guids from different 
sensor types.

All tests should still run after this change.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #909


commit 76455986a14d12ed720a83c091616a1904ffb2b1
Author: merrimanr <merrimanr@...>
Date:   2018-01-24T19:07:46Z

initial commit

commit a29ed14c39a79ce5918212baa9b075706319e4de
Author: merrimanr <merrimanr@...>
Date:   2018-01-24T20:31:11Z

added back metaalert test data

commit 5c6293dd5302cddf3f212a982fed8c2d0ec2e391
Author: merrimanr <merrimanr@...>
Date:   2018-01-24T22:33:53Z

Merge remote-tracking branch 'mirror/master' into METRON-1429




---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-01-23 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
I will address 6 shortly.  For 5, should we explore a more flexible store 
in this PR?  Or at least validate that an RDBMS is the right choice?  I think 
this is something we should tackle now as it will likely affect future work as 
well.  Does it make sense to start a discuss thread on this topic?


---

[GitHub] metron issue #891: METRON-1282 creating a new topic using the rest end point...

2018-01-18 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/891
  
I think a separate PR is fine.  Can you at least move it to a separate 
method? 


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-01-09 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Here are my thoughts on your responses.

1.  I think we're in agreement.  This is how this PR currently works.
2. No, saved searches are currently stored client-side.  I think the plan 
was to store these server-side but it hasn't been implemented yet.  The only 
thing we store in a relational DB at this point are REST user credentials.  
This would change if we switch to an LDAP approach for managing authentication 
(I think we should) and nothing we be stored in a DB at that point.
3. Agreed.
4. Agreed.  I don't see much value in versioning a user's displayable facet 
fields.
5. My point here is that a more flexible store for things like this (user 
profile settings) may eliminate this upgrade complexity altogether.
6. I'm good with that.


---

[GitHub] metron issue #891: METRON-1282 creating a new topic using the rest end point...

2018-01-09 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/891
  
@MohanDV this is a good start and will be a nice feature to have.  I have a 
couple suggestions.  

First, can we move this to it's own function/endpoint?  I think this would 
be useful for assigning permissions to existing topics as well.  

Second, I think we need some kind of check that Kerberos is enabled before 
we try to permission a topic for a user.  This would be similar to what's done 
in the 
[KafkaConfig](https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java)
 class on line 111.


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-01-09 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Done


---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2018-01-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/840
  
I ran this up in full dev again and verified the e2e tests now work similar 
to how they do in master.  I also manually tested several other areas including 
the Alerts UI, Kibana and Swagger.  Everything works as expected.  Assuming 
others are still testing but +1 from me.  


---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

2018-01-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/786
  
We don't have e2e tests for the management UI.  We do have unit tests and I 
added tests for these fixes there.


---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

2018-01-05 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/786
  
@justinleet 1, 2, and 3 have been addressed with the latest commit.  I also 
added a couple unit tests for good measure.  

I wasn't able to easily reproduce 4 so I didn't fix that one.  It is not 
related to this at all and would probably make more sense in a separate bug fix 
PR. 


---

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

2018-01-05 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/884
  
I spun this up again and all the issues I found have been resolved.  Great 
work.  +1


---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

2018-01-05 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/786
  
I believe 2 and 4 are preexisting but I'm happy to fix them here.  Will 
update when I have resolved these issues.


---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2018-01-04 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/840
  
I spun this up in full dev and spent all day testing it.  From a functional 
perspective, I can not find anything wrong with it.  I ran through the test 
plan in this PR and everything worked as expected.  I also tested this 
exhaustively with the Alerts UI and Swagger UI and everything works great.

The only issue I found are with the Alerts UI e2e tests.  These no longer 
run at all and I suspect it's because the templates have changed with ES 5.6.2 
so loading e2e test data and template no longer works.  I'm not sure that this 
should hold up this PR since e2e tests are actively being worked on in 
https://github.com/apache/metron/pull/857 but I wanted everyone to be aware.


---

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

2018-01-03 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/884
  
I also noticed that functions returning a list only display the first item 
in both the shell and Zeppelin output.

For example, the expression `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` 
returns `FOO` when I would expect it to return `['FOO', 'BAR']`.  

From what I can tell the problem is only in how the result is displayed.  
For example, the expression `'BAR' in MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) 
)` returns `true` as expected.


---

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

2018-01-03 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/884
  
I think this is an excellent start.  So far I have only reviewed it from a 
user perspective and it's working well so far.  

I've spun it up in full dev (not sure that's even necessary) and installed 
this via the instructions in the README.  I was able to run most of the 
examples in the Stellar README.

I found that the functions available in Zeppelin are a subset of what's 
available in the Stellar shell.  The missing functions include IS_EMAIL, 
ENRICHMENT*, GEO*, STATS* and many others.  Is this expected?

Is there a way to pass in the zookeeper url in Zeppelin?



---

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/831
  
I ran this up in full dev and everything worked as advertised.  I only 
noticed a couple minor issues and left comments for those.

I also am a little confused by the ra/batch vs es/hdfs issue.  I still see 
places (Ambari config parameter names, Ambari MPack scripts, Flux file paths, 
Flux properties, Storm topology names, etc) where the prefixes are ra/batch and 
not es/hdfs.  Reading through the PR comments I'm still not clear on what 
approach we decided on but I think consistency would be good.


---

[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r159322924
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/hdfs.properties.j2
 ---
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
--- End diff --

Should this file be git ignored since it is copied in at build time?


---

[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r159321881
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
 ---
@@ -197,9 +140,7 @@ public void test() throws Exception {
   //assert that our input docs are equivalent to the output docs, 
converting the input docs keys based
   // on the field name converter
   assertInputDocsMatchOutputs(inputDocs, docs, 
getFieldNameConverter());
-  assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir), x 
-> x);
-} catch(Throwable e) {
-  e.printStackTrace();
+  //assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir), 
x -> x);
--- End diff --

Is this comment intentional?


---

[GitHub] metron pull request #886: METRON-1385: Missing "properties" in index templat...

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/886

METRON-1385: Missing "properties" in index template causes 
ElasticsearchColumnMetadataDao.getColumnMetadata to fail

## Contributor Comments
A bug was recently discovered that causes a NPE when calling 
ElasticsearchColumnMetadataDao.getColumnMetadata to get ES column types.  It 
can be recreated by adding a template to ES that matches all indices and also 
contains a doc type mapping without the "properties" property.  For example:
```
curl -XPUT 'http://node1:9200/_template/default_string_template' -d '
 {
   "template": "*",
   "mappings" : {
 "bro_type": {
   "dynamic_templates": [
 {
   "strings": {
 "match_mapping_type": "string",
 "mapping": {
   "type": "text"
 }
   }
 }
   ]
 }
   }
 }
```
A NPE should now happen when calling the REST endpoint to get column 
metadata for bro.

This PR fixes that bug by first checking to see if the "properties" 
property exists and continuing on if it does not.  I also added an additional 
mapping in ElasticsearchSearchIntegrationTest that mimics this condition.  
Happy to remove it if people feel it's unnecessary.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1385

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/886.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #886


commit 8905d4d21926ac4e518d14fb0b4b44fbf4dea158
Author: merrimanr <merrimanr@...>
Date:   2018-01-02T16:22:12Z

initial commit




---

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/857
  
@justinleet I believe @iraghumitra is still working on the 
SELENIUM_PROMISE_MANAGER change proposed above but I will defer to him.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
@cestella are you good with merging this in to the feature branch?  I 
believe I addressed your comments.


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

2017-12-14 Thread merrimanr

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/858


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

2017-12-14 Thread merrimanr

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/858

METRON-1344: Externalize the infrastructural components using integration
tests

## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the
Alerts UI e2e test to be run. There are several outstanding issues that still
need to be worked out so DO NOT MERGE this yet.

This is a first pass at a potential Docker-based solution and is meant to
be a POC. The intention is to facilitate further discussion around the general
approach and provide a working example that we can build off of.

A good place to start is the .travis file. This provides a good guide on
how this infrastructure is spun up. It is assumed Docker and Docker Compose
are installed. To use outside of travis in a local dev environment (assuming
Mac OSX):

1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with
`metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env
metron-machine)`
1. Build the base Metron image that installs Java: `docker build
./metron-centos/ -t "metron-centos"`
1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose
&& docker-compose up -d`

A working environment should now be available. To verify get the Docker
machine address with `echo $DOCKER_HOST`. Services should be available on the
ports specified in
`metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example,
assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should
be available at http://192.168.99.100:9210.

At this point only a single e2e test is run due to the significant
refactoring being done in https://github.com/apache/metron/pull/857.

Here are my thoughts so far based on work in this PR:

- The Docker environment creation and startup adds about 3 minutes to the
build. This is about what I expected and hopefully we can get this time back
as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are
necessary or desired. We may be able to cut some time off the build by running
them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time
was actually faster

There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous
commits but have all but e2e tests commented out for now to make it easier to
see what's going on in Travis. These will need to be added back once we get
closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in
and tested
- The license check is commented out right now because I inadvertently
added some new dependency versions (have no idea why). Still need to track
that down.
- There is an intermittent error that happens when starting up REST. Still
working on tracking this down.

Looking forward to some feedback.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?

- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

2017-12-13 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/859#discussion_r156744095
  
--- Diff: metron-deployment/amazon-ec2/README.md ---
@@ -126,6 +126,10 @@ To provision only subsets of the entire Metron 
deployment, Ansible tags can be s
 ./run.sh --tags="ec2,sensors"
 ```
 
+### Setting REST API Profile
+
--- End diff --

Spring profiles are documented in the REST README:  
https://github.com/apache/metron/tree/master/metron-interface/metron-rest#spring-profiles.
  

Is there something we can do to make the REST README more accessible?  I 
feel like a lot of questions people ask are already answered there but no one 
ever reads it.  What can we do to make it more useful?  Table of contents 
maybe?  I would be happy to take that on in a follow-up PR.


---

[GitHub] metron issue #863: METRON-1347: Indexing Topology should fail tuples without...

2017-12-13 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/863
  
I would like to hear feedback from @ottobackwards on other required fields 
but this looks good to me otherwise.  


---

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

2017-12-13 Thread merrimanr

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/863#discussion_r156674159
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
 ---
@@ -229,17 +239,30 @@ public void execute(Tuple tuple) {
   LOG.trace("Writing enrichment message: {}", message);
   WriterConfiguration writerConfiguration = 
configurationTransformation.apply(
   new IndexingWriterConfiguration(bulkMessageWriter.getName(), 
getConfigurations()));
-  if(writerConfiguration.isDefault(sensorType)) {
-//want to warn, but not fail the tuple
-collector.reportError(new Exception("WARNING: Default and (likely) 
unoptimized writer config used for " + bulkMessageWriter.getName() + " writer 
and sensor " + sensorType));
+  if(sensorType == null) {
--- End diff --

@ottobackwards which fields should we validate here?


---

[GitHub] metron issue #834: METRON-1306: When index template install fails, we should...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/834
  
After reading this whole thread I agree with @ottobackwards.  If we 
installed and started all services and THEN installed our parsers separately 
this whole issue goes away.  If a parser's template doesn't exist or isn't 
installed correctly, it fails and hopefully we know exactly why.  That is 
obviously a major architectural change and won't be solved in this PR.  

Since we're in a situation where we do include default parsers as part of 
the installation, I think they should work out of the box and fail fast if 
there are not installed correctly.  Would some of the concerns be alleviated 
with https://github.com/apache/metron/pull/831?  

+1 from me either way.  


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Ambari does not manage this config.  It is only included in the base 
application.yml as a default setting.  I don't feel like this setting should be 
in Ambari for a couple reasons:  changing it requires a restart and the list of 
facets cannot be user specific.



---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/858

METRON-1344: Externalize the infrastructural components using integration
tests

At this point only a single e2e test is run due to the significant
refactoring being done in https://github.com/apache/metron/pull/857.

Here are my thoughts so far based on work in this PR:

Looking forward to some feedback.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/858


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Each environment will have it's own application.yaml.  Full dev has one, 
our testing environment has one, Ambari ships one etc.  There is also a base 
application.yaml that has the defaults (default facets are included here).  
Does that make sense?


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r156215449
  
--- Diff: metron-interface/metron-alerts/protractor.conf.js ---
@@ -25,27 +25,28 @@ var SpecReporter = 
require('jasmine-spec-reporter').SpecReporter;
 exports.config = {
   allScriptsTimeout: 25000,
   specs: [
-'./e2e/login/login.e2e-spec.ts',
-'./e2e/alerts-list/alerts-list.e2e-spec.ts',
+// './e2e/login/login.e2e-spec.ts',
--- End diff --

Done


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r156215392
  
--- Diff: 
metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
 ---
@@ -58,26 +58,26 @@ describe('metron-alerts configure table', function() {
 page.saveConfigureColumns();
   });
 
-  it('should rename columns from table configuration', () => {
-page.clearLocalStorage();
-page.navigateTo();
-
-page.clickConfigureTable();
-page.renameColumn('enrichments:geo:ip_dst_addr:country', 'Country');
-page.saveConfigureColumns();
-
-page.clickTableText('FR');
-expect(page.getSearchText()).toEqual('Country:FR');
-expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts 
(25)');
-page.clickClearSearch();
-
-expect(page.getChangesAlertTableTitle('Alerts (25)')).toEqual('Alerts 
(169)');
-page.setSearchText('Country:FR');
-expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts 
(25)');
-page.clickClearSearch();
-
-expect(page.getTableColumnNames()).toContain('Country', 'for renamed 
column names for alert list table');
-
-  });
+  // it('should rename columns from table configuration', () => {
--- End diff --

Done


---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/858#discussion_r156215355
  
--- Diff: 
metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
 ---
@@ -47,7 +47,7 @@ describe('metron-alerts configure table', function() {
 let newColNamesColumnConfig = [ 'score', 'timestamp', 'source:type', 
'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
   'ip_dst_addr', 'host', 'alert_status', 'guid' ];
 
-page.clearLocalStorage();
+//page.clearLocalStorage();
--- End diff --

Done


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...