[GitHub] metron pull request #806: METRON-1262: Unable to add comment for a alert in ...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/806
METRON-1262: Unable to add comment for a alert in a meta-alert
## Contributor Comments
This PR fixes a bug in the ElasticsearchMetaalertDao class (description is
in the Jira). I verified it in full dev using the following steps:
1. Search for alerts in an index (you need a couple guids) with the
http://node1:8082/swagger-ui.html#!/search-controller/searchUsingPOST endpoint.
I used the alerts_ui_e2e data set that can be created with
`https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/e2e/mock-data/setup.sh`:
```
{
"from": 0,
"indices": [
"alerts_ui_e2e"
],
"query": "*",
"size": 5
}
```
2. Pick a couple guids from the previous step and use the
http://node1:8082/swagger-ui.html#!/meta-alert-controller/createUsingPOST
endpoint to create a metaalert:
```
{
"groups": [
"string"
],
"guidToIndices": {
"c4c5e418-3938-099e-bb0d-37028a98dca8": "alerts_ui_e2e",
"fa91598f-51b2-2b60-11f2-6fbabc162b7e": "alerts_ui_e2e"
}
}
```
3. Use the
http://node1:8082/swagger-ui.html#!/search-controller/searchUsingPOST endpoint
to see validate the metaalert you created in the previous step:
```
{
"from": 0,
"indices": [
"metaalert"
],
"query": "*",
"size": 5
}
```
4. Update one of the alerts with the
http://node1:8082/swagger-ui.html#!/update-controller/patchUsingPATCH endpoint:
```
{
"guid": "c4c5e418-3938-099e-bb0d-37028a98dca8",
"index": "alerts_ui_e2e_index",
"patch": [
{
"op": "add",
"path": "/comments",
"value": [
{
"comment": "aaa",
"username": "admin",
"timestamp": 1508251594109
},
{
"comment": "aaa",
"username": "admin",
"timestamp": 1508251398188
},
{
"comment": "abcd",
"username": "admin",
"timestamp": 1508251201985
},
{
"comment": "ccc",
"username": "admin",
"timestamp": 1508244721089
},
{
"comment": "c123",
"username": "admin",
"timestamp": 1508244381778
}
]
}
]
}
```
5. Rerun the metadata search with the
http://node1:8082/swagger-ui.html#!/search-controller/searchUsingPOST endpoint:
```
{
"from": 0,
"indices": [
"metaalert"
],
"query": "*",
"size": 5
}
```
The alert you updated should also have the same update within the
metaalert. Both alerts should also be present.
A couple of things I want to point out:
- I ran into a NoSuchMethod error and had to remove an old jackson
dependency from metron-elasticsearch pom.xml. It was a pretty old dependency
and I haven't seen any issues since.
- I added a "guid" field to MetaAlertCreateResponse class. I believe this
makes it more useful when waiting on a metaalert create to propagate.
- It was convenient to include both the patch and replace operations in the
same test. I can split these out if desired (it will make the test more
verbose and add a duplicate test setup).
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and te
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/802 ---
[GitHub] metron issue #802: METRON-1255: MetaAlert search is not filtering on status
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/802 +1, thanks for this, it's good stuff ---
[GitHub] metron issue #802: METRON-1255: MetaAlert search is not filtering on status
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/802 Feedback has been addressed. Take another look when you get a chance. ---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145532341
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -177,15 +178,18 @@ public MetaAlertCreateResponse
createMetaAlert(MetaAlertCreateRequest request)
public SearchResponse search(SearchRequest searchRequest) throws
InvalidSearchException {
// Wrap the query to also get any meta-alerts.
QueryBuilder qb = constantScoreQuery(boolQuery()
-.should(new QueryStringQueryBuilder(searchRequest.getQuery()))
-.should(boolQuery()
-.must(termQuery(MetaAlertDao.STATUS_FIELD,
MetaAlertStatus.ACTIVE.getStatusString()))
-.must(nestedQuery(
+.must(boolQuery()
+.should(new QueryStringQueryBuilder(searchRequest.getQuery()))
+.should(nestedQuery(
ALERT_FIELD,
new QueryStringQueryBuilder(searchRequest.getQuery())
)
)
)
+.must(boolQuery()
--- End diff --
Done in latest commit.
---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145532292
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -302,12 +310,126 @@ public void test() throws Exception {
}
}
- protected boolean findUpdatedDoc(Map message0, String
guid)
+ /**
+ {
+ "guid": "search_by_status_active",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "active"
+ }
+ */
+ @Multiline
+ public static String activeMetaAlert;
+
+ /**
+ {
+ "guid": "search_by_status_inactive",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "inactive"
+ }
+ */
+ @Multiline
+ public static String inactiveMetaAlert;
+
+ @Test
+ public void shouldSearchByStatus() throws Exception {
+List> metaInputData = new ArrayList<>();
+Map activeMetaAlertJSON =
JSONUtils.INSTANCE.load(activeMetaAlert, new TypeReference>() {});
+metaInputData.add(activeMetaAlertJSON);
+Map inactiveMetaAlertJSON =
JSONUtils.INSTANCE.load(inactiveMetaAlert, new TypeReference>() {});
+metaInputData.add(inactiveMetaAlertJSON);
+
+// We pass MetaAlertDao.METAALERT_TYPE, because the "_doc" gets
appended automatically.
+elasticsearchAdd(metaInputData, MetaAlertDao.METAALERTS_INDEX,
MetaAlertDao.METAALERT_TYPE);
+// Wait for updates to persist
+findUpdatedDoc(inactiveMetaAlertJSON, "search_by_status_inactive",
MetaAlertDao.METAALERT_TYPE);
+
+SearchResponse searchResponse = metaDao.search(new SearchRequest() {
+ {
+setQuery("*");
+setIndices(Collections.singletonList(MetaAlertDao.METAALERT_TYPE));
+setFrom(0);
+setSize(5);
+setSort(Collections.singletonList(new SortField(){{
setField(Constants.GUID); }}));
+ }
+});
+Assert.assertEquals(1, searchResponse.getTotal());
+Assert.assertEquals(MetaAlertStatus.ACTIVE.getStatusString(),
searchResponse.getResults().get(0).getSource().get(MetaAlertDao.STATUS_FIELD));
+ }
+
+ /**
+ {
+ "guid": "search_by_nested_alert_0",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.1",
+ "ip_src_port": 8010
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert0;
+
+ /**
+ {
+ "guid": "search_by_nested_alert_1",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.2",
+ "ip_src_port": 8009
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert1;
+
+ @Test
+ public void shouldSearchByNestedAlert() throws Exception {
+List> inputData = new ArrayList<>();
+Map searchByNestedAlert0JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert0, new TypeReference>() {});
+inputData.add(searchByNestedAlert0JSON);
+Map searchByNestedAlert1JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert1, new TypeReference>() {});
+inputData.add(searchByNestedAlert1JSON);
+elasticsearchAdd(inputData, INDEX, SENSOR_NAME);
--- End diff --
Done in latest commit.
---
[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user as22323 commented on the issue: https://github.com/apache/metron/pull/754 Just an FYI: This PR is to simply change "control_path = ~/.ssh/ansible-ssh-%%C"(in ../amazon-ec2/ansible.cfg) to "control_path = ~/.ssh/ansbile-ssh-%%h-%%r" to allow linux systems to deploy to EC2. When I tested the change with Mac it didn't throw any errors at the same point for ""TASK [setup] ***" ---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/805 Okay, I just spun up full-dev and followed my testing instructions - all looks good to me. This is ready for a review. ---
Re: [DISCUSS] Build broken due to transitive dependencies
I was hesitant to believe Ryan that this was a compiler issue, but I
upgraded my compiler on CentOS 6 to 4.9.2 and the build worked on the
first try... Lesson learned: Never question Ryan again!
How to upgrade compiler on CentOS 6:
$ sudo yum install centos-release-scl
$ sudo yum install devtoolset-3-toolchain
$ scl enable devtoolset-3 bash
$
On 2017-10-13 11:12, Ryan Merriman wrote:
We recently ran into this and the cause was an old C++ compiler
version.
It wants a compiler that has support for C++11:
https://gcc.gnu.org/projects/cxx-status.html#cxx11.
On Fri, Oct 13, 2017 at 1:00 PM, Laurens Vets
wrote:
...
[INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config
---
[DEBUG] Configuring mojo
com.github.eirslett:frontend-maven-plugin:1.3:npm
from plugin realm ClassRealm[plugin>com.github.e
irslett:frontend-maven-plugin:1.3, parent:
sun.misc.Launcher$AppClassLoad
er@70dea4e]
[DEBUG] Configuring mojo
'com.github.eirslett:frontend-maven-plugin:1.3:npm'
with basic configurator -->
[DEBUG] (f) arguments = run build
[DEBUG] (f) npmInheritsProxyConfigFromMaven = false
[DEBUG] (f) project = MavenProject:
org.apache.metron:metron-config:0.4.1
@ /root/metron/metron-interface/metron-config/pom.xml
[DEBUG] (f) repositorySystemSession = org.eclipse.aether.DefaultRepo
sitorySystemSession@e883a51
[DEBUG] (f) session =
org.apache.maven.execution.MavenSession@2aaefbd
[DEBUG] (f) skip = false
[DEBUG] (f) skipTests = true
[DEBUG] (f) workingDirectory = /root/metron/metron-interface/
metron-config
[DEBUG] (f) execution =
com.github.eirslett:frontend-maven-plugin:1.3:npm
{execution: ng build}
[DEBUG] -- end configuration --
[INFO] npm not inheriting proxy config from Maven
[INFO] Running 'npm run build' in /root/metron/metron-interface/
metron-config
[INFO]
[INFO] > metron-management-ui@0.4.1 build
/root/metron/metron-interface/
metron-config
[INFO] > ./node_modules/angular-cli/bin/ng build -prod
[INFO]
[INFO] Cannot find module 'tough-cookie'
[INFO] Error: Cannot find module 'tough-cookie'
[INFO] at Function.Module._resolveFilename (module.js:440:15)
[INFO] at Function.Module._load (module.js:388:25)
[INFO] at Module.require (module.js:468:17)
[INFO] at require (internal/module.js:20:19)
[INFO] at Object. (/root/metron/metron-interface
/metron-config/node_modules/request/lib/cookies.js:3:13)
[INFO] at Module._compile (module.js:541:32)
[INFO] at Object.Module._extensions..js (module.js:550:10)
[INFO] at Module.load (module.js:458:32)
[INFO] at tryModuleLoad (module.js:417:12)
[INFO] at Function.Module._load (module.js:409:3)
[INFO] at Module.require (module.js:468:17)
[INFO] at require (internal/module.js:20:19)
[INFO] at Object. (/root/metron/metron-interface
/metron-config/node_modules/request/index.js:18:15)
[INFO] at Module._compile (module.js:541:32)
[INFO] at Object.Module._extensions..js (module.js:550:10)
[INFO] at Module.load (module.js:458:32)
[INFO] at tryModuleLoad (module.js:417:12)
[INFO] at Function.Module._load (module.js:409:3)
[INFO] at Module.require (module.js:468:17)
[INFO] at require (internal/module.js:20:19)
[INFO] at Leek._enqueue (/root/metron/metron-interface
/metron-config/node_modules/leek/lib/leek.js:60:30)
[INFO] at Leek.track (/root/metron/metron-interface
/metron-config/node_modules/leek/lib/leek.js:87:15)
[INFO] at Class.Command.validateAndRun
(/root/metron/metron-interface
/metron-config/node_modules/angular-cli/lib/models/command.js:119:18)
[INFO] at
/root/metron/metron-interface/metron-config/node_modules/ang
ular-cli/lib/cli/cli.js:86:22
[INFO] at tryCatch (/root/metron/metron-interface
/metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:198:12)
[INFO] at invokeCallback (/root/metron/metron-interface
/metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:211:13)
[INFO] at
/root/metron/metron-interface/metron-config/node_modules/rsv
p/dist/lib/rsvp/then.js:26:14
[INFO] at flush (/root/metron/metron-interface
/metron-config/node_modules/rsvp/dist/lib/rsvp/asap.js:80:5)
[INFO] at _combinedTickCallback
(internal/process/next_tick.js:67:7)
[INFO] at process._tickCallback
(internal/process/next_tick.js:98:9)
[ERROR]
[ERROR] npm ERR! Linux 2.6.32-696.13.2.el6.x86_64
[ERROR] npm ERR! argv
"/root/metron/metron-interface/metron-config/node/node"
"/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
"run" "build"
[ERROR] npm ERR! node v6.2.0
[ERROR] npm ERR! npm v3.8.9
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! metron-management-ui@0.4.1 build:
`./node_modules/angular-cli/bin/ng build -prod`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR!
[ERROR] npm ERR! Failed at the metron-management-ui@0.4.1 build script
'./node_modules/angular-cli/bin/ng build -prod'.
[ERROR] npm ERR! Make sure you have the latest version of node.js and
npm
installed.
[ERROR] npm ERR! If y
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/805 https://github.com/nickwallen/metron-commit-stuff/blob/master/checkout-pr ---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145431927
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -83,6 +90,7 @@ public static void setup() throws Exception {
put("es.date.format", DATE_FORMAT);
}
};
+accessConfig.setMaxSearchResults(1000);
--- End diff --
I wouldn't worry about it for this PR, just something to think about.
---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/805 Right @justinleet I've done that in the past as well, this is me just being lazy and not wanting to look up the PR # when drafting my instructions =) That said, those instructions don't depend on my repo, which means it's probably the right way to do it. ---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/805 It's actually easier than this to pull in a PR, for future reference ``` git fetch upstream pull/805/head:METRON-1261 git checkout METRON-1261 ``` Assuming upstream is GitHub, of course. It'll just make a branch in your local and you can check it out there. ---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/805 Maybe we should have the checkout-pr script in the metron repo? ---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145427376
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -83,6 +90,7 @@ public static void setup() throws Exception {
put("es.date.format", DATE_FORMAT);
}
};
+accessConfig.setMaxSearchResults(1000);
--- End diff --
I'm not sure but adding that should be trivial. The REST layer does
provide a default but I can add it here too.
---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145424998
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -83,6 +90,7 @@ public static void setup() throws Exception {
put("es.date.format", DATE_FORMAT);
}
};
+accessConfig.setMaxSearchResults(1000);
--- End diff --
Makes sense. Do you know if there's a reason we don't default there if it's
not set? It's outside the scope of this, so if you don't know I'm not worried,
but it seems like it's an opportunity.
---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user cestella commented on the issue: https://github.com/apache/metron/pull/805 As usual, I really love your test scripts. Well done, @JonZeolla ! ---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145423988
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -83,6 +90,7 @@ public static void setup() throws Exception {
put("es.date.format", DATE_FORMAT);
}
};
+accessConfig.setMaxSearchResults(1000);
--- End diff --
This is necessary or it won't work. There is a check in
ElasticsearchDao.search(SearchRequest searchRequest, QueryBuilder queryBuilder)
that calls accessConfig.getMaxSearchResults().
---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145424237
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -302,12 +310,126 @@ public void test() throws Exception {
}
}
- protected boolean findUpdatedDoc(Map message0, String
guid)
+ /**
+ {
+ "guid": "search_by_status_active",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "active"
+ }
+ */
+ @Multiline
+ public static String activeMetaAlert;
+
+ /**
+ {
+ "guid": "search_by_status_inactive",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "inactive"
+ }
+ */
+ @Multiline
+ public static String inactiveMetaAlert;
+
+ @Test
+ public void shouldSearchByStatus() throws Exception {
+List> metaInputData = new ArrayList<>();
+Map activeMetaAlertJSON =
JSONUtils.INSTANCE.load(activeMetaAlert, new TypeReference>() {});
+metaInputData.add(activeMetaAlertJSON);
+Map inactiveMetaAlertJSON =
JSONUtils.INSTANCE.load(inactiveMetaAlert, new TypeReference>() {});
+metaInputData.add(inactiveMetaAlertJSON);
+
+// We pass MetaAlertDao.METAALERT_TYPE, because the "_doc" gets
appended automatically.
+elasticsearchAdd(metaInputData, MetaAlertDao.METAALERTS_INDEX,
MetaAlertDao.METAALERT_TYPE);
+// Wait for updates to persist
+findUpdatedDoc(inactiveMetaAlertJSON, "search_by_status_inactive",
MetaAlertDao.METAALERT_TYPE);
+
+SearchResponse searchResponse = metaDao.search(new SearchRequest() {
+ {
+setQuery("*");
+setIndices(Collections.singletonList(MetaAlertDao.METAALERT_TYPE));
+setFrom(0);
+setSize(5);
+setSort(Collections.singletonList(new SortField(){{
setField(Constants.GUID); }}));
+ }
+});
+Assert.assertEquals(1, searchResponse.getTotal());
+Assert.assertEquals(MetaAlertStatus.ACTIVE.getStatusString(),
searchResponse.getResults().get(0).getSource().get(MetaAlertDao.STATUS_FIELD));
+ }
+
+ /**
+ {
+ "guid": "search_by_nested_alert_0",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.1",
+ "ip_src_port": 8010
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert0;
+
+ /**
+ {
+ "guid": "search_by_nested_alert_1",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.2",
+ "ip_src_port": 8009
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert1;
+
+ @Test
+ public void shouldSearchByNestedAlert() throws Exception {
+List> inputData = new ArrayList<>();
+Map searchByNestedAlert0JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert0, new TypeReference>() {});
+inputData.add(searchByNestedAlert0JSON);
+Map searchByNestedAlert1JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert1, new TypeReference>() {});
+inputData.add(searchByNestedAlert1JSON);
+elasticsearchAdd(inputData, INDEX, SENSOR_NAME);
--- End diff --
No problem.
---
[GitHub] metron pull request #785: METRON-1230: As a stopgap prior to METRON-777, add...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/785#discussion_r145423853
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/util/ParserIndex.java
---
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.util;
+
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.reflections.Reflections;
+import org.reflections.util.ClasspathHelper;
+import org.reflections.util.ConfigurationBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URL;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Index the parsers. Analyzing the classpath is a costly operation, so
caching it makes sense.
+ * Eventually, we will probably want to have a timer that periodically
reindexes so that new parsers show up.
+ */
+public enum ParserIndex {
+ INSTANCE;
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static Set> index;
+ private static Map availableParsers ;
+
+ static {
+load();
+ }
+
+ public synchronized Map getIndex() {
+if(availableParsers == null) {
+ load();
+}
+return availableParsers;
+ }
+
+ public synchronized Set> getClasses() {
+if(index == null) {
+ load();
+}
+return index;
+ }
+
+ public static void reload() {
+load();
+ }
+
+ /**
+ * To handle the situation where classpath is specified in the manifest
of the jar, we have to augment the URLs.
+ * This happens as part of the surefire plugin as well as elsewhere in
the wild.
+ * @param classLoaders
+ * @return
--- End diff --
Agreed. I corrected it here. Thanks for pointing it out, mike.
---
[GitHub] metron issue #805: METRON-1261: Apply bro security patch
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/805 # Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-1261 git clone https://github.com/apache/metron ~/metron-1261/metron cd ~/metron-1261/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-1261 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-1261/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro so all of the currently supported bro logs are being sent. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin # Look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr are expected, and should be addressed as a part of METRON-939) tail -f /var/log/storm/workers-artifacts/indexing-*/*/worker.log | grep -i "org.elasticsearch.index.mapper.MapperParsingException: failed to parse" # You may want to evaluate worker.log for other errors, but the prior command is helpful to cut through some of the failed indexing of IPv6 addresses ``` 1. Run bro against some public pcaps. ``` # In the first of your three terminals # These are kept separate so that the flat file log output won't stomp the prior ones, for ingest validation mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic ~/brotmp/ssh ~/brotmp/ftp ~/brotmp/radius wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true -O ~/brotmp/ftp/ftp.pcap wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true -O ~/brotmp/radius/radius_localhost.pcapng cd ~/brotmp/example-traffic bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/nitroba bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/ssh bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/ftp bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/radius editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C ``` 1. Validate that terminals 2 and 3 don't have any errors that you don't expect. 1. Verify proper indexing in ES and availability in kibana. ``` # Check around and make sure things look okay declare -a exists notexists; for protocol in http dns conn dpd dhcp ftp ssh ssl smtp radius weird files notice software known_certs x509 known_devices; do if [[ $(curl -s -XGET "node1:9200/bro*/_search?q=pro
[GitHub] metron pull request #800: METRON-1251: Typo and formatting fixes for metron-...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/800#discussion_r145415334 --- Diff: metron-interface/metron-rest/README.md --- @@ -112,42 +112,42 @@ The following configures the application for MySQL: 1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html)) 1. Create a metron user and REST database and permission the user for that database: - ``` -CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd'; -CREATE DATABASE IF NOT EXISTS metronrest; -GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1'; - ``` +``` +CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd'; +CREATE DATABASE IF NOT EXISTS metronrest; +GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1'; +``` 1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable: - ``` -cd $METRON_HOME/lib -wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz -tar xf mysql-connector-java-5.1.41.tar.gz - ``` +``` +cd $METRON_HOME/lib +wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz +tar xf mysql-connector-java-5.1.41.tar.gz +``` 1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL: - ``` -METRON_JDBC_DRIVER="com.mysql.jdbc.Driver" -METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest" -METRON_JDBC_USERNAME="metron" -METRON_JDBC_PLATFORM="mysql" -METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar - ``` +``` +METRON_JDBC_DRIVER="com.mysql.jdbc.Driver" +METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest" +METRON_JDBC_USERNAME="metron" +METRON_JDBC_PLATFORM="mysql" + METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar +``` 1. Switch to the metron user - ``` -sudo su - metron - ``` +``` +sudo su - metron +``` 1. Start the REST API. Adjust the password as necessary. - ``` -set -o allexport; -source /etc/metron/sysconfig; -set +o allexport; -export METRON_JDBC_PASSWORD='Myp@ssw0rd'; -$METRON_HOME/bin/metron-rest.sh -unset METRON_JDBC_PASSWORD; - ``` +``` +set -o allexport; +source /etc/sysconfig/metron; --- End diff -- Ahh, that would make more sense, I was just manually creating /etc/sysconfig/metron. Will update the docs. ---
[GitHub] metron pull request #805: METRON-1261: Apply bro security patch
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/805 METRON-1261: Apply bro security patch ## Contributor Comments This should update the version of bro that is auto-installed by full-dev/quick-dev, and update some manual instructions for setting up bro, to use bro 2.4.2, which recently had a security patch applied ([details](http://mailman.icsi.berkeley.edu/pipermail/bro/2017-October/012606.html) [here](http://blog.bro.org/2017/10/bro-252-242-release-security-update.html)). Additional testing instructions coming soon. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-1261 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/805.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #805 commit d39b72eaacba805703256f30b0e5ddc72e88599a Author: Jon Zeolla Date: 2017-10-18T13:08:21Z METRON-1261: Apply bro security patch ---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145393297
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -83,6 +90,7 @@ public static void setup() throws Exception {
put("es.date.format", DATE_FORMAT);
}
};
+accessConfig.setMaxSearchResults(1000);
--- End diff --
What's the reason for adding this to the test?
---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145378015
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
---
@@ -302,12 +310,126 @@ public void test() throws Exception {
}
}
- protected boolean findUpdatedDoc(Map message0, String
guid)
+ /**
+ {
+ "guid": "search_by_status_active",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "active"
+ }
+ */
+ @Multiline
+ public static String activeMetaAlert;
+
+ /**
+ {
+ "guid": "search_by_status_inactive",
+ "source:type": "metaalert",
+ "alert": [],
+ "status": "inactive"
+ }
+ */
+ @Multiline
+ public static String inactiveMetaAlert;
+
+ @Test
+ public void shouldSearchByStatus() throws Exception {
+List> metaInputData = new ArrayList<>();
+Map activeMetaAlertJSON =
JSONUtils.INSTANCE.load(activeMetaAlert, new TypeReference>() {});
+metaInputData.add(activeMetaAlertJSON);
+Map inactiveMetaAlertJSON =
JSONUtils.INSTANCE.load(inactiveMetaAlert, new TypeReference>() {});
+metaInputData.add(inactiveMetaAlertJSON);
+
+// We pass MetaAlertDao.METAALERT_TYPE, because the "_doc" gets
appended automatically.
+elasticsearchAdd(metaInputData, MetaAlertDao.METAALERTS_INDEX,
MetaAlertDao.METAALERT_TYPE);
+// Wait for updates to persist
+findUpdatedDoc(inactiveMetaAlertJSON, "search_by_status_inactive",
MetaAlertDao.METAALERT_TYPE);
+
+SearchResponse searchResponse = metaDao.search(new SearchRequest() {
+ {
+setQuery("*");
+setIndices(Collections.singletonList(MetaAlertDao.METAALERT_TYPE));
+setFrom(0);
+setSize(5);
+setSort(Collections.singletonList(new SortField(){{
setField(Constants.GUID); }}));
+ }
+});
+Assert.assertEquals(1, searchResponse.getTotal());
+Assert.assertEquals(MetaAlertStatus.ACTIVE.getStatusString(),
searchResponse.getResults().get(0).getSource().get(MetaAlertDao.STATUS_FIELD));
+ }
+
+ /**
+ {
+ "guid": "search_by_nested_alert_0",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.1",
+ "ip_src_port": 8010
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert0;
+
+ /**
+ {
+ "guid": "search_by_nested_alert_1",
+ "source:type": "test",
+ "ip_src_addr": "192.168.1.2",
+ "ip_src_port": 8009
+ }
+ */
+ @Multiline
+ public static String searchByNestedAlert1;
+
+ @Test
+ public void shouldSearchByNestedAlert() throws Exception {
+List> inputData = new ArrayList<>();
+Map searchByNestedAlert0JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert0, new TypeReference>() {});
+inputData.add(searchByNestedAlert0JSON);
+Map searchByNestedAlert1JSON =
JSONUtils.INSTANCE.load(searchByNestedAlert1, new TypeReference>() {});
+inputData.add(searchByNestedAlert1JSON);
+elasticsearchAdd(inputData, INDEX, SENSOR_NAME);
--- End diff --
Can we either modify this case, or add a new one with multiple alerts?
Basically just to make sure things function as expected when there's multiple
nested alerts.
---
[GitHub] metron pull request #802: METRON-1255: MetaAlert search is not filtering on ...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/802#discussion_r145372604
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -177,15 +178,18 @@ public MetaAlertCreateResponse
createMetaAlert(MetaAlertCreateRequest request)
public SearchResponse search(SearchRequest searchRequest) throws
InvalidSearchException {
// Wrap the query to also get any meta-alerts.
QueryBuilder qb = constantScoreQuery(boolQuery()
-.should(new QueryStringQueryBuilder(searchRequest.getQuery()))
-.should(boolQuery()
-.must(termQuery(MetaAlertDao.STATUS_FIELD,
MetaAlertStatus.ACTIVE.getStatusString()))
-.must(nestedQuery(
+.must(boolQuery()
+.should(new QueryStringQueryBuilder(searchRequest.getQuery()))
+.should(nestedQuery(
ALERT_FIELD,
new QueryStringQueryBuilder(searchRequest.getQuery())
)
)
)
+.must(boolQuery()
--- End diff --
Would you mind adding a comment explaining this? Basically something to the
effect of "Ensure that it's a meta alert with active status or that it's an
alert (signified by having no status)".
---
