[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/841
Hi @nickwallen
> I am not sure exactly what the problem is, but the same condition occurs
in master. I would call this a pre-existing condition that we can handle with a
separate PR.
>
> Of course that is just my opinion and you or others may feel differently.
Let me know what you think.
>
Sure, I am fine by it, since I did not see any anomalies due to this error.
Also, I was trying to test the same with CentOS 7.4 image and ran into some
challenges. I am constantly facing a 'Timed out' issue when attempting 'vagrant
up' on my local system.
```
â centos-7.4 git:(master) â vagrant up
Bringing machine 'source' up with 'virtualbox' provider...
Bringing machine 'sink' up with 'virtualbox' provider...
==> source: Box 'bento/centos-7.4' could not be found. Attempting to find
and install...
source: Box Provider: virtualbox
source: Box Version: >= 0
==> source: Loading metadata for box 'bento/centos-7.4'
source: URL: https://atlas.hashicorp.com/bento/centos-7.4
==> source: Adding box 'bento/centos-7.4' (v201710.25.0) for provider:
virtualbox
source: Downloading:
https://vagrantcloud.com/bento/boxes/centos-7.4/versions/201710.25.0/providers/virtualbox.box
==> source: Box download is resuming from prior download progress
==> source: Successfully added box 'bento/centos-7.4' (v201710.25.0) for
'virtualbox'!
==> source: Importing base box 'bento/centos-7.4'...
==> source: Matching MAC address for NAT networking...
==> source: Checking if box 'bento/centos-7.4' is up to date...
==> source: Setting the name of the VM: centos-74_source_1511196258710_55009
==> source: Clearing any previously set network interfaces...
==> source: Preparing network interfaces based on configuration...
source: Adapter 1: nat
source: Adapter 2: hostonly
==> source: Forwarding ports...
source: 22 (guest) => (host) (adapter 1)
==> source: Running 'pre-boot' VM customizations...
==> source: Booting VM...
==> source: Waiting for machine to boot. This may take a few minutes...
source: SSH address: 127.0.0.1:
source: SSH username: vagrant
source: SSH auth method: private key
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.
If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
â centos-7.4 git:(master) â vagrant status
Current machine states:
sourcerunning (virtualbox)
sink not created (virtualbox)
```
I did a fresh import of the 'bento/centos-7.4' box image multiple times
after cleaning up (`vagrant box remove bento/centos-7.4` followed by fresh
import), but I ran into the same timeout issue. Also tried bumping up
`config.vm.boot_timeout` in `Vagrantfile` to 600, but to no avail.
Further I am also not able to vagrant ssh into the 'source' VM despite it
showing as running.
Does it look like I missed something in my configuration?
---
[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/814 Bump? ---
[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/841 +1, ship it ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 +1 - wtg ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 Okay, so I spun up master, pushed my template via `curl`, and then ran the above commands to confirm backward compatibility with the template on bro 2.4.x. The only change to my steps that I had to make was to remove the new `RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG` logs from `logs_to_send`, but that was expected and I encountered no other issues. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/803
I made a metaalert with four entries
```
957f20a3-d67b-407a-a593-09bdcbca19df
b18e0949-9ac5-48e2-945f-74f9609667db
8fb8f6cf-861f-4337-8d34-1becc9cecad9
0c6543c8-c5b3-4540-ba83-b338b1aa52f0
```
When I delete an alert, e.g. `0c6543c8-c5b3-4540-ba83-b338b1aa52f0`, the
wrong alert is removed (in this case `957f20a3-d67b-407a-a593-09bdcbca19df`).
The wrong alert is passed
```
{
"metaAlertGuid": "f5cd050c-7a7d-4562-bfc1-1ca5796fa1e4",
"alerts": [
{
"guid": "957f20a3-d67b-407a-a593-09bdcbca19df",
"sensorType": "snort",
"index": ""
}
]
}
```
It appears to always submit a request to remove the first alert, not the
alert I choose in the UI (although I haven't extensively validated that)
---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/845 ---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/845 +1 ---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/845 +1 by inspection, assuming @ottobackwards is good. Thanks for expanding the comments out, it's definitely helpful. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/803 @justinleet my bad. The search query to fetch all the alerts in a group was returning a nested object since I was passing 'source: type' twice in the fields. I don't know why I was getting a nested object if I pass the same column name twice but for now, I fixed it in the UI. The test spec for the above issue is in flight... ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/803 @justinleet my bad. The search query to fetch all the alerts in a group was returning a nested object since I was passing 'source: type' twice in the fields. I don't know why I was getting a nested object if I pass the same column name twice but for now, I fixed it in the UI. ---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/845 I ran this up according to my testing instructions and it addresses the problem. Please take a look-see. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 Leaving this open a bit longer so @nickwallen has time to comment, if interested. ---
[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152085762
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
module Kafka;
export {
+ ## Specify which :bro:type:`Log::ID` to exclude from being sent to
kafka.
##
- ## which log streams should be sent to kafka?
- ## example:
- ## redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG,
DNS::LOG);
+ ## Example: redef Kafka::logs_to_exclude = set(SSH::LOG);
+ const logs_to_exclude: set[Log::ID] &redef;
+
+ ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+ ## Example: redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] &redef;
}
event bro_init() &priority=-5
{
for (stream_id in Log::active_streams)
{
- if (stream_id in Kafka::logs_to_send)
- {
- local filter: Log::Filter = [
- $name = fmt("kafka-%s", stream_id),
- $writer = Log::WRITER_KAFKAWRITER,
- $config = table(["stream_id"] = fmt("%s",
stream_id))
- ];
+ if ( stream_id in Kafka::logs_to_exclude ||
+ (|Kafka::logs_to_send| > 0 && stream_id !in
Kafka::logs_to_send) )
--- End diff --
Actually, wait, sorry. If `|Kafka::logs_to_send| > 0` is removed, this
doesn't send when `logs_to_send` is unset. Re-adding this.
---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152083195
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
I think it is really just a matter of what we'd expect a user to define as
scores for their threat triage rules. Are they really going to define values
greater than a 32-bit float? Or values that sum to greater than a 32-bit float?
I think it is a good point to discuss, Otto. I'd suggest we go with the
float approach now, as it minimizes the scope of change in this PR. But we can
revisit whether a double should be used after we migrate to ES 5.x.
---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152081656
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
I would not hold up the PR for this point if it is too much. Just seems
that we are coding around something else.
---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 [METRON-1322](https://issues.apache.org/jira/browse/METRON-1322) for your PCAP feature request. Also, I totally agree with your documentation notes. Cleaning this up has been on my to-do list for a while. ---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152077236
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
The `ElasticsearchMetaAlertDao` adds an overall threat score to the
Metaalert. The overall threat score can be any one of the following summary
aggregations of the child alerts; sum, min, max, count, average, or median.
These summary values are calculated in `MetaScores` and result in Double
values. Since the other sensor indices currently define `threat:triage:score`
as a float, this solution just casts this to a Double to match those.
I think an alternative way to solve this is to just make the
`threat:triage:score` in each of the sensor indices a Double as you mentioned.
I think your approach seems a little cleaner to me. Although, I am not sure if
there are other down sides I am not thinking about. Can anyone else think of a
problem with this approach? @justinleet @merrimanr ?
---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/845 Could we also add the threat score to the metaalert template, to match the other templates? ---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152075990
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
The calculations were done as Double and given to ES. However, there's no
definition of the field in ES (It just used automatic mapping), so it was given
the ES double.
---
[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r152075056
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
module Kafka;
export {
+ ## Specify which :bro:type:`Log::ID` to exclude from being sent to
kafka.
##
- ## which log streams should be sent to kafka?
- ## example:
- ## redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG,
DNS::LOG);
+ ## Example: redef Kafka::logs_to_exclude = set(SSH::LOG);
+ const logs_to_exclude: set[Log::ID] &redef;
+
+ ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+ ## Example: redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] &redef;
}
event bro_init() &priority=-5
{
for (stream_id in Log::active_streams)
{
- if (stream_id in Kafka::logs_to_send)
- {
- local filter: Log::Filter = [
- $name = fmt("kafka-%s", stream_id),
- $writer = Log::WRITER_KAFKAWRITER,
- $config = table(["stream_id"] = fmt("%s",
stream_id))
- ];
+ if ( stream_id in Kafka::logs_to_exclude ||
+ (|Kafka::logs_to_send| > 0 && stream_id !in
Kafka::logs_to_send) )
--- End diff --
Yeah, that's valid, I have removed the check and simplify.
Yeah, I would prefer a default 'send everything' policy when someone loads
the package, as long as it's otherwise configured. That said, it will require
a bit of Metron testing to make sure that it can handle that. We don't
currently handle some of the less interesting logs that are on by default, like
packet filter or loaded scripts.
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/803 I've verified the bug reported by Justin happens when you create a meta alert from a group that is nested by more than 1 level. Creating a meta alert from a top level group works. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 Documentation that cannot be found doesn't exist, people who aren't devs aren't going to look in the deployment code ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 That is fine. We should surface them regardless at some point. Burying them in the deployment is not ideal. How they index is an important part of any parser's base functionality. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 Thanks @ottobackwards While I feel like the ES template documentation is good enough for now, I really want to investigate something cleaner, probably via 777 but also potentially by splitting up indexes as discussed in [METRON-1010](https://issues.apache.org/jira/browse/METRON-1010?filter=-2). ---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/845#discussion_r152069979
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java
---
@@ -614,8 +625,15 @@ protected void calculateMetaScores(Document metaAlert)
{
}
metaScores = new MetaScores(scores);
}
+
+// add a summary (max, min, avg, count, sum) of all the threat scores
from the child alerts
metaAlert.getDocument().putAll(metaScores.getMetaScores());
-metaAlert.getDocument().put(threatTriageField,
metaScores.getMetaScores().get(threatSort));
+
+// the overall threat score for the metaalert; either max, min, avg,
count or sum of all child scores
--- End diff --
why isn't it a float to start with? isn't that the real issue?
---
[GitHub] metron issue #845: METRON-1321 Metaalert Threat Score Type Does Not Match Se...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/845 I am still testing this in Full Dev. Will respond once I verify this completely. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 I am +1 pending travis. Reviewed code Ran build and tests ( after fix*) Followed test instructions. Great work @JonZeolla ---
[GitHub] metron pull request #845: METRON-1321 Metaalert Threat Score Type Does Not M...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/845 METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices After creating Metaalerts in the Alerts UI, I am unable to sort by threat triage score. The exception that is logged is shown in the issue. When opening the Developer view in Chrome, I can see the API responds with an error. An error message is logged under `/var/log/metron/metron-rest.log`. ## What happened? An overall threat score is added to a Metaalert whenever it has any child alerts. This overall threat score is a summary of the threat scores from each of the child alerts. This field is also named the same as the threat score fields from the sensor indices; 'threat:triage:score'. The type of this field must be the same as the 'threat:triage:score' fields from each of the sensor indices. Otherwise the Alerts UI cannot properly sort alerts and metaalerts in the same table/view in the UI. ## Changes I updated the `ElasticsearchMetaalertDao` so that when it creates the overall threat score, it is added as a Float to match the type expected from the other sensor indices. ## Testing 1. Spin up Full Dev 2. Open up the Alerts UI 3. Sort by triage score and assure no errors occur 4. Create a Metaalert. 5. Sort by triage score and assure no errors occur ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1321 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/845.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #845 commit c780058aad1611edb0ea7a4fae84038ee07d50fd Author: Nick Allen Date: 2017-11-20T18:05:37Z METRON-1321 Metaalert Threat Score Type Does Not Match Sensor Indices ---
[GitHub] metron pull request #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/844#discussion_r152067166
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
---
@@ -1133,6 +1133,233 @@ public void testKnownDevicesBroMessage() throws
ParseException {
}
/**
--- End diff --
Fixed with my latest commit - thanks.
---
[GitHub] metron pull request #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/844#discussion_r152052808
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
---
@@ -1133,6 +1133,233 @@ public void testKnownDevicesBroMessage() throws
ParseException {
}
/**
--- End diff --
This extra /** is why the tests are failing
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/803
That's weird @justinleet . The create request is working for me. I'll
mess with it some more and try to replicate what you are seeing.
I am seeing a separate issue on the REST UI side.
```
17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
mismatch: snort_index_2017.11.20.16.threat:triage:score has type float while
metaalert_index.threat:triage:score has type double. Defaulting type to other.
17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type
mismatch: bro_index_2017.11.20.16.id has type string while
snort_index_2017.11.20.16.id has type integer. Defaulting type to other.
17/11/20 16:55:40 ERROR dao.ElasticsearchRequestSubmitter: Failed to
execute search; error='NotSerializableExceptionWrapper: class_cast_exception:
java.lang.Double cannot be cast to java.lang.Float',
search='{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"]}}],"must_not":{"exists":{"field":"metaalerts"}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichm
ents:geo:ip_dst_addr:country"'
Failed to execute phase [query], [reduce] ; shardFailures
{[0KqVPgyOT2KKCjTKZYIl3Q][bro_index_2017.11.20.16][0]:
RemoteTransportException[[node1][192.168.66.121:9300][indices:data/read/search[phase/query]]];
nested: SearchParseException[failed to parse search source
[{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"]}}],"must_not":{"exists":{"field":"metaalerts"}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichmen
ts:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"]];
nested: IllegalArgumentException[No mapper found for type [other]]; }
at
org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:176)
at
org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:147)
at
org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:144)
at
org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:41)
at
org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:819)
at
org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:803)
at
org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:793)
at
org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
at
org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:369)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at
org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecuto
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 The documentation in the template is nice. After 777, when each parser has a readme, this documentation of the index fields should be in the bro readme. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 Is there some way to document these PCAPS? Could we have a script that does what you do here just checked in? I think this would be useful. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 Ran tests as described, everything worked according to steps. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/803
@iraghumitra looks like the new API isn't being used quite right.
Sample from the dev tools
```
{
"alerts": [
{
"guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
"sensorType": [
"snort"
],
"index": "snort_index_2017.11.20.15"
},
...
```
This should be, I believe,
```
{
"alerts": [
{
"guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
"sensorType": "snort",
"index": "snort_index_2017.11.20.15"
},
...
],
"groups": [
"source:type"
]
}
```
---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/803 Merged the PR with master and used new API's for creating meta-alerts. Please feel free to review and let me know the feedback. ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 @iraghumitra I see that you merged some changes. Is this ready to test? ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 There is no requirement to upgrade bro with this change. All old fields and logs are still supported, this simply adds support for the new fields in existing logs or new logs altogether to be supported. You may notice that I removed capture password, as it is an internal field and never exposed to logs. It should not have been in there in the first place. ---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/844 Are there any consequences for users with external, existing and older bro installations? Will they have to upgrade bro if they take this build? ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user iraghumitra commented on a diff in the pull request:
https://github.com/apache/metron/pull/803#discussion_r151964471
--- Diff:
metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss
---
@@ -143,3 +167,12 @@ textarea {
.comment-container:hover i {
display: block;
}
+
+.input-group-addon {
+ cursor: pointer;
+}
+
+.disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+}
--- End diff --
Added newline
---
[GitHub] metron issue #844: METRON-1088: Upgrade bro to 2.5.2
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/844 I'm going to see if I can find some time today to fix the tests, but this is ready for review otherwise. Full-dev worked as expected for me. ---
