date:20171215

[jira] [Created] (METRON-1367) Stellar should optionally instrument function calls

2017-12-15 Thread Otto Fowler (JIRA)

Otto Fowler created METRON-1367:
---

 Summary: Stellar should optionally instrument function calls
 Key: METRON-1367
 URL: https://issues.apache.org/jira/browse/METRON-1367
 Project: Metron
  Issue Type: Improvement
Reporter: Otto Fowler


It would be nice if stellar processor(s) could optionally instrument function 
calls to test performance.

The result would be some structure with the calls, the times and maybe some 
context as a list or map



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1366) Add an entropy stellar function

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293623#comment-16293623
 ] 

ASF GitHub Bot commented on METRON-1366:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/872#discussion_r157332543
  
--- Diff: 
metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/informationtheory/InformationTheoryUtil.java
 ---
@@ -0,0 +1,52 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.statistics.informationtheory;
+
+import java.util.Map;
+
+public enum InformationTheoryUtil {
+  INSTANCE;
+  private static final double LOG2 = Math.log(2);
+
+  public double entropy(Map counts, double logOfBase) {
+double ret = 0.0;
+int n = 0;
+if(counts == null || counts.isEmpty()) {
+  return ret;
+}
+for(Integer f : counts.values()) {
+  n+=f;
+}
+
+for(Integer f : counts.values()) {
+  double p = f.doubleValue()/n;
+  ret -= p * Math.log(p) / logOfBase;
+}
+return ret;
+  }
+
--- End diff --

https://github.com/ottobackwards/stream_entropy


> Add an entropy stellar function
> ---
>
> Key: METRON-1366
> URL: https://issues.apache.org/jira/browse/METRON-1366
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Trending entropy for various volumetric statistics (e.g. netflow data) has 
> been a useful metric for intrusion detection (see 
> http://www.cs.bu.edu/techreports/pdf/2005-002-anomaly-mining.pdf).  To wit, 
> it makes some sense to add an entropy calculation for Stellar.  Used in 
> conjunction with the profiler and an outlier detector, we can recreate the 
> work of Lakhina referenced above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1366) Add an entropy stellar function

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293620#comment-16293620
 ] 

ASF GitHub Bot commented on METRON-1366:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/872#discussion_r157332057
  
--- Diff: 
metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/informationtheory/InformationTheoryUtil.java
 ---
@@ -0,0 +1,52 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.statistics.informationtheory;
+
+import java.util.Map;
+
+public enum InformationTheoryUtil {
+  INSTANCE;
+  private static final double LOG2 = Math.log(2);
+
+  public double entropy(Map counts, double logOfBase) {
+double ret = 0.0;
+int n = 0;
+if(counts == null || counts.isEmpty()) {
+  return ret;
+}
+for(Integer f : counts.values()) {
+  n+=f;
+}
+
+for(Integer f : counts.values()) {
+  double p = f.doubleValue()/n;
+  ret -= p * Math.log(p) / logOfBase;
+}
+return ret;
+  }
+
--- End diff --

for giggles

```java
 public double stream_entropy(Map counts, double logOfBase) {
double ret = 0.0;
if (counts == null || counts.isEmpty()) {
  return ret;
}
final int n = counts.values().stream().mapToInt((d) -> d).sum();
return counts.values().stream().collect(reducing(0.0, (f) -> 
f.doubleValue(), (a, b) -> {
  double p = b.doubleValue() / n;
  a -= p * Math.log(p) / logOfBase;
  return a;
}));
  }
```


> Add an entropy stellar function
> ---
>
> Key: METRON-1366
> URL: https://issues.apache.org/jira/browse/METRON-1366
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Trending entropy for various volumetric statistics (e.g. netflow data) has 
> been a useful metric for intrusion detection (see 
> http://www.cs.bu.edu/techreports/pdf/2005-002-anomaly-mining.pdf).  To wit, 
> it makes some sense to add an entropy calculation for Stellar.  Used in 
> conjunction with the profiler and an outlier detector, we can recreate the 
> work of Lakhina referenced above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1366) Add an entropy stellar function

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293407#comment-16293407
 ] 

ASF GitHub Bot commented on METRON-1366:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/872

METRON-1366: Add an entropy stellar function

## Contributor Comments
Trending entropy for various volumetric statistics (e.g. netflow data) has 
been a useful metric for intrusion detection (see 
http://www.cs.bu.edu/techreports/pdf/2005-002-anomaly-mining.pdf). To wit, it 
makes some sense to add an entropy calculation for Stellar. Used in conjunction 
with the profiler and an outlier detector, we can recreate the work of Lakhina 
referenced above.

You can validate this from the REPL:
* `IT_ENTROPY({ 'a' : 10, 'b' : 5, 'c' : 5} )` should yield `1.5`
  * By the definition of Entropy: `H(X) = -p(a)*log_2(p(a)) - 
p(b)*log_2(p(b)) - p(c)*log_2(p(c)) = -0.5*-1 - 0.25*-2 - 0.25*-2 = 1.5`
* `IT_ENTROPY({})` should yield `0.0`

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron entropy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/872.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #872


commit d4494854e482e2532f707b28ef2f891d13e129f3
Author: cstella 
Date:   2017-12-15T23:25:50Z

Added entropy function.




> Add an entropy stellar function
> ---
>
> Key: METRON-1366
> URL: https://issues.apache.org/jira/browse/METRON-1366
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Trending entropy for various volumetric statistics (e.g. netflow data) has 
> been a useful metric for intrusion detection (see 
> http://www.cs.bu.edu/techreports/pdf/2005-002-anomaly-mining.pdf).  To wit, 
> it makes some sense to add an entropy calculation for Stellar.  Used in 
> conjunction

[jira] [Created] (METRON-1366) Add an entropy stellar function

2017-12-15 Thread Casey Stella (JIRA)

Casey Stella created METRON-1366:


 Summary: Add an entropy stellar function
 Key: METRON-1366
 URL: https://issues.apache.org/jira/browse/METRON-1366
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella


Trending entropy for various volumetric statistics (e.g. netflow data) has been 
a useful metric for intrusion detection (see 
http://www.cs.bu.edu/techreports/pdf/2005-002-anomaly-mining.pdf).  To wit, it 
makes some sense to add an entropy calculation for Stellar.  Used in 
conjunction with the profiler and an outlier detector, we can recreate the work 
of Lakhina referenced above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1365) Allow PROFILE_GET to return a default value for a profile and entity that does not have a value written.

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293382#comment-16293382
 ] 

ASF GitHub Bot commented on METRON-1365:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/871

METRON-1365: Allow PROFILE_GET to return a default value for a profile and 
entity that does not have a value written.

## Contributor Comments
Right now the profiler is a sparse system, namely if data is not written 
during a profile period for a given profile and entity, no values are written 
to HBase. Some algorithms may need non-sparse data (e.g. time series outlier 
detection algorithms). We should keep the default behavior as it stands, but 
allow the user at read time to specify a default value. This is most cleanly 
done by using the existing profiler config overrides (the 4rd argument to 
`PROFILE_GET`) and adding a `profiler.default.value`.

You should be able to validate this in the REPL in full-dev by running:
```
PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(4, 'HOURS'), { 
'profiler.default.value' : 'metron'} )
```

You should see a list of strings with 16 or 17 copies of `metron` in it.
If you run it without the overrides, you should see an empty list returned.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
profiler_default_values

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #871


commit f763a728bce19f10b49c2d431814e25006a5b5d3
Author: cstella 
Date:   2017-12-15T22:37:22Z

Adding the ability to return default values from the profiler.

commit 44267b0850c7fc0912408f36badf64556a30d927
Author: cstella 
Date:   2017-12-15T22:39:14Z

Updating readme to support default values




> Allow PROFILE_GET to return a default value for a profile and entity that 
> does not

[jira] [Created] (METRON-1365) Allow PROFILE_GET to return a default value for a profile and entity that does not have a value written.

2017-12-15 Thread Casey Stella (JIRA)

Casey Stella created METRON-1365:


 Summary: Allow PROFILE_GET to return a default value for a profile 
and entity that does not have a value written.
 Key: METRON-1365
 URL: https://issues.apache.org/jira/browse/METRON-1365
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella


Right now the profiler is a sparse system, namely if data is not written during 
a profile period for a given profile and entity, no values are written to 
HBase.  Some algorithms may need non-sparse data (e.g. time series outlier 
detection algorithms).  We should keep the default behavior as it stands, but 
allow the user at read time to specify a default value.  This is most cleanly 
done by using the existing profiler config overrides (the 3rd argument to 
PROFILE_GET) and adding a profiler.default.value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1364) Add an implementation of Robust PCA outlier detection

2017-12-15 Thread ASF GitHub Bot (JIRA)

[
https://issues.apache.org/jira/browse/METRON-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293289#comment-16293289
]

ASF GitHub Bot commented on METRON-1364:

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/870

METRON-1364: Add an implementation of Robust PCA outlier detection

## Contributor Comments
With short circuiting in Stellar, we have the opportunity to delve into
more computationally intensive outlier detection techniques. Generally these
would be executed only if simpler outlier detection techniques indicated an
outlier (e.g. statistical outlier tests).
As the first one of these supported, I'd suggest a Robust PCA based
technique similar to Netflix's Surus. See
https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc
and
https://metamarkets.com/2012/algorithmic-trendspotting-the-meaning-of-interesting/
for more detail.
It should be noted that there are some caveats with this approach around
sparsity and orderedness.
Regarding sparsity,this outlier detection algorithm presumes dense output,
which is not the case for data spanning profiles (e.g. the profiler does not
write out data every period if no data was seen). To deal with this, I am
suggesting a modification to the profiler to allow PROFILE_GET to return a
default value. That will be done in a separate JIRA.
Regarding well-orderedness, this is an outlier detector for time series
data, so it is sensitive to order to a certain extent. Given its computational
intensity, it is likely to be used with a sample of the data to shrink the size
of the data. To that end, uniform sampling is not sensible here, but rather a
biased sample for recency. Without this, you may get poor results from this
outlier detector. This sampler should be done in a separate JIRA, but I will
ensure the infrastructure to add it is contributed in METRON-1350.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).

- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```

- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:

```
cd site-book
mvn site
```

Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron RAD_outlier

Alternatively you can review and apply these changes as the patch at:

[jira] [Created] (METRON-1364) Add an implementation of Robust PCA outlier detection

2017-12-15 Thread Casey Stella (JIRA)

Casey Stella created METRON-1364:


 Summary: Add an implementation of Robust PCA outlier detection
 Key: METRON-1364
 URL: https://issues.apache.org/jira/browse/METRON-1364
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella


With short circuiting in Stellar, we have the opportunity to delve into more 
computationally intensive outlier detection techniques.  Generally these would 
be executed only if simpler outlier detection techniques indicated an outlier 
(e.g. statistical outlier tests).

As the first one of these supported, I'd suggest a Robust PCA based technique 
similar to Netflix's Surus.  See 
https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc
 and 
https://metamarkets.com/2012/algorithmic-trendspotting-the-meaning-of-interesting/
 for more detail.

It should be noted that there are some caveats with this approach around 
sparsity and orderedness.  

Regarding sparsity,this outlier detection algorithm presumes dense output, 
which is not the case for data spanning profiles (e.g. the profiler does not 
write out data every period if no data was seen). To deal with this, I am 
suggesting a modification to the profiler to allow PROFILE_GET to return a 
default value.  That will be done in a separate JIRA.

Regarding well-orderedness, this is an outlier detector for time series data, 
so it is sensitive to order to a certain extent.  Given its computational 
intensity, it is likely to be used with a sample of the data to shrink the size 
of the data.  To that end, uniform sampling is not sensible here, but rather a 
biased sample for recency.  Without this, you may get poor results from this 
outlier detector.  This sampler should be done in a separate JIRA, but I will 
ensure the infrastructure to add it is contributed in METRON-1350.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293278#comment-16293278
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/865#discussion_r157303681
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -20,7 +20,9 @@ Apache Metron Bundles Maven Plugin helps to build Bundles 
Archives to support th
 
 - [Requirements](#requirements)
 - [Building](#building)
-- [Getting Stared](#getting_started)
+- [Getting Stared](#getting-started)
--- End diff --

`s/Stared/Started/`


> Bundles and Maven Plugin
> 
>
> Key: METRON-1212
> URL: https://issues.apache.org/jira/browse/METRON-1212
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>  Labels: metron-feature-canidate, 
> metron-feature-extensions-parsers
>
> The first effort will be to land the bundle system and supporting maven 
> plugin on master



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1158) Build backend for grouping alerts into meta alerts

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1158:

Fix Version/s: 0.4.2

> Build backend for grouping alerts into meta alerts
> --
>
> Key: METRON-1158
> URL: https://issues.apache.org/jira/browse/METRON-1158
> Project: Metron
>  Issue Type: New Feature
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> We should be able to handle meta alerts (manually grouped alerts, 
> particularly from the UI) in the system. This should be integrated with the 
> DAO composition put into place with IndexDao.
> While similar to faceting (and likely resulting from slicing and dicing from 
> faceting), these need to be interacted with and queryable alongside regular 
> alerts.
> This needs to handle:
> * ES (as a starting point. This shouldn't preclude Solr)
> * Creation of meta alerts
> * Maintain update semantics for alerts
> * Handling scores when a child alert is added, updated or deleted.
> * Continue to allow regular alerts to flow through cleanly to the original 
> sensor indices.
> * Allow for querying (plus sorting and so on) alongside the original sensor 
> indices.
> * Maintain UI grouping order as the minimum of metadata.
> * This should be configured via the same indexDao definition (or at least 
> similar) as the other indexDaos.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1190) Fix Meta Alert Type handling in calculation of scores

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1190:

Fix Version/s: 0.4.2

> Fix Meta Alert Type handling in calculation of scores
> -
>
> Key: METRON-1190
> URL: https://issues.apache.org/jira/browse/METRON-1190
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> The alerts field can actually come as two different types List vs 
> Object[] during calculation of meta score fields.  A "correction" during the 
> original PR to fix one actually broke the other.  Both should be able to be 
> handled (Contents should be the same, just wrapped differently.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-938) "service metron-rest start " does not work on CentOS 7.

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-938:
---
Fix Version/s: (was: Next + 1)
   0.4.2

> "service metron-rest start " does not work on CentOS 7.
> -
>
> Key: METRON-938
> URL: https://issues.apache.org/jira/browse/METRON-938
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
> Environment: Metron 0.4.0 on CentOS 7.
>Reporter: Laurens Vets
>Assignee: Justin Leet
>Priority: Minor
> Fix For: 0.4.2
>
>
> I suppose due to the change to systemd, the command "service metron-rest 
> start " does not work anymore in CentOS 7.
> Changing "METRON_JDBC_PASSWORD="$2"" to "METRON_JDBC_PASSWORD=" PASSWORD>"" in "/etc/rc.d/init.d/metron-rest" seems to fix this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1220) Create documentation around alert nested field

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1220:

Fix Version/s: 0.4.2

> Create documentation around alert nested field
> --
>
> Key: METRON-1220
> URL: https://issues.apache.org/jira/browse/METRON-1220
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> There is an additional requirement around defining a sensor's fields in 
> Elasticsearch as a result of the meta alert functionality resulting from 
> Elasticsearch 2.x. This should be added to to the appropriate READMEs as 
> needed.
> Specifically the sensors must all define a field as such:
> {code}
> "alert": {
>   "type": "nested"
> }
> {code}
> This field is a dummy field used exclusively to make searches work (because 
> an error will be thrown when a field is missing).
> {code}
> QueryParsingException[[nested] failed to find nested object under path 
> [alert]];
> {code}
> As noted https://github.com/apache/metron/pull/619#issuecomment-330529609, 
> this underlying ES behavior is changed with ES 5.x and a small modification 
> to the query can be made and this field removed. This is documentd by ES at 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1202) ElasticsearchDao Has extraneous sleep call

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1202:

Fix Version/s: (was: Next + 1)
   0.4.2

> ElasticsearchDao Has extraneous sleep call
> --
>
> Key: METRON-1202
> URL: https://issues.apache.org/jira/browse/METRON-1202
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> Pointless sleep call that snuck in.  Just dump it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1272) Hide child alerts from searches and grouping if they belong to meta alerts

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1272:

Fix Version/s: 0.4.2

> Hide child alerts from searches and grouping if they belong to meta alerts
> --
>
> Key: METRON-1272
> URL: https://issues.apache.org/jira/browse/METRON-1272
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> If an alert is already grouped into a meta alert, it's nice to route 
> everything through the same query structure and allow sorting alongside them, 
> etc.  However, showing alerts that are already contained in a meta alert is 
> potential clutter for a user and gives the impression an event has occurred 
> twice if it's in a standalone alert and a metaalert.
> This should hide alerts contained in a meta alert from searches (which will 
> always match the enclosing meta alert anyway, so nothing will be lost from 
> the search).
> They should also be hidden from grouping calls, because the user has already 
> manually grouped them during prior slicing and dicing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1247) REST search and findOne endpoints return unexpected or incorrect results for guids

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1247:

Fix Version/s: 0.4.2

> REST search and findOne endpoints return unexpected or incorrect results for 
> guids
> --
>
> Key: METRON-1247
> URL: https://issues.apache.org/jira/browse/METRON-1247
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.4.2
>
>
> When running a search for a particular GUID, e.g.
> {code}
> {
>   "indices": ["websphere", "snort", "asa", "bro", "yaf"],
>   "query": "guid:a2d7ba04-d335-4948-8bc8-c889e7c531c9",
>   "from": 0,
>   "size": 500
> }
> {code}
> Multiple results can be returned.  This appears to be because the "guid" 
> field gets the default analyzer, so the individual segments between the 
> hyphens can match.
> A workaround is simply to quote the string (escaping appropriately).  This 
> should be documented.  A longer term solution is likely needed, but requires 
> more discussion.
> For findOne, the same situation happens, but since only one item is returned 
> it can be incorrect (a partial match that happens to be sorted earlier).  
> This should be changed to a simple GET against the id (which is also the guid 
> by design).  It's also more efficient to do this anyway.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (METRON-1284) Remove extraneous dead query in ElasticsearchDao

2017-12-15 Thread Justin Leet (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-1284:

Fix Version/s: 0.4.2

> Remove extraneous dead query in ElasticsearchDao
> 
>
> Key: METRON-1284
> URL: https://issues.apache.org/jira/browse/METRON-1284
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
> Fix For: 0.4.2
>
>
> This is pointless, but not really harmful.
> {code}
>  org.elasticsearch.action.search.SearchResponse result = 
> client.prepareSearch("test*").setFetchSource(true).setQuery(QueryBuilders.matchAllQuery()).get();
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292889#comment-16292889
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Just a status update on this. We're currently waiting for 0.4.2 to roll out 
before this gets committed. We definitely want more eyes and testing on this PR 
considering its breadth and size. We do not have any +1's yet, and I would 
prefer to have at least 2 for good measure, if at all possible.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

59 matches

Mail list logo