[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Bump... checking to see if there is any other change required for this pull 
request. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #429: METRON-660 [Umbrella] up-to-date versioned docu...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/429
  
+1, this is great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/316#discussion_r99253356
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCliWrapperTest.java
 ---
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service.impl;
+
+import org.apache.metron.rest.MetronRestConstants;
+import org.apache.metron.rest.RestException;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.core.env.Environment;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.anyVararg;
+import static org.mockito.Mockito.verify;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.verifyNew;
+import static org.powermock.api.mockito.PowerMockito.when;
+import static org.powermock.api.mockito.PowerMockito.whenNew;
+
+@SuppressWarnings("unchecked")
+@RunWith(PowerMockRunner.class)
+@PrepareForTest({DockerStormCLIWrapper.class, ProcessBuilder.class})
+public class StormCliWrapperTest {
--- End diff --

I know this is a very minor thing, but I think the name of this class 
should change to `StormCLIWrapperTest` or all other CLI usages should be 
changed to Cli. I'm not sure if we've decided on either one of these being our 
standard though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/316#discussion_r99252325
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/GrokServiceImpl.java
 ---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service.impl;
+
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import org.apache.directory.api.util.Strings;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.GrokValidation;
+import org.apache.metron.rest.service.GrokService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.core.env.Environment;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.core.context.SecurityContextHolder;
+import org.springframework.stereotype.Service;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.StringReader;
+import java.util.Map;
+
+import static 
org.apache.metron.rest.MetronRestConstants.GROK_TEMP_PATH_SPRING_PROPERTY;
+
+@Service
+public class GrokServiceImpl implements GrokService {
+
+private Environment environment;
+
+private Grok commonGrok;
+
+@Autowired
+public GrokServiceImpl(Environment environment, Grok commonGrok) {
+this.environment = environment;
+this.commonGrok = commonGrok;
+}
+
+@Override
+public Map getCommonGrokPatterns() {
+return commonGrok.getPatterns();
+}
+
+@Override
+public GrokValidation validateGrokStatement(GrokValidation 
grokValidation) throws RestException {
+Map results;
+try {
+String statement = 
Strings.isEmpty(grokValidation.getStatement()) ? "" : 
grokValidation.getStatement();
+
+Grok grok = new Grok();
+grok.addPatternFromReader(new 
InputStreamReader(getClass().getResourceAsStream("/patterns/common")));
+grok.addPatternFromReader(new StringReader(statement));
+String patternLabel = statement.substring(0, 
statement.indexOf(" "));
+String grokPattern = "%{" + patternLabel + "}";
+grok.compile(grokPattern);
+Match gm = grok.match(grokValidation.getSampleData());
+gm.captures();
+results = gm.toMap();
+results.remove(patternLabel);
+} catch (StringIndexOutOfBoundsException e) {
+throw new RestException("A pattern label must be included (eg. 
PATTERN_LABEL %{PATTERN:field} ...)", e.getCause());
+} catch (Exception e) {
+throw new RestException(e);
+}
+grokValidation.setResults(results);
+return grokValidation;
+}
+
+@Override
+public File saveTemporary(String statement, String name) throws 
RestException {
+if (statement != null) {
+try {
+File grokDirectory = new File(getTemporaryGrokRootPath());
+if (!grokDirectory.exists()) {
+  grokDirectory.mkdirs();
+}
+File path = new File(grokDirectory, name);
+FileWriter fileWriter = new FileWriter(new 
File(grokDirectory, name));
--- End diff --

There may be an issue where the same `name` is passed in causing files to 
be overwritten. I think we should use Java's `File. createTempFile`. Does this 
make sense to do this? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please

Re: [DISCUSS] Next Release (0.3.1) Content

[Matt Foley]

Thanks, Casey.  Btw, I’m not far enough along on METRON-322 (batch timeout 
flush), so it will have to wait for next cycle.
--Matt

On 2/2/17, 2:03 PM, "Casey Stella"  wrote:

Just a quick release update.  As of now, we are waiting on

   - METRON-660 to get reviewed and make it in
   - METRON-692 to get our upgrade.md completed for this release

Mike Miklavcic, you sent an email to the legal-discuss about our kraken
dependency and it looked like we didn't have to change, but could you
comment on this thread in the dev list so I know if we need to wait for
METRON-650.

Casey

On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella  wrote:

> Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
> the things that I know about and the things Jon mentioned here.  Please, 
if
> you have knowledge of other breaking/non-compatible changes between the
> 0.3.0 release and master, comment on this PR (https://github.com/apache/
> incubator-metron/pull/437) and I will incorporate them.
>
> On Fri, Jan 27, 2017 at 10:04 AM, zeo...@gmail.com 
> wrote:
>
>> To start I was mostly concerned with having a per-version list of
>> non-backwards-compatible changes, so upgrades that may skip a version of
>> two can look at what all may be impacted.  We should also probably
>> document
>> any sort of upgrade flaws as well, such as METRON-447
>> , METRON-448
>> , etc.  I do think that
>> we should have a more rigorous document, but I wouldn't push that for the
>> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as
>> key
>> (required) components of a 1.0 release.  I'd just like to see the
>> foundation begin to be laid and iterated on.
>>
>> That said, this probably constitutes a mention in the development
>> guidelines
>> > pageId=61332235>
>> once
>> it's in master.
>>
>> Jon
>>
>> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella  wrote:
>>
>> > I should add, you may be thinking something more rigorous and
>> > step-by-step.  If so, you think you might be interested in volunteering
>> to
>> > do a first draft as a PR that we can adjust?
>> >
>> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella 
>> wrote:
>> >
>> > > So, I agree with the Upgrading.md and I was going to submit a PR at
>> least
>> > > to describe the the changes to indexing configurations that I made
>> during
>> > > the 3.0.1 release.
>> > >
>> > >
>> > > On Thu, Jan 26, 2017 at 10:50 PM, zeo...@gmail.com 
>> > > wrote:
>> > >
>> > >> I haven't had a chance to look through the unresolved JIRAs but I 
did
>> > want
>> > >> to mention a few quick things.
>> > >>
>> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
>> > things
>> > >> that was discussed was putting together a method of documenting
>> upgrades
>> > >> from one version to the next.  As one of the first steps toward
>> making
>> > >> that
>> > >> a reasonable process, I think we should assemble more detailed
>> release
>> > >> notes, especially outlining non-backwards compatible changes.  In 
the
>> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
>> we
>> > >> use
>> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
>> I'm
>> > >> open to alternatives.
>> > >>
>> > >> Separately, a *nice to have* would be *METRON-660*, which was
>> discussed
>> > in
>> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give
>> us
>> > >> some
>> > >> cleaner documentation using the existing READMEs.  I'd be happy to
>> help
>> > >> with this one, I'm just not sure what the next steps are, aside from
>> the
>> > >> start that Matt has here
>> > >> .
>> > >>
>> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
>> here
>> > >> *.  If I could
>> get
>> > >> someone else to reproduce the error that I'm seeing I would be happy
>> to
>> > >> pursue additional testing, troubleshooting, etc.  I've seen others
>> > report
>> > >> the same issue
>> > >> > > >> question=search%2Fsearch=relevance=scp_if_ssh>
>> > >> on the HCC boards, so I'm fairly confident that it is not an issue
>> with
>> > my
>> > >> local 

[GitHub] incubator-metron issue #429: METRON-660 [Umbrella] up-to-date versioned docu...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/429
  
@justinleet thanks for the catch.  Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Next Release (0.3.1) Content

[Casey Stella]

Just a quick release update.  As of now, we are waiting on

   - METRON-660 to get reviewed and make it in
   - METRON-692 to get our upgrade.md completed for this release

Mike Miklavcic, you sent an email to the legal-discuss about our kraken
dependency and it looked like we didn't have to change, but could you
comment on this thread in the dev list so I know if we need to wait for
METRON-650.

Casey

On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella  wrote:

> Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
> the things that I know about and the things Jon mentioned here.  Please, if
> you have knowledge of other breaking/non-compatible changes between the
> 0.3.0 release and master, comment on this PR (https://github.com/apache/
> incubator-metron/pull/437) and I will incorporate them.
>
> On Fri, Jan 27, 2017 at 10:04 AM, zeo...@gmail.com 
> wrote:
>
>> To start I was mostly concerned with having a per-version list of
>> non-backwards-compatible changes, so upgrades that may skip a version of
>> two can look at what all may be impacted.  We should also probably
>> document
>> any sort of upgrade flaws as well, such as METRON-447
>> , METRON-448
>> , etc.  I do think that
>> we should have a more rigorous document, but I wouldn't push that for the
>> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as
>> key
>> (required) components of a 1.0 release.  I'd just like to see the
>> foundation begin to be laid and iterated on.
>>
>> That said, this probably constitutes a mention in the development
>> guidelines
>> > pageId=61332235>
>> once
>> it's in master.
>>
>> Jon
>>
>> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella  wrote:
>>
>> > I should add, you may be thinking something more rigorous and
>> > step-by-step.  If so, you think you might be interested in volunteering
>> to
>> > do a first draft as a PR that we can adjust?
>> >
>> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella 
>> wrote:
>> >
>> > > So, I agree with the Upgrading.md and I was going to submit a PR at
>> least
>> > > to describe the the changes to indexing configurations that I made
>> during
>> > > the 3.0.1 release.
>> > >
>> > >
>> > > On Thu, Jan 26, 2017 at 10:50 PM, zeo...@gmail.com 
>> > > wrote:
>> > >
>> > >> I haven't had a chance to look through the unresolved JIRAs but I did
>> > want
>> > >> to mention a few quick things.
>> > >>
>> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
>> > things
>> > >> that was discussed was putting together a method of documenting
>> upgrades
>> > >> from one version to the next.  As one of the first steps toward
>> making
>> > >> that
>> > >> a reasonable process, I think we should assemble more detailed
>> release
>> > >> notes, especially outlining non-backwards compatible changes.  In the
>> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
>> we
>> > >> use
>> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
>> I'm
>> > >> open to alternatives.
>> > >>
>> > >> Separately, a *nice to have* would be *METRON-660*, which was
>> discussed
>> > in
>> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give
>> us
>> > >> some
>> > >> cleaner documentation using the existing READMEs.  I'd be happy to
>> help
>> > >> with this one, I'm just not sure what the next steps are, aside from
>> the
>> > >> start that Matt has here
>> > >> .
>> > >>
>> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
>> here
>> > >> *.  If I could
>> get
>> > >> someone else to reproduce the error that I'm seeing I would be happy
>> to
>> > >> pursue additional testing, troubleshooting, etc.  I've seen others
>> > report
>> > >> the same issue
>> > >> > > >> question=search%2Fsearch=relevance=scp_if_ssh>
>> > >> on the HCC boards, so I'm fairly confident that it is not an issue
>> with
>> > my
>> > >> local environment.
>> > >>
>> > >> Jon
>> > >>
>> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella 
>> > wrote:
>> > >>
>> > >> I took the liberty of adding the pull requests since the new year
>> into
>> > the
>> > >> in-progress list in the previous email.  If you own one of these and
>> do
>> > >> not
>> > >> believe that you can complete the review in the next week or so, let
>> me
>> > >> know and we can remove.  The only one of these that I see as
>> mandatory
>> > is
>> > >> METRON-650 because the reliance on the opensoc github repo might
>> cause
>> > >> issues with the release being acceptable (see discussion by Mike
>> > 

Re: [DISCUSS] Next Release (0.3.1) Content

[Casey Stella]

Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included the
things that I know about and the things Jon mentioned here.  Please, if you
have knowledge of other breaking/non-compatible changes between the 0.3.0
release and master, comment on this PR (
https://github.com/apache/incubator-metron/pull/437) and I will incorporate
them.

On Fri, Jan 27, 2017 at 10:04 AM, zeo...@gmail.com  wrote:

> To start I was mostly concerned with having a per-version list of
> non-backwards-compatible changes, so upgrades that may skip a version of
> two can look at what all may be impacted.  We should also probably document
> any sort of upgrade flaws as well, such as METRON-447
> , METRON-448
> , etc.  I do think that
> we should have a more rigorous document, but I wouldn't push that for the
> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as key
> (required) components of a 1.0 release.  I'd just like to see the
> foundation begin to be laid and iterated on.
>
> That said, this probably constitutes a mention in the development
> guidelines
>  >
> once
> it's in master.
>
> Jon
>
> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella  wrote:
>
> > I should add, you may be thinking something more rigorous and
> > step-by-step.  If so, you think you might be interested in volunteering
> to
> > do a first draft as a PR that we can adjust?
> >
> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella 
> wrote:
> >
> > > So, I agree with the Upgrading.md and I was going to submit a PR at
> least
> > > to describe the the changes to indexing configurations that I made
> during
> > > the 3.0.1 release.
> > >
> > >
> > > On Thu, Jan 26, 2017 at 10:50 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > >> I haven't had a chance to look through the unresolved JIRAs but I did
> > want
> > >> to mention a few quick things.
> > >>
> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
> > things
> > >> that was discussed was putting together a method of documenting
> upgrades
> > >> from one version to the next.  As one of the first steps toward making
> > >> that
> > >> a reasonable process, I think we should assemble more detailed release
> > >> notes, especially outlining non-backwards compatible changes.  In the
> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
> we
> > >> use
> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
> I'm
> > >> open to alternatives.
> > >>
> > >> Separately, a *nice to have* would be *METRON-660*, which was
> discussed
> > in
> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give us
> > >> some
> > >> cleaner documentation using the existing READMEs.  I'd be happy to
> help
> > >> with this one, I'm just not sure what the next steps are, aside from
> the
> > >> start that Matt has here
> > >> .
> > >>
> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
> here
> > >> *.  If I could
> get
> > >> someone else to reproduce the error that I'm seeing I would be happy
> to
> > >> pursue additional testing, troubleshooting, etc.  I've seen others
> > report
> > >> the same issue
> > >>  > >> question=search%2Fsearch=relevance=scp_if_ssh>
> > >> on the HCC boards, so I'm fairly confident that it is not an issue
> with
> > my
> > >> local environment.
> > >>
> > >> Jon
> > >>
> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella 
> > wrote:
> > >>
> > >> I took the liberty of adding the pull requests since the new year into
> > the
> > >> in-progress list in the previous email.  If you own one of these and
> do
> > >> not
> > >> believe that you can complete the review in the next week or so, let
> me
> > >> know and we can remove.  The only one of these that I see as mandatory
> > is
> > >> METRON-650 because the reliance on the opensoc github repo might cause
> > >> issues with the release being acceptable (see discussion by Mike
> > Miklavcic
> > >> with the mentors a couple weeks ago).
> > >>
> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella 
> > wrote:
> > >>
> > >> > Hello Everyone!
> > >> >
> > >> > It's been almost 2 months since the last major release and I think
> it
> > >> > might be time to do a minor release of 0.3.1. The purpose of this
> > email
> > >> is
> > >> > multifold:
> > >> >
> > >> >- to take stock of what we have already committed
> > >> >- figure out what (if anything) is missing to make a release that
> > >> >we're proud of
> > >> >- find volunteers for the missing items
> > >> >
> > >> 

[GitHub] incubator-metron pull request #437: METRON-692: Update Upgrading.md for 0.3....

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/437

METRON-692: Update Upgrading.md for 0.3.0 -> 0.3.1

We need a document to describe the manual upgrade issues from one release 
to the other.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-692

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit c217b6291725fc9226a6690d2d09a72a5fde9b23
Author: cstella 
Date:   2017-02-02T21:55:10Z

Creating upgrade document




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #429: METRON-660 [Umbrella] up-to-date versioned docu...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/429
  
@mattf-horton, thanks for taking the time/effort to migrate the script.  
Tried it out and it works great.

The only thing I'd like to see is the indentation in the Python script made 
consistent.  I think spaces is more standard Python and swapping potentially 
creates issues because of the way Python handles tabs (searching around online 
suggests it's treated as 8 spaces).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #434: METRON-687 Create String Formatting Function fo...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/434
  
nice, quick, to the point and necessary.  +1 by inspection.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #435: METRON-684: Decouple Timestamp calculation from...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/435
  
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).

## Preliminaries
* Set an environment variable to indicate `METRON_HOME`:
`export METRON_HOME=/usr/metron/0.3.0` 

* Create the profiler hbase table
`echo "create 'profiler', 'P'" | hbase shell`

* Open `~/rand_gen.py` and paste the following:
```
#!/usr/bin/python
import random
import sys
import time
def main():
  mu = float(sys.argv[1])
  sigma = float(sys.argv[2])
  freq_s = int(sys.argv[3])
  while True:
out = '{ "value" : ' + str(random.gauss(mu, sigma)) + ' }'
print out
sys.stdout.flush()
time.sleep(freq_s)

if __name__ == '__main__':
  main()
```
This will generate random JSON maps with a numeric field called `value`

* Set the profiler to use 1 minute tick durations:
  * Edit `$METRON_HOME/config/profiler.properties` to adjust the capture 
duration by changing `profiler.period.duration=15` to 
`profiler.period.duration=1`
  * Edit `$METRON_HOME/config/zookeeper/global.json` and add the following 
properties:
```
"profiler.client.period.duration" : "1",
"profiler.client.period.duration.units" : "MINUTES"
```

## Free Up Space on the virtual machine

First, let's free up some headroom on the virtual machine.  If you are 
running this on a
multinode cluster, you would not have to do this.
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print 
$2}');do kill -9 $i;done`
* Kill existing parser topologies via 
   * `storm kill snort`
   * `storm kill bro`
* We won't need the enrichment or indexing topologies for this test, so you 
can kill them via:
   * `storm kill enrichment`
   * `storm kill indexing`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 
$i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 
$i;done`

## Start the profiler
* `$METRON_HOME/bin/start_profiler_topology.sh`

## Test Case

* Set up a profile to accept some synthetic data with a numeric `value` 
field and persist a stats summary of the data
  * Edit `$METRON_HOME/config/zookeeper/profiler.json` and paste in the 
following:
```
{
  "profiles": [
{
  "profile": "stat",
  "foreach": "'global'",
  "onlyif": "true",
  "init" : {
   },
  "update": {
"s": "STATS_ADD(s, value)"
},
  "result": "s"
}
  ]
}
```

* Send some synthetic data directly to the profiler:
`python ~/rand_gen.py 0 1 1 | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
node1:6667 --topic indexing`
* Wait for at least 10 minutes and execute the following via the Stellar 
REPL:
```
# Grab the last 10 minutes worth of timestamps
PROFILE_FIXED( 10, 'MINUTES')
# Looks like 10 were returned, great.  Now, validate that I get 10 profile 
measurements back
PROFILE_GET('stat', 'global', PROFILE_FIXED( 10, 'MINUTES' ) )
# Ok, now look at the mean across the distribution
# STATS_MEAN( STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_FIXED( 10, 
'MINUTES' ) )))
```
For me, the following was the result:
```
Stellar, Go!
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, 
es.date.format=.MM.dd.HH, profiler.client.period.duration=1, 
profiler.client.period.duration.units=MINUTES}
[Stellar]>>> # Grab the last 10 minutes worth of timestamps
[Stellar]>>> PROFILE_FIXED( 10, 'MINUTES')
Functions loaded, you may refer to functions now...
[24767772, 24767773, 24767774, 24767775, 24767776, 2476, 24767778, 
24767779, 24767780, 24767781, 24767782]
[Stellar]>>> # Looks like 10 were returned, great.  Now, validate that I 
get 10 profile measurements back
[Stellar]>>> PROFILE_GET('stat', 'global', PROFILE_FIXED( 10, 'MINUTES' ) )
[org.apache.metron.statistics.OnlineStatisticsProvider@44749031, 
org.apache.metron.statistics.OnlineStatisticsProvider@d2a7fbb9, 
org.apache.metron.statistics.OnlineStatisticsProvider@a217cfd7, 
org.apache.metron.statistics.OnlineStatisticsProvider@c5e42aed, 
org.apache.metron.statistics.OnlineStatisticsProvider@c4f4753d, 
org.apache.metron.statistics.OnlineStatisticsProvider@87a1606a, 
org.apache.metron.statistics.OnlineStatisticsProvider@e1b4c8dc, 
org.apache.metron.statistics.OnlineStatisticsProvider@fdb7b8d8]
[Stellar]>>> # Ok, now look at the mean across the distribution
[Stellar]>>> STATS_MEAN( 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

GitHub user dlyle65535 opened a pull request:

https://github.com/apache/incubator-metron/pull/436

METRON-671: Refactor existing Ansible deployment to use Ambari MPack

**WORK IN PROGRESS - DO NOT MERGE**

This is the first set of changes to enable Ansible installation to use the 
Ambari MPack. It currently works (in my environment) with full-dev using 
sensor-stubs. It will not work with Quick-Dev or EC2 at this time.

It's (well past - sorry) starting to get large, so I wanted to push it out 
for feedback while I'm working issues with the distributed install.

Some points of interest:
- As discussed, components (ES, Kibana, Metron Topologies) that are managed 
by Ambari are removed from Monit
- I had added a notion of 'required configurations' to the blueprint and 
ambari_cluster_state script so blueprint verification would pass.
- I've refactored our Ansible host groupings (see 
inventory/full-dev-platform/hosts). The Metron group is now the host that we 
use as the Metron services master.
- I cleaned up the very minimum number of issues that @mattf-horton found 
in [METRON-609](https://issues.apache.org/jira/browse/METRON-609) required to 
make single-node deployment work with the MPack. It's my intention to work to 
get his changes incorporated on top of the final version of this changeset.
- I removed the Solr role from Ansible. My current intention is to open a 
Jira to add a Solr Indexing component to the MPack.

Immediate next steps:

- [ ] Get the EC2 distributed install working.
- [ ] Adjust readmes and docs where required.
- [ ] Merge in Matt's changes and make required PRs on his branch (if any).
- [ ] Document test plan

I also hope to receive and respond to feedback.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dlyle65535/incubator-metron METRON-671

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #436


commit 01cd4216754861c7eaf58c04472fb2a8125a6f28
Author: David Lyle 
Date:   2017-01-23T20:47:14Z

METRON-671: Refactor existing Ansible deployment to use Ambari MPack




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #435: METRON-684: Decouple Timestamp calculati...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/435

METRON-684: Decouple Timestamp calculation from PROFILE_GET

Currently `PROFILE_GET` only supports a static lookback of a fixed 
duration. As we have more complicated, potentially sparse, lookbacks (e.g. the 
same time slice every tuesday for a month), it would be nice to decouple the 
construction of timestamps from `PROFILE_GET` into its own set of functions.

The decoupling plus the default timestamp function is provided here, which 
is a fixed window starting from now.  This is implemented as `PROFILE_FIXED`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-684

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #435


commit 6d6b0385a23fb2b8512e6cb85425425e73159be1
Author: cstella 
Date:   2017-02-02T17:17:57Z

METRON-684: Decouple Timestamp calculation from PROFILE_GET




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #434: METRON-687 Create String Formatting Func...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/434

METRON-687 Create String Formatting Function for Stellar

Created a string formatting function for Stellar. 
* Uses Java's String formatting conventions.  
* Any non-existent variables will be formatted as 'null'.  This failure 
scenario would arise from a data quality issue.
* Forgetting to pass an argument required by the format string will throw 
an exception.  
* Not passing a format string will throw an exception.

 FORMAT
* Description: Returns a formatted string using the specified format string 
and arguments.  Uses Java's string formatting conventions.
* Input:
* format - string
* arguments... - object(s)
* Returns: A formatted string.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-687

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #434


commit 5b1fad7ecc73f2e7a44ba644812eb1f008e82853
Author: Nick Allen 
Date:   2017-02-02T16:42:13Z

METRON-687 Create String Formatting Function for Stellar




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

[Casey Stella]

Ok, so how I'm going to manifest this discussion into reality is construct
3 JIRAs:

   - METRON-684: Decouple Timestamp calculation from PROFILE_GET
  - This enables pluggable timestamp computation from other functions
   - METRON-689: Create Cron-syntax based timestamp lookup for profiler to
   enable sparse windows
   - METRON-690: Create a DSL-based timestamp lookup for profiler to enable
   sparse windows

I'm starting 684 now and will probably pick 690 afterwards.  If someone
wants to do 689, I think it'd be great to have different approaches.

Thanks for the feedback.

On Tue, Jan 31, 2017 at 2:25 PM, Casey Stella  wrote:

> +1 to that.  Here here! :)
>
> On Tue, Jan 31, 2017 at 2:21 PM, Nick Allen  wrote:
>
>> Yes, agreed.  However we specify the schedule, it would be decoupled from
>> the Profiler client functions.
>>
>> timestamps := CRON("* * * ? * foo bar")
>> profiles := PROFILE_GET("profile1", "entity1", timestamps)
>>
>>
>> Or
>>
>> timestamps := STELLAR_DSL("on every other Tuesday")
>> profiles := PROFILE_GET("profile1", "entity1", timestamps)
>>
>>
>>
>>
>> On Tue, Jan 31, 2017 at 2:01 PM, Casey Stella  wrote:
>>
>> > One more point, one of the reasons for decoupling the PROFILE_GET from
>> > PROFILE_LOOKUP means that we could ahve alternative implementations of
>> > PROFILE_LOOKUP.  We could have a PROFILE_LOOKUP_CRON as well.
>> >
>> > On Tue, Jan 31, 2017 at 1:43 PM, Casey Stella 
>> wrote:
>> >
>> > > Regarding the "?" syntax:
>> > > Wouldn't that be forking cron syntax so now we have a metron cron?  If
>> > > we're constructing our own syntax, then why not do it so that it reads
>> > like
>> > > natural language?
>> > >
>> > > Regarding the holiday problem:
>> > > Agreed, it's a smaller problem than constructing a DSL, but that's not
>> > > really the point, I think. The concern is that it would be unable to
>> be
>> > > expressed using cron syntax in a natural way without modifying cron
>> > syntax,
>> > > which would be constructing a new DSL.  If quartz has a clever way of
>> > doing
>> > > that, then I'd like to see it.  From a quick search, I haven't seen a
>> > > scheduling example with a compact syntax that shows skipping holidays
>> > with
>> > > cron syntax.
>> > >
>> > >
>> > > On Tue, Jan 31, 2017 at 1:29 PM, Nick Allen 
>> wrote:
>> > >
>> > >> >
>> > >> >- Cron syntax allows you to construct only absolute lookbacks
>> (i.e.
>> > >> >"every tuesday at 3PM" not "every tuesday at the current hour")
>> > >>
>> > >>
>> > >> I think Cron would work for this.  I am no expert on cron
>> expressions,
>> > but
>> > >> I think the following examples would work.
>> > >>
>> > >>- If you want "every Tuesday at 3 PM"
>> > >>   - 0 0 15 ? * TUE *
>> > >>- If you want "every Tuesday at current hour" then use something
>> like
>> > >>the "?" placeholder maybe.
>> > >>   - 0 0 ? ? * TUE *
>> > >>
>> > >> - Cron syntax allows you to specify a point in time, not a
>> duration.  We
>> > >> >could, of course, specify a duration as another argument
>> > >>
>> > >>
>> > >> Yes, a separate argument would be necessary.  We would have to allow
>> the
>> > >> user to specify either a "start from date/time" or the "number of
>> > >> intervals
>> > >> to look back".
>> > >>
>> > >> Cron syntax does not allow you to skip things like holidays, etc.
>> > >>
>> > >>
>> > >> I agree, out-of-the-box Cron does not solve holiday calendars.  But
>> this
>> > >> would be a smaller problem to solve then creating our own DSL.
>> > >>
>> > >> There is a tradition of creating shortcuts that look something like
>> > @Daily
>> > >> or @Weekdays or @Tuesdays that we could also use to make things
>> easier
>> > for
>> > >> users.
>> > >>
>> > >> I have used Quartz with cron expressions in the past and there was
>> some
>> > >> way
>> > >> to handle holidays with that.  I think you could create a custom
>> > calendar
>> > >> for the holidays and call it something; aka @USHolidays.  And then
>> you
>> > >> would say "every Tuesday" except @USHolidays or something like that.
>> > I'd
>> > >> have to look into this some more.
>> > >>
>> > >> And there are also nice online Cron expression "translators" that we
>> > could
>> > >> mimic in a Metron user interface.  For example, https://crontab.guru
>> .
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tue, Jan 31, 2017 at 12:00 PM, Casey Stella 
>> > >> wrote:
>> > >>
>> > >> > I actually did consider cron initially but dismissed it for the
>> > >> following
>> > >> > reasons:
>> > >> >
>> > >> >- Cron syntax allows you to construct only absolute lookbacks
>> (i.e.
>> > >> >"every tuesday at 3PM" not "every tuesday at the current hour")
>> > >> >- Cron syntax allows you to specify a point in time, not a
>> > >> duration.  We
>> > >> >could, of course, specify a duration as another argument
>> > >> >- Cron 

Re: [Discuss] Improve Alerting

[Nick Allen]

I created 3 separate JIRAs to track the "Threat Triage Transparency"
portion of the work falling out of this discussion thread.  The first would
create a mechanism to do string interpolation.  The second would enhance
threat triage to use the string interpolation.  The third would enhance the
output of threat triage.

[1] Create String Formatting Function for Stellar
https://issues.apache.org/jira/browse/METRON-687

[2] Allow Threat Triage Comment Field to Contain Stellar Expressions
https://issues.apache.org/jira/browse/METRON-688

[3] Record of Rule Set that Fired During Threat Triage
https://issues.apache.org/jira/browse/METRON-686

Please let me know if anyone's concerns were not captured.  I will create
additional JIRAs for the other portion of the effort (*Triage Calculated
Values from the Profiler)* once I've given everyone a little more time to
voice an opinion.
​

On Thu, Feb 2, 2017 at 9:46 AM, Nick Allen  wrote:

> Oh, I see.  Yes, very useful.
>
>
> On Thu, Feb 2, 2017 at 9:39 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> That’s a part of it, certainly (and fixes another of my bug bears, so
>> thank you!)
>>
>> In addition to the aggregation being stellar, I want score to be a
>> stellar statement, I’ve put in a separate ticket for that.
>> https://issues.apache.org/jira/browse/METRON-685 <
>> https://issues.apache.org/jira/browse/METRON-685>
>>
>> Simon
>>
>> > On 2 Feb 2017, at 14:31, Nick Allen  wrote:
>> >
>> >> I would much rather be able to say something like score = some stellar
>> >> statement that returns a float...
>> >
>> >
>> > Completely agree.  FYI - We added METRON-683 yesterday that I believe
>> > supports what you are saying.  Feel free to add commentary.
>> >
>> > https://issues.apache.org/jira/browse/METRON-683
>> >
>> > On Thu, Feb 2, 2017 at 9:02 AM, Simon Elliston Ball <
>> > si...@simonellistonball.com> wrote:
>> >
>> >> I completely agree with Nick’s transparency comments, and like the
>> design
>> >> of the configuration, especially provision for messaging around the
>> nature
>> >> of the rule fired.
>> >>
>> >> I would just like to add a small point on the capabilities here. If the
>> >> message could have embedded values through some sort of template for a
>> >> stellar statement, it would make for a better more dynamic alert
>> reason.
>> >>
>> >> I would also like to see the score field capable of outputting the
>> value
>> >> of a stellar statement. At the moment the idea of a static score being
>> >> passed on means that if I have a probabilistic result I want to combine
>> >> with other triage sources, I have to do a lot of bucketing into fixed
>> >> values. I would much rather be able to say something like score = some
>> >> stellar statement that returns a float, ‘alertness' = threshold of
>> this.
>> >> That way I can combine multiple triage rules to trigger an overall
>> alert,
>> >> making the aggregators more meaningful.
>> >>
>> >> Simon
>> >>
>> >>
>> >>> On 2 Feb 2017, at 12:40, Carolyn Duby  wrote:
>> >>>
>> >>> For profiler alerts it will be helpful during analysis to see the
>> alerts
>> >> that caused the anomaly.  The meta alert is useful for incidents
>> involving
>> >> correlation of multiple events.
>> >>>
>> >>> Also you will need to filter out known hosts that trigger anomalies.
>> >> For example vulnerability scanning software.
>> >>>
>> >>> One final thing to consider is anomalies happen every day without a
>> >> security incident.  Depending on the network the profiler alerts could
>> get
>> >> very noisy so it might be better to correlate profiler alerts with
>> other
>> >> alerts.
>> >>>
>> >>> Thanks
>> >>> Carolyn
>> >>>
>> >>>
>> >>>
>> >>> Sent from my Verizon, Samsung Galaxy smartphone
>> >>>
>> >>>
>> >>>  Original message 
>> >>> From: Casey Stella 
>> >>> Date: 2/1/17 2:28 PM (GMT-05:00)
>> >>> To: dev@metron.incubator.apache.org
>> >>> Subject: Re: [Discuss] Improve Alerting
>> >>>
>> >>> I like the direction.  One thing that we may want is for comment to
>> just
>> >> be
>> >>> a stellar expression and construct a function to essentially do
>> >>> String.format().  So, that'd become:
>> >>> "triageConfig" : {
>> >>> "riskLevelRules" : [
>> >>>   {
>> >>> "name" : "Abnormal Value",
>> >>> "comment" : "FORMAT('For %s; the value %s exceeds threshold of
>> %d',
>> >>> hostname, value, value_threshold)"
>> >>> "rule" : "value > value_threshold",
>> >>> "score" : 10
>> >>>   }
>> >>> ],
>> >>> "aggregator" : "MAX"
>> >>> }
>> >>>
>> >>> The reason:
>> >>>
>> >>>  - It's integrated and stellar is our default scripting layer
>> >>>  - It supports doing some computation in the message
>> >>>
>> >>>
>> >>> On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen 
>> wrote:
>> >>>
>>  Like I said, here is a proposed solution to one of the gaps I
>> >> identified in
>>  the previous email.

Re: [Discuss] Improve Alerting

[Nick Allen]

Oh, I see.  Yes, very useful.


On Thu, Feb 2, 2017 at 9:39 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> That’s a part of it, certainly (and fixes another of my bug bears, so
> thank you!)
>
> In addition to the aggregation being stellar, I want score to be a stellar
> statement, I’ve put in a separate ticket for that.
> https://issues.apache.org/jira/browse/METRON-685 <
> https://issues.apache.org/jira/browse/METRON-685>
>
> Simon
>
> > On 2 Feb 2017, at 14:31, Nick Allen  wrote:
> >
> >> I would much rather be able to say something like score = some stellar
> >> statement that returns a float...
> >
> >
> > Completely agree.  FYI - We added METRON-683 yesterday that I believe
> > supports what you are saying.  Feel free to add commentary.
> >
> > https://issues.apache.org/jira/browse/METRON-683
> >
> > On Thu, Feb 2, 2017 at 9:02 AM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> I completely agree with Nick’s transparency comments, and like the
> design
> >> of the configuration, especially provision for messaging around the
> nature
> >> of the rule fired.
> >>
> >> I would just like to add a small point on the capabilities here. If the
> >> message could have embedded values through some sort of template for a
> >> stellar statement, it would make for a better more dynamic alert reason.
> >>
> >> I would also like to see the score field capable of outputting the value
> >> of a stellar statement. At the moment the idea of a static score being
> >> passed on means that if I have a probabilistic result I want to combine
> >> with other triage sources, I have to do a lot of bucketing into fixed
> >> values. I would much rather be able to say something like score = some
> >> stellar statement that returns a float, ‘alertness' = threshold of this.
> >> That way I can combine multiple triage rules to trigger an overall
> alert,
> >> making the aggregators more meaningful.
> >>
> >> Simon
> >>
> >>
> >>> On 2 Feb 2017, at 12:40, Carolyn Duby  wrote:
> >>>
> >>> For profiler alerts it will be helpful during analysis to see the
> alerts
> >> that caused the anomaly.  The meta alert is useful for incidents
> involving
> >> correlation of multiple events.
> >>>
> >>> Also you will need to filter out known hosts that trigger anomalies.
> >> For example vulnerability scanning software.
> >>>
> >>> One final thing to consider is anomalies happen every day without a
> >> security incident.  Depending on the network the profiler alerts could
> get
> >> very noisy so it might be better to correlate profiler alerts with other
> >> alerts.
> >>>
> >>> Thanks
> >>> Carolyn
> >>>
> >>>
> >>>
> >>> Sent from my Verizon, Samsung Galaxy smartphone
> >>>
> >>>
> >>>  Original message 
> >>> From: Casey Stella 
> >>> Date: 2/1/17 2:28 PM (GMT-05:00)
> >>> To: dev@metron.incubator.apache.org
> >>> Subject: Re: [Discuss] Improve Alerting
> >>>
> >>> I like the direction.  One thing that we may want is for comment to
> just
> >> be
> >>> a stellar expression and construct a function to essentially do
> >>> String.format().  So, that'd become:
> >>> "triageConfig" : {
> >>> "riskLevelRules" : [
> >>>   {
> >>> "name" : "Abnormal Value",
> >>> "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
> >>> hostname, value, value_threshold)"
> >>> "rule" : "value > value_threshold",
> >>> "score" : 10
> >>>   }
> >>> ],
> >>> "aggregator" : "MAX"
> >>> }
> >>>
> >>> The reason:
> >>>
> >>>  - It's integrated and stellar is our default scripting layer
> >>>  - It supports doing some computation in the message
> >>>
> >>>
> >>> On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:
> >>>
>  Like I said, here is a proposed solution to one of the gaps I
> >> identified in
>  the previous email.
> 
>  *Problem*
> 
>  There is little transparency into the Threat Triage process itself.
> >> When
>  Threat Triage runs, all I get is a score.  I don't know how that score
> >> was
>  arrived at, which rules were triggered, and the specific values that
> >> caused
>  a rule to trigger.
> 
>  More specifically, there is no way to generate a message that looks
> like
>  "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> the
>  threshold of '202'".  This makes it difficult for an analyst to action
> >> the
>  alert.
> 
>  *Proposed Solution*
> 
>  To improve the transparency of the Threat Triage process, I am
> proposing
>  these enhancements.
> 
>  1. Threat Triage should attach to each message all of the rules that
> >> fired
>  in addition to the total calculated threat triage score.
> 
>  2. Threat Triage should allow a custom message to be generated for
> each
>  rule.  The custom message would allow for some form of string
> >> interpolation
>  so that I can 

Re: [Discuss] Improve Alerting

[Simon Elliston Ball]

That’s a part of it, certainly (and fixes another of my bug bears, so thank 
you!) 

In addition to the aggregation being stellar, I want score to be a stellar 
statement, I’ve put in a separate ticket for that. 
https://issues.apache.org/jira/browse/METRON-685 


Simon

> On 2 Feb 2017, at 14:31, Nick Allen  wrote:
> 
>> I would much rather be able to say something like score = some stellar
>> statement that returns a float...
> 
> 
> Completely agree.  FYI - We added METRON-683 yesterday that I believe
> supports what you are saying.  Feel free to add commentary.
> 
> https://issues.apache.org/jira/browse/METRON-683
> 
> On Thu, Feb 2, 2017 at 9:02 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> I completely agree with Nick’s transparency comments, and like the design
>> of the configuration, especially provision for messaging around the nature
>> of the rule fired.
>> 
>> I would just like to add a small point on the capabilities here. If the
>> message could have embedded values through some sort of template for a
>> stellar statement, it would make for a better more dynamic alert reason.
>> 
>> I would also like to see the score field capable of outputting the value
>> of a stellar statement. At the moment the idea of a static score being
>> passed on means that if I have a probabilistic result I want to combine
>> with other triage sources, I have to do a lot of bucketing into fixed
>> values. I would much rather be able to say something like score = some
>> stellar statement that returns a float, ‘alertness' = threshold of this.
>> That way I can combine multiple triage rules to trigger an overall alert,
>> making the aggregators more meaningful.
>> 
>> Simon
>> 
>> 
>>> On 2 Feb 2017, at 12:40, Carolyn Duby  wrote:
>>> 
>>> For profiler alerts it will be helpful during analysis to see the alerts
>> that caused the anomaly.  The meta alert is useful for incidents involving
>> correlation of multiple events.
>>> 
>>> Also you will need to filter out known hosts that trigger anomalies.
>> For example vulnerability scanning software.
>>> 
>>> One final thing to consider is anomalies happen every day without a
>> security incident.  Depending on the network the profiler alerts could get
>> very noisy so it might be better to correlate profiler alerts with other
>> alerts.
>>> 
>>> Thanks
>>> Carolyn
>>> 
>>> 
>>> 
>>> Sent from my Verizon, Samsung Galaxy smartphone
>>> 
>>> 
>>>  Original message 
>>> From: Casey Stella 
>>> Date: 2/1/17 2:28 PM (GMT-05:00)
>>> To: dev@metron.incubator.apache.org
>>> Subject: Re: [Discuss] Improve Alerting
>>> 
>>> I like the direction.  One thing that we may want is for comment to just
>> be
>>> a stellar expression and construct a function to essentially do
>>> String.format().  So, that'd become:
>>> "triageConfig" : {
>>> "riskLevelRules" : [
>>>   {
>>> "name" : "Abnormal Value",
>>> "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
>>> hostname, value, value_threshold)"
>>> "rule" : "value > value_threshold",
>>> "score" : 10
>>>   }
>>> ],
>>> "aggregator" : "MAX"
>>> }
>>> 
>>> The reason:
>>> 
>>>  - It's integrated and stellar is our default scripting layer
>>>  - It supports doing some computation in the message
>>> 
>>> 
>>> On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:
>>> 
 Like I said, here is a proposed solution to one of the gaps I
>> identified in
 the previous email.
 
 *Problem*
 
 There is little transparency into the Threat Triage process itself.
>> When
 Threat Triage runs, all I get is a score.  I don't know how that score
>> was
 arrived at, which rules were triggered, and the specific values that
>> caused
 a rule to trigger.
 
 More specifically, there is no way to generate a message that looks like
 "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
 threshold of '202'".  This makes it difficult for an analyst to action
>> the
 alert.
 
 *Proposed Solution*
 
 To improve the transparency of the Threat Triage process, I am proposing
 these enhancements.
 
 1. Threat Triage should attach to each message all of the rules that
>> fired
 in addition to the total calculated threat triage score.
 
 2. Threat Triage should allow a custom message to be generated for each
 rule.  The custom message would allow for some form of string
>> interpolation
 so that I can add specific values from each message to the generated
 alert.  We could allow this in one or both of the new fields that Casey
 just added, name and comment.
 
 
 *Example*
 
 1. In this example, we have a telemetry message with a field called
>> 'value'
 that we need to monitor.  In Enrichment, I calculate some sort of value
 

Re: [Discuss] Improve Alerting

[Nick Allen]

> I would much rather be able to say something like score = some stellar
> statement that returns a float...


Completely agree.  FYI - We added METRON-683 yesterday that I believe
supports what you are saying.  Feel free to add commentary.

https://issues.apache.org/jira/browse/METRON-683

On Thu, Feb 2, 2017 at 9:02 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> I completely agree with Nick’s transparency comments, and like the design
> of the configuration, especially provision for messaging around the nature
> of the rule fired.
>
> I would just like to add a small point on the capabilities here. If the
> message could have embedded values through some sort of template for a
> stellar statement, it would make for a better more dynamic alert reason.
>
> I would also like to see the score field capable of outputting the value
> of a stellar statement. At the moment the idea of a static score being
> passed on means that if I have a probabilistic result I want to combine
> with other triage sources, I have to do a lot of bucketing into fixed
> values. I would much rather be able to say something like score = some
> stellar statement that returns a float, ‘alertness' = threshold of this.
> That way I can combine multiple triage rules to trigger an overall alert,
> making the aggregators more meaningful.
>
> Simon
>
>
> > On 2 Feb 2017, at 12:40, Carolyn Duby  wrote:
> >
> > For profiler alerts it will be helpful during analysis to see the alerts
> that caused the anomaly.  The meta alert is useful for incidents involving
> correlation of multiple events.
> >
> > Also you will need to filter out known hosts that trigger anomalies.
> For example vulnerability scanning software.
> >
> > One final thing to consider is anomalies happen every day without a
> security incident.  Depending on the network the profiler alerts could get
> very noisy so it might be better to correlate profiler alerts with other
> alerts.
> >
> > Thanks
> > Carolyn
> >
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
> >
> >  Original message 
> > From: Casey Stella 
> > Date: 2/1/17 2:28 PM (GMT-05:00)
> > To: dev@metron.incubator.apache.org
> > Subject: Re: [Discuss] Improve Alerting
> >
> > I like the direction.  One thing that we may want is for comment to just
> be
> > a stellar expression and construct a function to essentially do
> > String.format().  So, that'd become:
> > "triageConfig" : {
> >  "riskLevelRules" : [
> >{
> >  "name" : "Abnormal Value",
> >  "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
> > hostname, value, value_threshold)"
> >  "rule" : "value > value_threshold",
> >  "score" : 10
> >}
> >  ],
> >  "aggregator" : "MAX"
> > }
> >
> > The reason:
> >
> >   - It's integrated and stellar is our default scripting layer
> >   - It supports doing some computation in the message
> >
> >
> > On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:
> >
> >> Like I said, here is a proposed solution to one of the gaps I
> identified in
> >> the previous email.
> >>
> >> *Problem*
> >>
> >> There is little transparency into the Threat Triage process itself.
> When
> >> Threat Triage runs, all I get is a score.  I don't know how that score
> was
> >> arrived at, which rules were triggered, and the specific values that
> caused
> >> a rule to trigger.
> >>
> >> More specifically, there is no way to generate a message that looks like
> >> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> >> threshold of '202'".  This makes it difficult for an analyst to action
> the
> >> alert.
> >>
> >> *Proposed Solution*
> >>
> >> To improve the transparency of the Threat Triage process, I am proposing
> >> these enhancements.
> >>
> >> 1. Threat Triage should attach to each message all of the rules that
> fired
> >> in addition to the total calculated threat triage score.
> >>
> >> 2. Threat Triage should allow a custom message to be generated for each
> >> rule.  The custom message would allow for some form of string
> interpolation
> >> so that I can add specific values from each message to the generated
> >> alert.  We could allow this in one or both of the new fields that Casey
> >> just added, name and comment.
> >>
> >>
> >> *Example*
> >>
> >> 1. In this example, we have a telemetry message with a field called
> 'value'
> >> that we need to monitor.  In Enrichment, I calculate some sort of value
> >> threshold, over which an alert should be generated.
> >>
> >>
> >> 2. In Threat Triage, I use the calculated value threshold to alert on
> any
> >> message that has a value exceeding this threshold.
> >>
> >> 3. I can embed values from the message, like the hostname, value, and
> value
> >> threshold, into the alert produced by Threat Triage.  Notice that I am
> >> using ${this} for string interpolation, but it could be any syntax that
> we
> >> choose.
> >>
> 

[GitHub] incubator-metron issue #433: METRON-680: GeoLiteDatabase incorrectly using c...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/433
  
Ok, sounds good to me.  +1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #433: METRON-680: GeoLiteDatabase incorrectly using c...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/433
  
I don't mind this change, but could we add another field for "countryLocId" 
so that, if so desired, people could reconfigure that instead?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #433: METRON-680: GeoLiteDatabase incorrectly using c...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/433
  
@cestella My mistake, fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #433: Metron 680: GeoLiteDatabase incorrectly using c...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/433
  
Can you reopen with the right JIRA prefix: METRON-680 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Discuss] Improve Alerting

[Simon Elliston Ball]

I completely agree with Nick’s transparency comments, and like the design of 
the configuration, especially provision for messaging around the nature of the 
rule fired. 

I would just like to add a small point on the capabilities here. If the message 
could have embedded values through some sort of template for a stellar 
statement, it would make for a better more dynamic alert reason. 

I would also like to see the score field capable of outputting the value of a 
stellar statement. At the moment the idea of a static score being passed on 
means that if I have a probabilistic result I want to combine with other triage 
sources, I have to do a lot of bucketing into fixed values. I would much rather 
be able to say something like score = some stellar statement that returns a 
float, ‘alertness' = threshold of this. That way I can combine multiple triage 
rules to trigger an overall alert, making the aggregators more meaningful.

Simon


> On 2 Feb 2017, at 12:40, Carolyn Duby  wrote:
> 
> For profiler alerts it will be helpful during analysis to see the alerts that 
> caused the anomaly.  The meta alert is useful for incidents involving 
> correlation of multiple events.
> 
> Also you will need to filter out known hosts that trigger anomalies.  For 
> example vulnerability scanning software.
> 
> One final thing to consider is anomalies happen every day without a security 
> incident.  Depending on the network the profiler alerts could get very noisy 
> so it might be better to correlate profiler alerts with other alerts.
> 
> Thanks
> Carolyn
> 
> 
> 
> Sent from my Verizon, Samsung Galaxy smartphone
> 
> 
>  Original message 
> From: Casey Stella 
> Date: 2/1/17 2:28 PM (GMT-05:00)
> To: dev@metron.incubator.apache.org
> Subject: Re: [Discuss] Improve Alerting
> 
> I like the direction.  One thing that we may want is for comment to just be
> a stellar expression and construct a function to essentially do
> String.format().  So, that'd become:
> "triageConfig" : {
>  "riskLevelRules" : [
>{
>  "name" : "Abnormal Value",
>  "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
> hostname, value, value_threshold)"
>  "rule" : "value > value_threshold",
>  "score" : 10
>}
>  ],
>  "aggregator" : "MAX"
> }
> 
> The reason:
> 
>   - It's integrated and stellar is our default scripting layer
>   - It supports doing some computation in the message
> 
> 
> On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:
> 
>> Like I said, here is a proposed solution to one of the gaps I identified in
>> the previous email.
>> 
>> *Problem*
>> 
>> There is little transparency into the Threat Triage process itself.  When
>> Threat Triage runs, all I get is a score.  I don't know how that score was
>> arrived at, which rules were triggered, and the specific values that caused
>> a rule to trigger.
>> 
>> More specifically, there is no way to generate a message that looks like
>> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
>> threshold of '202'".  This makes it difficult for an analyst to action the
>> alert.
>> 
>> *Proposed Solution*
>> 
>> To improve the transparency of the Threat Triage process, I am proposing
>> these enhancements.
>> 
>> 1. Threat Triage should attach to each message all of the rules that fired
>> in addition to the total calculated threat triage score.
>> 
>> 2. Threat Triage should allow a custom message to be generated for each
>> rule.  The custom message would allow for some form of string interpolation
>> so that I can add specific values from each message to the generated
>> alert.  We could allow this in one or both of the new fields that Casey
>> just added, name and comment.
>> 
>> 
>> *Example*
>> 
>> 1. In this example, we have a telemetry message with a field called 'value'
>> that we need to monitor.  In Enrichment, I calculate some sort of value
>> threshold, over which an alert should be generated.
>> 
>> 
>> 2. In Threat Triage, I use the calculated value threshold to alert on any
>> message that has a value exceeding this threshold.
>> 
>> 3. I can embed values from the message, like the hostname, value, and value
>> threshold, into the alert produced by Threat Triage.  Notice that I am
>> using ${this} for string interpolation, but it could be any syntax that we
>> choose.
>> 
>> 
>> "triageConfig" : {
>>  "riskLevelRules" : [
>>{
>>  "name" : "Abnormal Value",
>>  "comment" : "For ${hostname}; the value ${value} exceeds threshold of
>> ${value_threshold}",
>>  "rule" : "value > value_threshold",
>>  "score" : 10
>>}
>>  ],
>>  "aggregator" : "MAX"
>> }
>> 
>> 
>> 4. The Threat Triage process today would add only the total calculated
>> score.
>> 
>> "threat.triage.level": 10.0
>> 
>> 
>> With this proposal, Threat Triage would add the following to the message.
>> 
>> Notice how each of the ${variables} have been replaced 

[GitHub] incubator-metron pull request #433: Metron 680: GeoLiteDatabase incorrectly ...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/433

Metron 680: GeoLiteDatabase incorrectly using country geoname_id instead of 
city

Swaps out country's geoname_id for city's geoname_id.  Both ids are the 
same in format (they're the ids for the geonames.org dataset).

The only direct effect I know of from this is the Kibana dashboard uses 
this field for unique locations (so changing what feeds this changes the 
dashboard output).  The main wrinkle is that city can be unpopulated, even 
though a country exists (e.g. if an IP is just assigned to the U.S., it may not 
have a city).  This isn't always true that (at least one IP range for Japan 
appears to fill in Japan as the city's geoname id).

I'm opening the PR with just the direct change, but it's definitely open to 
discussion if we want to adjust.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/incubator-metron METRON-680

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #433


commit 8cd13f44d9c23c6355f21f46284a42ddf55885f9
Author: justinjleet 
Date:   2017-02-01T03:07:31Z

METRON-680: GeoLiteDatabase incorrectly using country geoname_id instead of 
city

commit 4d26af8a46a8f0dbd2c3b67319aaf96beda9d83c
Author: justinjleet 
Date:   2017-02-01T13:43:34Z

Updating tests that geo test data carries through




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: [Discuss] Improve Alerting

[Carolyn Duby]

For profiler alerts it will be helpful during analysis to see the alerts that 
caused the anomaly.  The meta alert is useful for incidents involving 
correlation of multiple events.

Also you will need to filter out known hosts that trigger anomalies.  For 
example vulnerability scanning software.

One final thing to consider is anomalies happen every day without a security 
incident.  Depending on the network the profiler alerts could get very noisy so 
it might be better to correlate profiler alerts with other alerts.

Thanks
Carolyn



Sent from my Verizon, Samsung Galaxy smartphone


 Original message 
From: Casey Stella 
Date: 2/1/17 2:28 PM (GMT-05:00)
To: dev@metron.incubator.apache.org
Subject: Re: [Discuss] Improve Alerting

I like the direction.  One thing that we may want is for comment to just be
a stellar expression and construct a function to essentially do
String.format().  So, that'd become:
"triageConfig" : {
  "riskLevelRules" : [
{
  "name" : "Abnormal Value",
  "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
hostname, value, value_threshold)"
  "rule" : "value > value_threshold",
  "score" : 10
}
  ],
  "aggregator" : "MAX"
}

The reason:

   - It's integrated and stellar is our default scripting layer
   - It supports doing some computation in the message


On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:

> Like I said, here is a proposed solution to one of the gaps I identified in
> the previous email.
>
> *Problem*
>
> There is little transparency into the Threat Triage process itself.  When
> Threat Triage runs, all I get is a score.  I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused
> a rule to trigger.
>
> More specifically, there is no way to generate a message that looks like
> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> threshold of '202'".  This makes it difficult for an analyst to action the
> alert.
>
> *Proposed Solution*
>
> To improve the transparency of the Threat Triage process, I am proposing
> these enhancements.
>
> 1. Threat Triage should attach to each message all of the rules that fired
> in addition to the total calculated threat triage score.
>
> 2. Threat Triage should allow a custom message to be generated for each
> rule.  The custom message would allow for some form of string interpolation
> so that I can add specific values from each message to the generated
> alert.  We could allow this in one or both of the new fields that Casey
> just added, name and comment.
>
>
> *Example*
>
> 1. In this example, we have a telemetry message with a field called 'value'
> that we need to monitor.  In Enrichment, I calculate some sort of value
> threshold, over which an alert should be generated.
>
>
> 2. In Threat Triage, I use the calculated value threshold to alert on any
> message that has a value exceeding this threshold.
>
> 3. I can embed values from the message, like the hostname, value, and value
> threshold, into the alert produced by Threat Triage.  Notice that I am
> using ${this} for string interpolation, but it could be any syntax that we
> choose.
>
>
> "triageConfig" : {
>   "riskLevelRules" : [
> {
>   "name" : "Abnormal Value",
>   "comment" : "For ${hostname}; the value ${value} exceeds threshold of
> ${value_threshold}",
>   "rule" : "value > value_threshold",
>   "score" : 10
> }
>   ],
>   "aggregator" : "MAX"
> }
>
>
> 4. The Threat Triage process today would add only the total calculated
> score.
>
> "threat.triage.level": 10.0
>
>
> With this proposal, Threat Triage would add the following to the message.
>
> Notice how each of the ${variables} have been replaced with the actual
> values extracted from the message.  This allows for more contextual
> information to action the alert.
>
> "threat.triage": {
> "score": 10.0,
> "rules": [
>   {
> "name": "Abnormal Value",
> "comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
> "score" : 10
>   }
> ]
> }
>
>
>
> What do you think?  Any alternative ideas?
>
>
>
> On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
>
> > I'd like to explore the functionality that we have in Metron using a
> > motivating example.  I think this will help highlight some gaps where we
> > can enhance Metron.
> >
> > The motivating example is that I would like to create an alert if the
> > number of inbound flows to any host over a 15 minute interval is
> abnormal.
> > I would like the alert to contain the specific information below to
> > streamline the triage process.
> >
> > Rule: Abnormal number of inbound flows
> > Bin: 15 mins
> > Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> > the threshold of '202'
> >
> >
> > *What Works*
> >
> > In some ways, this example is similar to the "Outlier Detection" demo
>