[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/864
  
I did another 12-node deployment on Centos 7 with this PR (bypassed the 
kafka issue by installing Kafka broker on all nodes). The fix worked just 
perfect. Thanks much @nickwallen ! 

+1 (non-binding)


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
Ok, I'll change it.  Feels a little crossing the streams, but we'll see


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/856
  
The justification that you mentioned just doesn't seem strong enough to me. 
 Unless there is more that I am missing.

IMHO We should only use magic commands for things that can't be 
accomplished in the language using the preferred extension mechanism; aka 
defining Stellar functions.


---

[GitHub] metron issue #871: METRON-1365: Allow PROFILE_GET to return a default value ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/871
  
+1 Looks great.


---

[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/864
  
Hi @nickwallen , I tried this on a 12-node cluster. I validated that 
`clusterHostInfo` is populated properly for the alerts_ui, management_ui and 
rest_ui hosts. 

However, in my case it failed on the parser service check since the 'Metron 
Check' step landed on a host without Kafka broker installed.

Here's the error excerpt:
```

2017-12-20 18:42:54,285 - Performing Parser service check
2017-12-20 18:42:54,285 - Checking for grok patterns in HDFS for Parsers
2017-12-20 18:42:54,285 - Checking HDFS; directory=/apps/metron/patterns 
user=metron
2017-12-20 18:42:54,285 - Execute['/usr/hdp/2.5.3.0-37/hadoop/bin/hdfs dfs 
-test -d /apps/metron/patterns'] {'logoutput': True, 'path': 
['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user': 'metron', 
'try_sleep': 5}
2017-12-20 18:42:56,822 - Checking Kafka topics for Parsers
2017-12-20 18:42:56,822 - Checking existence of Kafka topic 'bro'
2017-12-20 18:42:56,823 - 
Execute['/usr/hdp/current/kafka-broker/bin/kafka-topics.sh   --zookeeper 
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
   --list |   awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit 
0} else {exit 1}}''] {'logoutput': True, 'path': 
['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user': 'kafka', 
'try_sleep': 5}
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or 
directory
2017-12-20 18:42:56,900 - Retrying after 5 seconds. Reason: Execution of 
'/usr/hdp/current/kafka-broker/bin/kafka-topics.sh   --zookeeper 
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
   --list |   awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit 
0} else {exit 1}}'' returned 1. -bash: 
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or directory
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or 
directory
2017-12-20 18:43:01,987 - Retrying after 5 seconds. Reason: Execution of 
'/usr/hdp/current/kafka-broker/bin/kafka-topics.sh   --zookeeper 
metronc-1.openstacklocal:2181,metronc-11.openstacklocal:2181,metronc-10.openstacklocal:2181
   --list |   awk 'BEGIN {cnt=0;} /bro/ {cnt++} END {if (cnt > 0) {exit 
0} else {exit 1}}'' returned 1. -bash: 
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or directory
-bash: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh: No such file or 
directory

Command failed after 1 tries

```

I noticed that the `clusterHostInfo` indeed has a list of the 
`kafka_broker_hosts` (see attached

[clusterHostInfo-12-node.txt](https://github.com/apache/metron/files/1576752/clusterHostInfo-12-node.txt).
 Would it be possible to either a) force Ambari to run metron service check on 
one of the kafka broker hosts; or b) run 
[check_kafka_topics](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py#L259)
 on a `kafka_broker_host`

I am perfectly fine if you think the kafka_broker fix should be a different 
PR than this.


---

[GitHub] metron issue #875: METRON-1373 RAT failure for metron-interface/metron-alert...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/875
  
Ran up through
```bash
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T 2C 
surefire:test@unit-tests && time mvn -q surefire:test@integration-tests  && 
time mvn -q test --projects metron-interface/metron-config && time 
build_utils/verify_licenses.sh
```
Worked fine.  I'm +1


---

[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/864
  
Ran up in full dev, works as described.
Nice Job.

+1


---

Re: [DISCUSS] Resources for how to contribute to Apache Metron

[Casey Stella]

That's really good feedback, Jon.  I agree that we have a significant
barrier to get to the point of tinkering.  Full-dev really wasn't intended
to be that entry point; it's more of a way to test PRs in something
resembling a realistic scenario (and it is still not super realistic).  I
would welcome creative ideas around how to accomplish that goal.

On Wed, Dec 20, 2017 at 10:15 AM, zeo...@gmail.com  wrote:

> For nearly everybody I've talked to about this project that had complaints,
> I've heard something about the significant barrier to entry, divided into
> two general categories.  Category 1 is that a lot of security teams lack
> substantial experience with Hadoop and would like to get a better
> understanding of how the involved components fit together - not
> just kafka goes to storm goes to kafka, or a link to the kafka docs for
> details about kafka, but a little bit more detail as to _why_ those
> components are in use in metron, what properties those components possess
> at a high level _which makes them appealing to us_, and how they're
> _currently used_ in the metron environment.  Category 2 is that it is
> generally more difficult than it should be to get a testing/poc environment
> running - running it on a laptop (especially non-macOS) can be a pain to
> get running, some laptops simply cannot run it, etc.  I've heard a few
> times that a company uses Azure (not AWS) and they would like to quickly
> spin it up there.
>
> Just my $0.02
>
> Jon
>
> On Tue, Dec 19, 2017 at 9:02 AM Otto Fowler 
> wrote:
>
> > Like any project, Apache Metron needs to maintain and grow it’s
> contributor
> > community. We think that we could be doing a better job of this, and
> would
> > like to discuss issues and possible improvements. Issues
> >
> > What are some of the issues that may inhibit people contributing?
> >
> >- Barrier of entry (issues getting Metron running in vagrant or local)
> >- Documentation : finding current
> >- Documentation : content and quality
> >- Source Code navigation/documentation/guides
> >- Testing guides
> >- Use Case Guides
> >- Don’t know how they *can* contribute
> >- Others that I’m missing?
> >
> > Remediation Barrier of entry
> >
> > How can we make the local deployment workflow easier ( other discuss
> thread
> > touches on this)?
> > Documentation : Finding Current
> >
> > When people look for Metron info, where are they looking? What comes up
> in
> > search? - Hortonworks Community forums ( preview release stuff ? ), old
> > blog posts? - Mailing list archives? - wiki? (not current) - site-book?
> >
> > How can we reduce the out of data information, and make the relevant
> > information more prominent?
> > Documentation : Content and Quality
> >
> > ( this is a little bit of a chicken and egg issue, since documentation
> is a
> > wonderful way to contribute…. ) - Up to data architecture documentation -
> > Non-developer focused ‘feature’ documentation - Developer focused
> > documentation ( how to add a XX guides )
> > Source Code Guides
> >
> >- Structure of the code tree
> >- What is where, how it is logically setup
> >- How to maintain concistancy when working in the code
> >- Javadoc
> >
> > Testing Guides
> >
> >- Tests that we have are buried in PR’s
> >- No regression tests
> >
> > Use case guides
> >
> >- more how-to guides
> >
> > Contributing guide
> >
> >- right now, have dev env guide
> >- review and submit doc changes
> >- review PR guide
> >- pr testing guide ( better pr testing steps?)
> >
> > These are things I can think of, anyone have any comment, additions,
> > priorities?
> >
> --
>
> Jon
>

Re: [DISCUSS] Resources for how to contribute to Apache Metron

[zeo...@gmail.com]

For nearly everybody I've talked to about this project that had complaints,
I've heard something about the significant barrier to entry, divided into
two general categories.  Category 1 is that a lot of security teams lack
substantial experience with Hadoop and would like to get a better
understanding of how the involved components fit together - not
just kafka goes to storm goes to kafka, or a link to the kafka docs for
details about kafka, but a little bit more detail as to _why_ those
components are in use in metron, what properties those components possess
at a high level _which makes them appealing to us_, and how they're
_currently used_ in the metron environment.  Category 2 is that it is
generally more difficult than it should be to get a testing/poc environment
running - running it on a laptop (especially non-macOS) can be a pain to
get running, some laptops simply cannot run it, etc.  I've heard a few
times that a company uses Azure (not AWS) and they would like to quickly
spin it up there.

Just my $0.02

Jon

On Tue, Dec 19, 2017 at 9:02 AM Otto Fowler  wrote:

> Like any project, Apache Metron needs to maintain and grow it’s contributor
> community. We think that we could be doing a better job of this, and would
> like to discuss issues and possible improvements. Issues
>
> What are some of the issues that may inhibit people contributing?
>
>- Barrier of entry (issues getting Metron running in vagrant or local)
>- Documentation : finding current
>- Documentation : content and quality
>- Source Code navigation/documentation/guides
>- Testing guides
>- Use Case Guides
>- Don’t know how they *can* contribute
>- Others that I’m missing?
>
> Remediation Barrier of entry
>
> How can we make the local deployment workflow easier ( other discuss thread
> touches on this)?
> Documentation : Finding Current
>
> When people look for Metron info, where are they looking? What comes up in
> search? - Hortonworks Community forums ( preview release stuff ? ), old
> blog posts? - Mailing list archives? - wiki? (not current) - site-book?
>
> How can we reduce the out of data information, and make the relevant
> information more prominent?
> Documentation : Content and Quality
>
> ( this is a little bit of a chicken and egg issue, since documentation is a
> wonderful way to contribute…. ) - Up to data architecture documentation -
> Non-developer focused ‘feature’ documentation - Developer focused
> documentation ( how to add a XX guides )
> Source Code Guides
>
>- Structure of the code tree
>- What is where, how it is logically setup
>- How to maintain concistancy when working in the code
>- Javadoc
>
> Testing Guides
>
>- Tests that we have are buried in PR’s
>- No regression tests
>
> Use case guides
>
>- more how-to guides
>
> Contributing guide
>
>- right now, have dev env guide
>- review and submit doc changes
>- review PR guide
>- pr testing guide ( better pr testing steps?)
>
> These are things I can think of, anyone have any comment, additions,
> priorities?
>
-- 

Jon

Re: [DISCUSS] Stellar in a Zeppelin Notebook

[zeo...@gmail.com]

This is some awesome work, I'm looking forward to being able to play with
it.

Jon

On Tue, Dec 19, 2017 at 1:12 PM Nick Allen  wrote:

> Yes, I definitely want auto-complete also.
>
> I am factoring out some of the logic you did for auto-complete in the REPL
> in hopes of being able to apply that in Zeppelin.  I believe I saw
> references in the other Zeppelin interpreters for handling auto-complete,
> but I'll have to research it a bit more.
>
> On Tue, Dec 19, 2017 at 1:07 PM, Casey Stella  wrote:
>
> > I love it!  I wonder if we could get more of the REPL-like experience
> > (i.e. I crave autocomplete ;) if we integrated it with jquery shell like
> > they did with nanook (https://github.com/aeshell/nanook).  I know
> > zeppelin lets you integrate with more complex javascript.   Regardless,
> > this is awesome, great job!
> >
> > On Tue, Dec 19, 2017 at 12:44 PM, Otto Fowler 
> > wrote:
> >
> >> That looks great!
> >>
> >>
> >> On December 19, 2017 at 12:34:47, Nick Allen (n...@nickallen.org)
> wrote:
> >>
> >> Ah, dang.  Hopefully this works...
> >>
> >> https://www.dropbox.com/s/44qz3518dn4jtzq/Stellar%20in%20a%
> >> 20Zeppelin%20Notebook.png?dl=0
> >>
> >> On Tue, Dec 19, 2017 at 10:23 AM, Otto Fowler 
> >> wrote:
> >>
> >> > The image is stripped for me, can you post it as a link?
> >> >
> >> > This seems like it would look awesome ;)
> >> >
> >> >
> >> > On December 19, 2017 at 10:03:26, Nick Allen (n...@nickallen.org)
> >> wrote:
> >> >
> >> > (1) I love the REPL, but I hate how inaccessible it is.
> >> >
> >> > (2) I love our use cases
> >> >  >> raphic_login_outliers> and
> >> > examples
> >> >  >> cs/metron-profiler#creating-profiles>,
> >> > but I hate how difficult it is for a new user to run them.
> >> >
> >> > (3) I love the extensibility of Metron, but I hate looking at JSON.
> >> >
> >> > (4) I love the Profiler, but I hate not being able to *see* my
> profiles
> >> as
> >> > plots.
> >> >
> >> > ...
> >> >
> >> > Let me introduce, Stellar running in a Zeppelin Notebook.
> >> >
> >> > (1) Access the REPL from any web browser.
> >> >
> >> > (2) Create executable use cases that can be easily shared between
> users.
> >> >
> >> > (3) Use the simpler management functions to interact with Metron (less
> >> > JSON).
> >> >
> >> > (4) Extract your profiles and create a time series plot.
> >> >
> >> >
> >> >
> >> > [image: Inline image 1]
> >> > The screenshot above is a very lightweight MVP showing that we can run
> >> > Stellar from Zeppelin.  I have a lot more work ahead in refactoring
> the
> >> > existing Stellar Shell/REPL functionality so that we get the same
> >> > experience in Zeppelin as we get on the command line.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>
-- 

Jon

[GitHub] metron issue #871: METRON-1365: Allow PROFILE_GET to return a default value ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/871
  
Thanks for the review, @nickwallen I reacted to your comments. :)


---

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/871#discussion_r158040472
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerClientConfig.java
 ---
@@ -56,8 +56,11 @@
   /**
* A global property that defines the salt divisor used to store profile 
data.
*/
-  PROFILER_SALT_DIVISOR("profiler.client.salt.divisor", 1000L, Long.class);
-
+  PROFILER_SALT_DIVISOR("profiler.client.salt.divisor", 1000L, Long.class)
+  /**
+   * The default value to be returned if a profile is not written for a 
given period for a profile and entity.
+   */
+  ,PROFILER_DEFAULT_VALUE("profiler.default.value", null, Object.class);
--- End diff --

Good catch, corrected.


---

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/871#discussion_r158040446
  
--- Diff: 
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
 ---
@@ -39,11 +39,7 @@
 import org.junit.Before;
 import org.junit.Test;
 
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
--- End diff --

Good catch, corrected.


---

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/871#discussion_r158040425
  
--- Diff: 
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
 ---
@@ -310,6 +306,36 @@ public void testOutsideTimeHorizon() {
 Assert.assertEquals(0, result.size());
   }
 
+  /**
+   * Default value should be able to be specified
+   */
+  @Test
+  public void testWithDefaultValue() {
+String expr = "PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(4, 
'HOURS'))";
+@SuppressWarnings("unchecked")
+List result = run(expr, List.class);
+
+// validate - expect to fail to read any values because we didn't 
write any.
+Assert.assertEquals(0, result.size());
+
+// execute - read the profile values - with config_override.
+// first two override values are strings, third is deliberately a 
number.
+testOverride("{'profiler.default.value' : 0}", 0);
+testOverride("{'profiler.default.value' : 'metron'}", "metron");
+testOverride("{'profiler.default.value' : []}", new ArrayList<>());
+  }
+
+  private void testOverride(String overrides, Object defaultVal) {
+  String expr = "PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(4, 
'HOURS'), [], " + overrides + ")"
+  ;
--- End diff --

Good catch, corrected.


---

[GitHub] metron pull request #867: METRON-1350: Add reservoir sampling functions to S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/867


---

[GitHub] metron issue #870: METRON-1364: Add an implementation of Robust PCA outlier ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/870
  
Yep, I owe a manual testing plan.  It's going to essentially be the same 
instructions as we have for 
[MAD](https://github.com/apache/metron/tree/master/metron-analytics/metron-statistics#median-absolute-deviation).
  I'll also be adding a new use-case as a follow-on that tracks outliers in 
entropy for netflow data.


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
Do you feel strongly that this should be a Function?  @cestella ?  I'm not 
opposed to changing it if you are.  I would like to here some more feedback


---

Re: [DISCUSS] Lowering the barrier to entry to for new users

[zeo...@gmail.com]

I agree we should streamline #2 and lower the bar, and we can readdress if
we are getting PRs that don't follow the contributing guidelines.  We
should also make a contributing.md as not everybody knows about the wiki.

For #3, I think the scripts that Nick, Otto, and others have written for
looking at PRs, testing RCs, etc. should get moved in and referenced more
frequently to users.  I believe progress on this has already started, but
we should continue to contribute little utils that simplify baseline tasks.

Finally, I think the migration of docs from wiki to git, and hosted in the
site-book is very important.  I know this was on my list to do and I will
still happily do it when I can get to it, but it may unfortunately be a
while.  Happy to take volunteers or crowd source this a bit.

Jon

On Tue, Dec 19, 2017, 11:56 Laurens Vets  wrote:

> On 2017-12-19 06:19, Justin Leet wrote:
> > One of the topics that came up in recent community meeting was about
> > lowering the barrier to entry for new users.
> >
> > This is a fairly broad topic that I think covers a few different
> > subtopics.
> >
> > 1) Addressing (or making it easier to address) some of the things we've
> > seen on the user group from people getting started.
> > 2) Making contributing easier and the ways to do so more obvious.  This
> > includes things like making it easier to find on our site (compare our
> > page
> > to Storm's, for example).  It also includes things like reassessing our
> > PR
> > template (For example, is everything still useful enough to keep it?).
> > 3) Anything else that would make help users adopt Metron and become
> > actively involved in reviewing, fixes, docs, and all the other sorts of
> > things that make our stuff better.
> >
> > I'm mostly going to open this up to a general discussion and
> > brainstorming,
> > and presumably we come out with some tickets at the end of this.
>
> I use a github gist that Otto once created for me, maybe I can try and
> turn that into something more official?
>
-- 

Jon