[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/857
  
Took a first pass at this and I feel like the e2e tests are much improved.  
Great progress and good job so far.  I was able to get several successful runs 
whereas before it was difficult to get a single successful run.

A couple of comments/suggestions:

- I noticed a warning message stating I wasn't on a recent enough version 
of nodejs.  I think we should be using the maven frontend plugin to run tests 
so that we guarantee a consistent version is used.  I submitted a 
[PR](https://github.com/iraghumitra/incubator-metron/pull/6) as an example of 
how to do this.

- I noticed there are still several browser.sleep statements throughout the 
tests (I counted 20).  I think our goal should be to remove all of them.  I 
know some of these should definitely be removed (alerts-list.po.ts, line 87) 
and may have just been missed.  If there are cases where we MUST have them, I 
think those cases needed to be discussed and justified.

- I feel like the "should expand all facets" and "should collapse all 
facets" tests in alert-filters.e2e-spec.ts are unnecessary (and would allow us 
to remove a couple browser.sleep statements).  These tests are simply opening 
and closing bootstrap widgets which is controlled by code we don't maintain 
(bootstrap).  I would instead prefer a test that selects a filter and checks 
that the search box and results are properly updated. 

- I have ran into these errors a couple times.  Not sure if I just ended up 
in a bad state somehow or if it's because I'm running tests through Maven:
```
✗ should have all the steps for meta alerts workflow
- Failed: unknown error: Element  is not clickable at point (1278, 102). Other element 
would receive the click: ...

✗ should create a meta alert from nesting of more than one level
- Failed: unknown error: Element  is not 
clickable at point (1350, 503). Other element would receive the click: ...
```




---

[GitHub] metron pull request #857: METRON-1340: Improve e2e tests for metron alerts

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/857#discussion_r155067123
  
--- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts ---
@@ -38,5 +38,6 @@ export let TREE_SUB_GROUP_SIZE = 5;
 export let DEFAULT_FACETS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let DEFAULT_GROUPS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 
'host', 'enrichments:geo:ip_dst_addr:country'];
 export let INDEXES =  environment.indices ? environment.indices.split(',') 
: [];
+export let POLLING_DEFAULT_STATE = environment.defaultPollingState;
--- End diff --

What does the POLLING_DEFAULT_STATE setting do?


---

[GitHub] metron issue #785: METRON-1230: As a stopgap prior to METRON-777, add more s...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/785
  
So, when 777 hits, this is going to be removed?


---

[GitHub] metron pull request #785: METRON-1230: As a stopgap prior to METRON-777, add...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/785#discussion_r155033012
  
--- Diff: metron-platform/metron-parsers/3rdPartyParser.md ---
@@ -0,0 +1,306 @@
+# Custom Metron Parsers
+
+We have many stock parsers for normal operations.  Some of these are
+networking and cybersecurity focused (e.g. the ASA Parser), some of
+these are general purpose (e.g. the CSVParser), but inevitably users
+will want to extend the system to process their own data formats.  To
+enable this, this is a walkthrough of how to create and use a custom
+parser within Metron.
+
+# Writing A Custom Parser
+Before we can use a parser, we will need to create a custom parser.  The
+parser is the workhorse of Metron ingest.  It provides the mapping
+between the raw data coming in via the Kafka value and a `JSONObject`,
+the internal data structure provided.
+
+## Implementation
+
+In order to do create a custom parser, we need to do one of the following:
+* Write a class which conforms to the 
`org.apache.metron.parsers.interfaces.MessageParser` and 
`java.util.Serializable` interfaces
+  * Implement `init()`, `validate(JSONObject message)`, and 
`List parse(byte[] rawMessage)`
+* Write a class which extends `org.apache.metron.parsers.BasicParser`
+  * Provides convenience implementations to `validate` which ensures 
`timestamp` and `original_string` fields exist.
+
+## Example
+
+In order to illustrate how this might be done, let's create a very
+simple parser that takes a comma separated pair and creates a couple of
+fields:
+* `original_string` -- the raw data
+* `timestamp` -- the current time
+* `first` -- the first field of the comma separated pair
+* `last` -- the last field of the comma separated pair
+
+For this demonstration, let's create a maven project to compile our
+project.  We'll call it `extra_parsers`, so in your workspace, let's set
+up the maven project:
+* Create the maven infrastructure for `extra_parsers` via
+```
+mkdir -p extra_parsers/src/{main,test}/java
+```
+* Create a pom file indicating how we should build our parsers by
+  editing `extra_parsers/pom.xml` with the following content:
+```
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+  4.0.0
+  com.3rdparty
+  extra-parsers
+  jar
+  1.0-SNAPSHOT
+  extra-parsers
+  http://thirdpartysoftware.org
+  
+
+1.8
+
+0.4.1
+
+19.0
+
+2.4.3
+  
+  
+
+
+  org.apache.metron
+  metron-parsers
+  ${metron_version}
+  provided
+
+
+  com.google.guava
+  guava
+  ${guava_version}
+
+
+  junit
+  junit
+  3.8.1
+  test
+
+  
+  
+
+ 
+
+  
+org.apache.maven.plugins
+maven-shade-plugin
+${shade_version}
+
+  true
+  
+
+  
+  *slf4j*
+
+  
+
+
+  
+package
+
+  shade
+
+
+  true
+  uber
+  
+
+  
+  *:*
+  
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+  
+
+
+  com.google
+  com.thirdparty.guava
+
+  
+  
+
+  
+  storm:storm-core:*
+  storm:storm-lib:*
+  org.slf4j.impl*
+  org.slf4j:slf4j-log4j*
+
+  
+
+  
+
+  
+  
+  
+org.apache.maven.plugins
+maven-compiler-plugin
+3.5.1
+
+  true
+  ${java_version}
+  -Xlint:unchecked
+  ${java_version}
+  true
+
+  
+
+  
+
+```
+* Now let's create our parser  `com.thirdparty.SimpleParser` by creating 
the file `extra-parsers/src/main/java/com/thirdparty/SimpleParser.java` with 
the following content:
+```
+package 

[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/850
  
please take care to close the jira.  Thanks again


---

[GitHub] metron pull request #850: METRON-1335: Install metron-maas-service RPM as a ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/850


---

[GitHub] metron pull request #836: METRON-1308: Fix Metron Documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/836


---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
Merged master, ran tests successfully, built site-books and did some 
clicking around.  Going to merge.


---

[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/850
  
Sure, that would be great.


---

[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/850
  
Do you need me to land it?


---

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

Where are you seeing the errors?  Screenshot?


On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Which of the indexing options are you changing the batch size for?  HDFS?
Elasticsearch?  Both?

Can you give an example?



On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote:

No specific error in the logs. I haven't enabled debug/trace, though.

On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
wrote:

> My first thought is what are the errors when you get a high error rate?
>
>
> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Any thoughts?
>
> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
> wrote:
>
> > Hi,
> >
> > We have noticed recently that no matter what batch size we use for Metron
> > indexing feeds, as long as we start using different batch size for
> > different Metron feeds, indexing topology throughput will start dropping
> > due to the high error rate! So I was wondering whether based on the
> current
> > indexing topology design, we have to choose the same batch size for all
> the
> > feeds or not. Otherwise, throughout will be dropped. I assume since it is
> > acceptable to use different batch sizes for different feeds, it is not
> > expected by design.
> >
> > Moreover, I have noticed in practice that even if we change the batch
> > size, it will not affect the messages that are already in enrichments or
> > indexing topics, and it will only affect the new messages that are coming
> > to the parser. Therefore, we need to let all the messages pass the
> indexing
> > topology so that we can change the batch size!
> >
> > It would be great if we can have more details regarding the design of
> this
> > section so we can understand our observations are based on the design or
> > some kind of bug.
> >
> > Regards,
> > Ali
> >
>
>
>
> --
> A.Nazemian
>
>


--
A.Nazemian

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

Which of the indexing options are you changing the batch size for?  HDFS?
Elasticsearch?  Both?

Can you give an example?



On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote:

No specific error in the logs. I haven't enabled debug/trace, though.

On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
wrote:

> My first thought is what are the errors when you get a high error rate?
>
>
> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Any thoughts?
>
> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
> wrote:
>
> > Hi,
> >
> > We have noticed recently that no matter what batch size we use for Metron
> > indexing feeds, as long as we start using different batch size for
> > different Metron feeds, indexing topology throughput will start dropping
> > due to the high error rate! So I was wondering whether based on the
> current
> > indexing topology design, we have to choose the same batch size for all
> the
> > feeds or not. Otherwise, throughout will be dropped. I assume since it is
> > acceptable to use different batch sizes for different feeds, it is not
> > expected by design.
> >
> > Moreover, I have noticed in practice that even if we change the batch
> > size, it will not affect the messages that are already in enrichments or
> > indexing topics, and it will only affect the new messages that are coming
> > to the parser. Therefore, we need to let all the messages pass the
> indexing
> > topology so that we can change the batch size!
> >
> > It would be great if we can have more details regarding the design of
> this
> > section so we can understand our observations are based on the design or
> > some kind of bug.
> >
> > Regards,
> > Ali
> >
>
>
>
> --
> A.Nazemian
>
>


--
A.Nazemian

[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/850
  
Thanks for the contribution, installed as described in full dev.
+1


---

[GitHub] metron issue #850: METRON-1335: Install metron-maas-service RPM as a part of...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/850
  
I'm running it up now



---