date:20181003



[ 
https://issues.apache.org/jira/browse/METRON-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637544#comment-16637544
 ] 

ASF GitHub Bot commented on METRON-1805:


GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/1221

METRON-1805: Provide a default value for the Storm 
topology.max.spout.pending setting

## Contributor Comments
Users have reported problems in the random and batch indexing topologies 
when this setting is not set.  The primary purpose of this PR is to start a 
discussion around providing a default value for the Storm 
topology.max.spout.pending setting in our topologies and implement the changes 
if we decide to do it.  

1. Should we provide defaults or continue to defer to the Storm default 
where a maximum isn't enforced?
2. Which topologies should we provide defaults for?
3. What should the default values be for the topologies we include?

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1805

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1221.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1221


commit 6988e0607afc1cd402063e0696a1220cc134d9d1
Author: merrimanr 
Date:   2018-10-03T21:45:32Z

initial commit




> Provide a default value for the Storm topology.max.spout.pending setting
> 
>
> Key: METRON-1805
> URL: https://issues.apache.org/jira/browse/METRON-1805
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Should we provide a default value for this property?  Which topologies should 
> that apply to?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1221: METRON-1805: Provide a default value for the Stor...

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/1221

METRON-1805: Provide a default value for the Storm 
topology.max.spout.pending setting

## Contributor Comments
Users have reported problems in the random and batch indexing topologies 
when this setting is not set.  The primary purpose of this PR is to start a 
discussion around providing a default value for the Storm 
topology.max.spout.pending setting in our topologies and implement the changes 
if we decide to do it.  

1. Should we provide defaults or continue to defer to the Storm default 
where a maximum isn't enforced?
2. Which topologies should we provide defaults for?
3. What should the default values be for the topologies we include?

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1805

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1221.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1221


commit 6988e0607afc1cd402063e0696a1220cc134d9d1
Author: merrimanr 
Date:   2018-10-03T21:45:32Z

initial commit




---

[jira] [Created] (METRON-1805) Provide a default value for the Storm topology.max.spout.pending setting

2018-10-03 Thread Ryan Merriman (JIRA)

Ryan Merriman created METRON-1805:
-

 Summary: Provide a default value for the Storm 
topology.max.spout.pending setting
 Key: METRON-1805
 URL: https://issues.apache.org/jira/browse/METRON-1805
 Project: Metron
  Issue Type: Improvement
Reporter: Ryan Merriman


Should we provide a default value for this property?  Which topologies should 
that apply to?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637512#comment-16637512
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
I just pushed a commit out based on some feedback from Nick: 

- The stellar context is now passed into the ParserRunner abstraction as a 
dependency.  I imagine different environments will require different Stellar 
initialization strategies so this made sense to me.  I also like it because it 
makes the abstraction simpler.
-  Changed ParserRunner to an interface with a single implementation.  I 
could change this to ParserRunnerStrategy in case we think we might want to use 
that pattern in the future.  Then all that would be missing is a ParserContext 
class (and possibly a ParserRunnerStrategies enum) which we could add now or 
when we need it.


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
I just pushed a commit out based on some feedback from Nick: 

- The stellar context is now passed into the ParserRunner abstraction as a 
dependency.  I imagine different environments will require different Stellar 
initialization strategies so this made sense to me.  I also like it because it 
makes the abstraction simpler.
-  Changed ParserRunner to an interface with a single implementation.  I 
could change this to ParserRunnerStrategy in case we think we might want to use 
that pattern in the future.  Then all that would be missing is a ParserContext 
class (and possibly a ParserRunnerStrategies enum) which we could add now or 
when we need it.


---

[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates



[ 
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637507#comment-16637507
 ] 

ASF GitHub Bot commented on METRON-1771:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222470180
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
 ---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String 
sensorType) throws IOExcepti
   public List getIndices() {
 return indices;
   }
+
+  private Document getDocument(List documentContainers) 
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+  if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+  }
+  else {
+if(dc.getDocument().isPresent()) {
+  Document d = dc.getDocument().get();
+  if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+  }
+}
+  }
+}
--- End diff --

No problem, don't mind working it out here and now.  The case that causes 
the test to fail (although I'm not sure that was the intention of the test) is 
where 2 DAOs are contained in a MultiIndexDao and the first DAO returns an 
invalid result (no document or exception) while the second returns a valid 
result.  I don't know what would cause a document to be missing both a document 
and an exception.  

The question is what should happen when this unexpected condition does 
happen?  Should it blow up and throw and exception or return a document if at 
least one DAO returns a valid one.


> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return 
> true/false or nothing.  This puts the responsibility of retrieving the 
> updated state of the object on the client in a separate call or 
> optimistically applying the changes and reverting when an update fails.  This 
> can be problematic if a client attempts to get the current state immediately 
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the 
> need to look up the updated state in a separate call.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222470180
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
 ---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String 
sensorType) throws IOExcepti
   public List getIndices() {
 return indices;
   }
+
+  private Document getDocument(List documentContainers) 
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+  if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+  }
+  else {
+if(dc.getDocument().isPresent()) {
+  Document d = dc.getDocument().get();
+  if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+  }
+}
+  }
+}
--- End diff --

No problem, don't mind working it out here and now.  The case that causes 
the test to fail (although I'm not sure that was the intention of the test) is 
where 2 DAOs are contained in a MultiIndexDao and the first DAO returns an 
invalid result (no document or exception) while the second returns a valid 
result.  I don't know what would cause a document to be missing both a document 
and an exception.  

The question is what should happen when this unexpected condition does 
happen?  Should it blow up and throw and exception or return a document if at 
least one DAO returns a valid one.


---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637474#comment-16637474
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
Thanks for pointing me to that @mmiklavc, I had not looked at it yet.  I am 
in agreement that we should do our best to match other classes and patterns we 
already have.  I will continue studying that to see where we can make things 
more consistent.  If you can provide some examples that demonstrate your ideas 
that would help me understand it better.  

From what I can tell, the strategy pattern was used in the 
UnifiedEnrichmentBolt because there are 2 separate strategies for enriching a 
message:  enrichment and threat intel.  Do we need the strategy pattern here 
since there is only one parser running strategy?  We could implement this as a 
strategy pattern in anticipation of one day needing another parser running 
strategy.  Do you think it's worth doing now or later when we need it?

The other difference I see is that Future objects are used to join the 
different messages after they are processed in parallel.  I believe this is 
done because all enrichments/threat intel must be done before the message can 
be pieced back together and sent along.  In this case we don't have that 
limitation.  The documents that are parsed do not depend on each other and can 
be passed along as soon as they are processed.  We are set up for 
post-processing these documents in parallel but I see that as a low-latency 
operation and I'm not sure it's worth the extra overhead.




> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
Thanks for pointing me to that @mmiklavc, I had not looked at it yet.  I am 
in agreement that we should do our best to match other classes and patterns we 
already have.  I will continue studying that to see where we can make things 
more consistent.  If you can provide some examples that demonstrate your ideas 
that would help me understand it better.  

From what I can tell, the strategy pattern was used in the 
UnifiedEnrichmentBolt because there are 2 separate strategies for enriching a 
message:  enrichment and threat intel.  Do we need the strategy pattern here 
since there is only one parser running strategy?  We could implement this as a 
strategy pattern in anticipation of one day needing another parser running 
strategy.  Do you think it's worth doing now or later when we need it?

The other difference I see is that Future objects are used to join the 
different messages after they are processed in parallel.  I believe this is 
done because all enrichments/threat intel must be done before the message can 
be pieced back together and sent along.  In this case we don't have that 
limitation.  The documents that are parsed do not depend on each other and can 
be passed along as soon as they are processed.  We are set up for 
post-processing these documents in parallel but I see that as a low-latency 
operation and I'm not sure it's worth the extra overhead.




---

[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222440594
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
+
+  protected transient Consumer onSuccess;
+  protected transient Consumer onError;
+
+  private HashSet sensorTypes;
+  private Map sensorToParserComponentMap;
+
+  // Stellar variables
+  private transient Context stellarContext;
--- End diff --

Context is not serializable.


---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637431#comment-16637431
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222440594
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
+
+  protected transient Consumer onSuccess;
+  protected transient Consumer onError;
+
+  private HashSet sensorTypes;
+  private Map sensorToParserComponentMap;
+
+  // Stellar variables
+  private transient Context stellarContext;
--- End diff --

Context is not serializable.


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1804) Update version to 0.6.1



[ 
https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637415#comment-16637415
 ] 

ASF GitHub Bot commented on METRON-1804:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1220


> Update version to 0.6.1
> ---
>
> Key: METRON-1804
> URL: https://issues.apache.org/jira/browse/METRON-1804
> Project: Metron
>  Issue Type: Task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Post release version bump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1220: METRON-1804: Update version to 0.6.1

2018-10-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1220


---

[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.



[ 
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637393#comment-16637393
 ] 

ASF GitHub Bot commented on METRON-1761:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1184
  
@mmiklavc  but we don't have messages to split, we have bytes.  If we where 
going to leave the 'parser's as single object -> single result | single 
exceception', ie not change the interface and not subvert validate, then we 
would have to introduce 'splitStrategies' at the bolt.


> Allow a grok statement to be applied to each line in a file.
> 
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
>  Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be 
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is 
> to be parsed separately) and send them to Metron without having to split the 
> content.
> Example content of a log file where a grok statement needs to be applied to 
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ 
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ 
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-" 
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...

2018-10-03 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1184
  
@mmiklavc  but we don't have messages to split, we have bytes.  If we where 
going to leave the 'parser's as single object -> single result | single 
exceception', ie not change the interface and not subvert validate, then we 
would have to introduce 'splitStrategies' at the bolt.


---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637376#comment-16637376
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1213
  
> @mmiklavc Have you looked over this by chance [2]? The reason I bring 
this up is because it looks like it already addresses a number of the 
points/questions brought up in this PR's discussion. For example, there's a 
strategy for handling successful messages as well as error results [3]. If 
we're going through the effort of refactoring, it's probably worth our while to 
match the other idioms to some degree. It just makes continued maintenance that 
much easier going forward.

Are you making the case for returning a `List` here then?  Sounds like 
you're thinking along these lines?
```
ParserRunner {
   List execute(...);
} 
```

That would be similar to the unified enrichment work that you linked to in 
your comment.
```
ParallelEnricher {
  EnrichmentResult apply(...);
}
```


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1213
  
> @mmiklavc Have you looked over this by chance [2]? The reason I bring 
this up is because it looks like it already addresses a number of the 
points/questions brought up in this PR's discussion. For example, there's a 
strategy for handling successful messages as well as error results [3]. If 
we're going through the effort of refactoring, it's probably worth our while to 
match the other idioms to some degree. It just makes continued maintenance that 
much easier going forward.

Are you making the case for returning a `List` here then?  Sounds like 
you're thinking along these lines?
```
ParserRunner {
   List execute(...);
} 
```

That would be similar to the unified enrichment work that you linked to in 
your comment.
```
ParallelEnricher {
  EnrichmentResult apply(...);
}
```


---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637363#comment-16637363
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222423266
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
--- End diff --

I don't get `ParserContext`.  The `ParserRunner` is something that 
runs/executes parsers, no?  It doesn't provide context for something else to 
run parsers.  But maybe I am misunderstanding why you think `ParserContext` 
makes sense.  Can you explain your thoughts on that?


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222423266
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
--- End diff --

I don't get `ParserContext`.  The `ParserRunner` is something that 
runs/executes parsers, no?  It doesn't provide context for something else to 
run parsers.  But maybe I am misunderstanding why you think `ParserContext` 
makes sense.  Can you explain your thoughts on that?


---

[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates



[ 
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637356#comment-16637356
 ] 

ASF GitHub Bot commented on METRON-1771:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222421545
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --

I think you missed the javadoc here.


> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return 
> true/false or nothing.  This puts the responsibility of retrieving the 
> updated state of the object on the client in a separate call or 
> optimistically applying the changes and reverting when an update fails.  This 
> can be problematic if a client attempts to get the current state immediately 
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the 
> need to look up the updated state in a separate call.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222421545
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --

I think you missed the javadoc here.


---

[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates



[ 
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637352#comment-16637352
 ] 

ASF GitHub Bot commented on METRON-1771:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222420697
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
 ---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String 
sensorType) throws IOExcepti
   public List getIndices() {
 return indices;
   }
+
+  private Document getDocument(List documentContainers) 
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+  if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+  }
+  else {
+if(dc.getDocument().isPresent()) {
+  Document d = dc.getDocument().get();
+  if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+  }
+}
+  }
+}
--- End diff --

Hmm. Interesting.  I don't really understand what happens in that else 
condition then.  

The OCD side of me wants to make sure I understand this to ensure there is 
not a pre-existing bug.  But at the same time I can't fault you for this.  All 
you did was refactor this code out into a method.

It seems that a `DocumentContainer` should either have an exception 
(`dc.getException().isPresent()`) or it should have a document 
(`dc.getDocument.isPresent()`).  We would hit the else when neither of those 
are the case.  So under what conditions would a `DocumentContainer` have 
neither of those?  

I guess a `null` Document or null Throwable is added to the 
`DocumentContainer`?  I hope this is what we expect to happen.

```
if(dc.getException().isPresent()) { 
  ..
} else if(dc.getDocument.isPresent()) {
  ..
} else {
 // when does this occur? what do we expect to happen?
}
```


> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return 
> true/false or nothing.  This puts the responsibility of retrieving the 
> updated state of the object on the client in a separate call or 
> optimistically applying the changes and reverting when an update fails.  This 
> can be problematic if a client attempts to get the current state immediately 
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the 
> need to look up the updated state in a separate call.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari



[ 
https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637354#comment-16637354
 ] 

ASF GitHub Bot commented on METRON-1695:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1207
  
@nickwallen suggested a great idea of making use of parser aggregation and 
thus starting the default parsers on full-dev as an aggregated parser topology. 
The latest commit makes use of PR #1215 which provides that ability.

Now, on full-dev we start a single aggregated topology for the default 
configured sensors viz. "bro,snort,yaf".

## Testing Done
- Spun up full-dev and noticed that a single topology named 
`bro__snort__yaf` is started
- Verified Alerts UI has data from all three sensors:


![image](https://user-images.githubusercontent.com/20395490/46431349-33d6aa80-c769-11e8-8b6e-33940ef9b9eb.png)



> Expose pcap properties through Ambari
> -
>
> Key: METRON-1695
> URL: https://issues.apache.org/jira/browse/METRON-1695
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the 
> defaults. One has to hand edit the file before deploying the PCAP topology. 
> These properties should be configurable via Ambari.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1207
  
@nickwallen suggested a great idea of making use of parser aggregation and 
thus starting the default parsers on full-dev as an aggregated parser topology. 
The latest commit makes use of PR #1215 which provides that ability.

Now, on full-dev we start a single aggregated topology for the default 
configured sensors viz. "bro,snort,yaf".

## Testing Done
- Spun up full-dev and noticed that a single topology named 
`bro__snort__yaf` is started
- Verified Alerts UI has data from all three sensors:


![image](https://user-images.githubusercontent.com/20395490/46431349-33d6aa80-c769-11e8-8b6e-33940ef9b9eb.png)



---

[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1190#discussion_r222420697
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
 ---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String 
sensorType) throws IOExcepti
   public List getIndices() {
 return indices;
   }
+
+  private Document getDocument(List documentContainers) 
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+  if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+  }
+  else {
+if(dc.getDocument().isPresent()) {
+  Document d = dc.getDocument().get();
+  if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+  }
+}
+  }
+}
--- End diff --

Hmm. Interesting.  I don't really understand what happens in that else 
condition then.  

The OCD side of me wants to make sure I understand this to ensure there is 
not a pre-existing bug.  But at the same time I can't fault you for this.  All 
you did was refactor this code out into a method.

It seems that a `DocumentContainer` should either have an exception 
(`dc.getException().isPresent()`) or it should have a document 
(`dc.getDocument.isPresent()`).  We would hit the else when neither of those 
are the case.  So under what conditions would a `DocumentContainer` have 
neither of those?  

I guess a `null` Document or null Throwable is added to the 
`DocumentContainer`?  I hope this is what we expect to happen.

```
if(dc.getException().isPresent()) { 
  ..
} else if(dc.getDocument.isPresent()) {
  ..
} else {
 // when does this occur? what do we expect to happen?
}
```


---

[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari



[ 
https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637343#comment-16637343
 ] 

ASF GitHub Bot commented on METRON-1695:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1207
  
@MohanDV , as a part of my commit, I have moved all the UI related elements 
from REST to the PCAP config tab. 

In my understanding, the following parameters are not directly related to 
the PCAP core service, but are functionally related to REST (used by the PCAP 
query panel UI):
* pdml.script.path
* base.path
* base.interim.result.path
* final.output.path
* page.size
* yarn.queue
* finalizer.threadpool.size

For this reason, I did not move these properties out of the 
rest_application.yml. 

> @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology 

With my current changes, modifying any config setting in the PCAP tab will 
require restarting both Metron PCAP and Metron REST services--which is 
necessary since user can change a PCAP-service related property or a PCAP-REST 
related property (or both). The converse, which you noted, i.e. changing any 
REST config does not require restarting the PCAP topology. Here's a screenshot 
after making changes to REST config...


![image](https://user-images.githubusercontent.com/20395490/46430735-8e6f0700-c767-11e8-8308-728771635480.png)



> Expose pcap properties through Ambari
> -
>
> Key: METRON-1695
> URL: https://issues.apache.org/jira/browse/METRON-1695
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the 
> defaults. One has to hand edit the file before deploying the PCAP topology. 
> These properties should be configurable via Ambari.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1207
  
@MohanDV , as a part of my commit, I have moved all the UI related elements 
from REST to the PCAP config tab. 

In my understanding, the following parameters are not directly related to 
the PCAP core service, but are functionally related to REST (used by the PCAP 
query panel UI):
* pdml.script.path
* base.path
* base.interim.result.path
* final.output.path
* page.size
* yarn.queue
* finalizer.threadpool.size

For this reason, I did not move these properties out of the 
rest_application.yml. 

> @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology 

With my current changes, modifying any config setting in the PCAP tab will 
require restarting both Metron PCAP and Metron REST services--which is 
necessary since user can change a PCAP-service related property or a PCAP-REST 
related property (or both). The converse, which you noted, i.e. changing any 
REST config does not require restarting the PCAP topology. Here's a screenshot 
after making changes to REST config...


![image](https://user-images.githubusercontent.com/20395490/46430735-8e6f0700-c767-11e8-8308-728771635480.png)



---

[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.



[ 
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637336#comment-16637336
 ] 

ASF GitHub Bot commented on METRON-1761:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1184
  
@ottobackwards I see what you're saying. It looks like that could 
definitely work. Thinking out loud here, but might that conflate the semantics 
of our validation a bit? Validate currently does things like ensure that a 
timestamp exists on the message, though I don't see why we couldn't expand it 
to validations outside of our global Metron context.

One class that might be worth checking out is the unified enrichment 
topology. This was changed to include a parallel enricher that handles errors 
and message results in an EnrichmentResult class.

1. 
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270
2. 
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63

It looks to me like there might be some possible collaboration opportunity 
and overlap with the work you're doing here and the work @merrimanr is doing on 
this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142

I'm just wondering if we might be able to kill 2 birds with one stone. We 
probably don't want to change the MessageParser interface, but maybe we can 
manage the bulk processing through a more generalized bridge between the 
ParserBolt and parser implementations. I haven't dug too deep into 
implementation feasibility, but it seems worth considering.


> Allow a grok statement to be applied to each line in a file.
> 
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
>  Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be 
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is 
> to be parsed separately) and send them to Metron without having to split the 
> content.
> Example content of a log file where a grok statement needs to be applied to 
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ 
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ 
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-" 
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...

2018-10-03 Thread mmiklavc

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1184

@ottobackwards I see what you're saying. It looks like that could
definitely work. Thinking out loud here, but might that conflate the semantics
of our validation a bit? Validate currently does things like ensure that a
timestamp exists on the message, though I don't see why we couldn't expand it
to validations outside of our global Metron context.

One class that might be worth checking out is the unified enrichment
topology. This was changed to include a parallel enricher that handles errors
and message results in an EnrichmentResult class.

1.
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270
2.
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63

It looks to me like there might be some possible collaboration opportunity
and overlap with the work you're doing here and the work @merrimanr is doing on
this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142

I'm just wondering if we might be able to kill 2 birds with one stone. We
probably don't want to change the MessageParser interface, but maybe we can
manage the bulk processing through a more generalized bridge between the
ParserBolt and parser implementations. I haven't dug too deep into
implementation feasibility, but it seems worth considering.

---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637313#comment-16637313
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222397888
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
--- End diff --

Maybe this should be "Context"


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637312#comment-16637312
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222376135
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
+
+  protected transient Consumer onSuccess;
+  protected transient Consumer onError;
+
+  private HashSet sensorTypes;
+  private Map sensorToParserComponentMap;
+
+  // Stellar variables
+  private transient Context stellarContext;
--- End diff --

Why the change to making this transient?


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...

2018-10-03 Thread mmiklavc

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222376135
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
+
+  protected transient Consumer onSuccess;
+  protected transient Consumer onError;
+
+  private HashSet sensorTypes;
+  private Map sensorToParserComponentMap;
+
+  // Stellar variables
+  private transient Context stellarContext;
--- End diff --

Why the change to making this transient?


---

[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...

2018-10-03 Thread mmiklavc

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1213#discussion_r222397888
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.FieldValidator;
+import org.apache.metron.common.configuration.ParserConfigurations;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.metadata.RawMessage;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.parsers.filters.Filters;
+import org.apache.metron.parsers.interfaces.MessageFilter;
+import org.apache.metron.parsers.interfaces.MessageParser;
+import org.apache.metron.parsers.topology.ParserComponent;
+import org.apache.metron.stellar.common.CachingStellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.json.simple.JSONObject;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ParserRunner implements Serializable {
--- End diff --

Maybe this should be "Context"


---

[jira] [Commented] (METRON-1804) Update version to 0.6.1



[ 
https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637306#comment-16637306
 ] 

ASF GitHub Bot commented on METRON-1804:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1220
  
Looks good.  Thanks! +1


> Update version to 0.6.1
> ---
>
> Key: METRON-1804
> URL: https://issues.apache.org/jira/browse/METRON-1804
> Project: Metron
>  Issue Type: Task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Post release version bump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1220: METRON-1804: Update version to 0.6.1

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1220
  
Looks good.  Thanks! +1


---

[jira] [Commented] (METRON-1804) Update version to 0.6.1



[ 
https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637120#comment-16637120
 ] 

ASF GitHub Bot commented on METRON-1804:


GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/1220

METRON-1804: Update version to 0.6.1

## Contributor Comments
Updating version to 0.6.1 post release.

Changed per 
https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number

0.6.0 references should be gone (barring site stuff, other dependencies 
like Zeppelin, etc.).  Pom and doc references should all be updated.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron version061

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1220


commit efb46065aa4a1b5618489c7e7948859f60c2f6f6
Author: justinjleet 
Date:   2018-09-18T16:03:58Z

Upping version ot 0.6.1




> Update version to 0.6.1
> ---
>
> Key: METRON-1804
> URL: https://issues.apache.org/jira/browse/METRON-1804
> Project: Metron
>  Issue Type: Task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Post release version bump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1220: METRON-1804: Update version to 0.6.1

2018-10-03 Thread justinleet

GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/1220

METRON-1804: Update version to 0.6.1

## Contributor Comments
Updating version to 0.6.1 post release.

Changed per 
https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number

0.6.0 references should be gone (barring site stuff, other dependencies 
like Zeppelin, etc.).  Pom and doc references should all be updated.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron version061

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1220


commit efb46065aa4a1b5618489c7e7948859f60c2f6f6
Author: justinjleet 
Date:   2018-09-18T16:03:58Z

Upping version ot 0.6.1




---

[jira] [Created] (METRON-1804) Update version to 0.6.1

2018-10-03 Thread Justin Leet (JIRA)

Justin Leet created METRON-1804:
---

 Summary: Update version to 0.6.1
 Key: METRON-1804
 URL: https://issues.apache.org/jira/browse/METRON-1804
 Project: Metron
  Issue Type: Task
Reporter: Justin Leet
Assignee: Justin Leet


Post release version bump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637047#comment-16637047
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
That sounds correct to me.  Specifically decoupling the Kafka writing step 
since our writing implementation requires a tuple.


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1213
  
That sounds correct to me.  Specifically decoupling the Kafka writing step 
since our writing implementation requires a tuple.


---

[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic



[ 
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637041#comment-16637041
 ] 

ASF GitHub Bot commented on METRON-1681:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1213
  
@merrimanr @nickwallen @justinleet - I just want to be sure that I'm 
following this correctly. This particular callback appears to be a synchronous, 
blocking callback. The crux of the implementation is that you're passing in a 
function pointer to be executed when the parsing either succeeds or fails in 
order to decouple the parsing logic from Storm itself (e.g. tuples, etc.)?


> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are 
> concerns about the ParserBolt needed some refactoring.  The discussion didn't 
> hold the PR up, but it was generally agreed that we should decouple some of 
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.



[ 
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637040#comment-16637040
 ] 

ASF GitHub Bot commented on METRON-1761:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1184
  
@mmiklavc I looked through the validation stuff more, I think that 
validation is the way to go here.  The grok parser will add invalid message for 
each exception, parser failure, and then in the validation call fail those 
messages.  It will have to be done so that the returned message makes sense 
when it is sent to the error topic.  

What do you think?   @cestella ?


> Allow a grok statement to be applied to each line in a file.
> 
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
>  Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be 
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is 
> to be parsed separately) and send them to Metron without having to split the 
> content.
> Example content of a log file where a grok statement needs to be applied to 
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ 
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ 
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-" 
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...

2018-10-03 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1184
  
@mmiklavc I looked through the validation stuff more, I think that 
validation is the way to go here.  The grok parser will add invalid message for 
each exception, parser failure, and then in the validation call fail those 
messages.  It will have to be done so that the returned message makes sense 
when it is sent to the error topic.  

What do you think?   @cestella ?


---

[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI



[ 
https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637036#comment-16637036
 ] 

ASF GitHub Bot commented on METRON-1749:


Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1217
  
@mmiklavc I was unable to find a discuss thread. I understand this is 
standard for a change like this and I would be more than happy to start one if 
the community desires, otherwise I'm hoping we can have that discussion here. 
There is now a brief explanation in the Contributor Comments for why we should 
upgrade the Management UI like we did the Alerts UI.


> Update Angular to latest release in Management UI
> -
>
> Key: METRON-1749
> URL: https://issues.apache.org/jira/browse/METRON-1749
> Project: Metron
>  Issue Type: Improvement
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Major
>
> Currently, the Management UI is on Angular v2. Not only has there been many 
> updates and improvements to Angular, that release is no longer supported by 
> the Angular team. This means critical fixes and security patches are no 
> longer being done.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...

2018-10-03 Thread sardell

Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1217
  
@mmiklavc I was unable to find a discuss thread. I understand this is 
standard for a change like this and I would be more than happy to start one if 
the community desires, otherwise I'm hoping we can have that discussion here. 
There is now a brief explanation in the Contributor Comments for why we should 
upgrade the Management UI like we did the Alerts UI.


---

[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js



[ 
https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636991#comment-16636991
 ] 

ASF GitHub Bot commented on METRON-1796:


Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1219#discussion_r222317156
  
--- Diff: metron-interface/metron-alerts/src/app/utils/utils.spec.ts ---
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Utils} from './utils'
+
+describe('timeRangeToDisplayStr', () => {
--- End diff --


[timeRangeToDisplayStr](https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/utils/utils.ts#L77-L206)
 relies heavily on moment.js so before touching the implementation I covered it 
with unit tests to make sure that the util produces the exact same result after 
the removal of moment.js.


> [UI] Migrate off moment.js
> --
>
> Key: METRON-1796
> URL: https://issues.apache.org/jira/browse/METRON-1796
> Project: Metron
>  Issue Type: Improvement
>Reporter: Tamas Fodor
>Assignee: Tamas Fodor
>Priority: Minor
>
> Remove Moment.js and replace with another smaller library.
> Moment.js requires us to import the entire library vs. a few necessary 
> modules.
> Moment.js can prevent bundlers from supporting tree-shaking.
> By removing Moment.js, we can decrease our overall bundle size and prevent 
> issues with tree-shaking in the future.
> Here you can find the discussion on the mailing list:
> https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js



[ 
https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636992#comment-16636992
 ] 

ASF GitHub Bot commented on METRON-1796:


Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1219#discussion_r222315533
  
--- Diff: metron-interface/metron-alerts/package.json ---
@@ -46,9 +46,7 @@
 "@types/ace": "0.0.32",
 "@types/es6-promise": "0.0.33",
 "@types/jasmine": "2.5.38",
-"@types/moment": "^2.13.0",
 "@types/node": "~6.0.60",
-"@types/pikaday-time": "^1.4.2",
--- End diff --

The reason why I removed this is because moment.js is the dependency of 
this to make it able to add the type `moment` to the return value of the 
`getMoment` method. Therefore moment.js has to be inside the node_modules 
folder but we wan't to avoid it to be there. 
I don't see any benefits of having the type definitions of pikaday-time in 
the code base.


> [UI] Migrate off moment.js
> --
>
> Key: METRON-1796
> URL: https://issues.apache.org/jira/browse/METRON-1796
> Project: Metron
>  Issue Type: Improvement
>Reporter: Tamas Fodor
>Assignee: Tamas Fodor
>Priority: Minor
>
> Remove Moment.js and replace with another smaller library.
> Moment.js requires us to import the entire library vs. a few necessary 
> modules.
> Moment.js can prevent bundlers from supporting tree-shaking.
> By removing Moment.js, we can decrease our overall bundle size and prevent 
> issues with tree-shaking in the future.
> Here you can find the discussion on the mailing list:
> https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1219#discussion_r222315533
  
--- Diff: metron-interface/metron-alerts/package.json ---
@@ -46,9 +46,7 @@
 "@types/ace": "0.0.32",
 "@types/es6-promise": "0.0.33",
 "@types/jasmine": "2.5.38",
-"@types/moment": "^2.13.0",
 "@types/node": "~6.0.60",
-"@types/pikaday-time": "^1.4.2",
--- End diff --

The reason why I removed this is because moment.js is the dependency of 
this to make it able to add the type `moment` to the return value of the 
`getMoment` method. Therefore moment.js has to be inside the node_modules 
folder but we wan't to avoid it to be there. 
I don't see any benefits of having the type definitions of pikaday-time in 
the code base.


---

[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1219#discussion_r222317156
  
--- Diff: metron-interface/metron-alerts/src/app/utils/utils.spec.ts ---
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Utils} from './utils'
+
+describe('timeRangeToDisplayStr', () => {
--- End diff --


[timeRangeToDisplayStr](https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/utils/utils.ts#L77-L206)
 relies heavily on moment.js so before touching the implementation I covered it 
with unit tests to make sure that the util produces the exact same result after 
the removal of moment.js.


---

[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js

Github user ruffle1986 commented on a diff in the pull request:

https://github.com/apache/metron/pull/1219#discussion_r222308215
  
--- Diff: metron-interface/metron-alerts/package.json ---
@@ -22,17 +22,17 @@
 "@angular/platform-browser": "^6.1.6",
 "@angular/platform-browser-dynamic": "^6.1.6",
 "@angular/router": "^6.1.6",
+"@ruffle1986/pikaday-time": "^1.6.1",
 "@types/bootstrap": "^4.1.1",
 "@types/jquery": "^3.3.4",
 "ace-builds": "^1.2.6",
 "ajv": "^6.5.1",
 "angular-confirmation-popover": "^4.2.0",
 "bootstrap": "4.0.0-alpha.6",
 "core-js": "^2.4.1",
+"date-fns": "^1.29.0",
 "font-awesome": "^4.7.0",
-"moment": "^2.22.2",
 "ng2-dragula": "^1.5.0",
-"pikaday-time": "^1.6.1",
--- End diff --

pikaday-time is an extension of 
[Pikaday](https://github.com/dbushell/Pikaday). Pikaday is a great library with 
zero dependency. Pikaday-time is extends its functionality with time selection. 
However moment is just an optional dependency of Pikaday, there's an unpleasant 
side-effect in Pikaday-time where moment.js is set as a dependency in the 
package.json. So everytime we install pikaday-time, moment.js will be installed 
as well.

[I've tried to reach out the maintainer of the 
pikaday-time](https://github.com/owenmead/Pikaday/issues/60) library but he's 
not so active on Github and the library is no longer maintained anyway so it 
makes things a bit complicated. Hopefully we'll manage this and a patch for 
this will be introduced shortly.

Until then, [I've forked the pikaday-time 
repo](https://github.com/owenmead/Pikaday/compare/master...ruffle1986:master), 
made the changes and [published it on npm under my 
name](https://www.npmjs.com/package/@ruffle1986/pikaday-time). Originally I 
wanted to published it under 
[hortonworks](https://www.npmjs.com/org/hortonworks) but I don't know who have 
access to give me publish rights. The only change I made is setting moment.js 
as a `peerDependency` insteadOf setting it as an `optionalDependency`. Don't 
let the name `optionalDependency` mislead you. [It's basically a dependency but 
if it's cannot be found on npm on any given registries, npm install doesn't 
fail](https://docs.npmjs.com/files/package.json#optionaldependencies).

As soon as this issue is fixed and published on npm, we can remove the 
scoped pikaday-time and switch back to the original package.


---

[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...

2018-10-03 Thread sardell

Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1217
  
@mmiklavc Thanks for starting to take a look at this PR!

You're correct that all these changes are related to the upgrade. This 
means removing deprecated items, updating syntax, utilizing updated or new 
features where we should and accommodating for much stricter type checking and 
compiling. 


---

[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI



[ 
https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636951#comment-16636951
 ] 

ASF GitHub Bot commented on METRON-1749:


Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1217
  
@mmiklavc Thanks for starting to take a look at this PR!

You're correct that all these changes are related to the upgrade. This 
means removing deprecated items, updating syntax, utilizing updated or new 
features where we should and accommodating for much stricter type checking and 
compiling. 


> Update Angular to latest release in Management UI
> -
>
> Key: METRON-1749
> URL: https://issues.apache.org/jira/browse/METRON-1749
> Project: Metron
>  Issue Type: Improvement
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Major
>
> Currently, the Management UI is on Angular v2. Not only has there been many 
> updates and improvements to Angular, that release is no longer supported by 
> the Angular team. This means critical fixes and security patches are no 
> longer being done.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636948#comment-16636948
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1215


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

2018-10-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1215


---

[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js

[
https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636946#comment-16636946
]

ASF GitHub Bot commented on METRON-1796:

GitHub user ruffle1986 opened a pull request:

https://github.com/apache/metron/pull/1219

METRON-1796: [UI] Migrate off moment.js

## Contributor Comments

[DISCUSS] thread on the dev mailing list:
https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E

Original ASF Jira ticket: https://issues.apache.org/jira/browse/METRON-1796

The purpose of this PR is the complete removal of
[moment.js](http://momentjs.com/) from the code base and replacing it with
either native functions or [date-fns](https://date-fns.org/), another date
manipulation library.

**Motivation**

In the Metron Alerts UI, we use a few functions only from moment.js to deal
with time like formatting or displaying the relative time from "now". In order
to do that, we have to import the entire library which causes a significant
footprint in the compiled production bundle. Sometimes they can be replaced
with native built-in solutions, sometimes they cannot but we can use date-fns
which has a smaller size comparing to moment.js

As of date-fns v2 is in alpha phase, I rather chose the latest stable
version which is 1.29.0. Tree-shaking is currently not supported in version 1
but [it will be in version
2](https://date-fns.org/v2.0.0-alpha.9/docs/ECMAScript-Modules). So once it's
released and we can upgrade to it, we can see more size loss in the bundle
size. Angular uses Webpack under the hood so if you're interested in what
tree-shaking is and how Webpack does it, [follow this
link](https://webpack.js.org/guides/tree-shaking/) for further details.

**Changes included**

- Every code used moment.js is replaced with a built-in function or the
appropriate function from date-fns.

I highly recommend you to go through all the commits I've made one by one.
They're straightforward and easier to understand which parts of the app are
affected and in what way.

**Testing the bundle file size**

1. Checkout `master`
1. Go to `metron-interface/metron-alerts`
1. Make sure you have the latest packages in the `node_modules` folder via
`npm ci`
1. Run `npm run build`
1. Take a look at the size of the `main.js` file
1. Checkout `ruffle1986/METRON-1796`
1. Run `npm ci` (this will install the date-fns library and remove
moment.js)
1. Run `npm run build`
1. Take a look at the size of the `main.js` file and compare it to the
number you've got in step 4

Here's the output of `npm run build` on the `master` branch:
![screen shot 2018-10-03 at 13 22
29](https://user-images.githubusercontent.com/2196208/46411822-08c66980-c71d-11e8-964b-6884c69b9d28.png)

And here's the output on this branch:
![screen shot 2018-10-03 at 13 28
08](https://user-images.githubusercontent.com/2196208/46411889-2d224600-c71d-11e8-9ad9-fb0fbd58807a.png)

As you can see, by removing moment.js, we could reduce the bundle size to
1.65 MB which is **15.6%** comparing to the bundle with moment.js. Not so
significant but still.

You've probably noticed the warning message on the second picture below the
output of the build process. This is the result of compiling `pikaday-time`
into our bundle via Angular. For the record, it's totally fine. Moment.js is
just an optional dependency of [Pikaday](https://dbushell.com/Pikaday/), it
works perfectly fine without it. This warning message is there because Pikaday
[checks whether it's a Node.js environment and since it is, it wants to load
moment from the node_modules
folder](https://github.com/dbushell/Pikaday/blob/master/pikaday.js#L12-L16).
However Pikaday doesn't throw and fails silently, the Angular compiler is
clever enough to catch this and kindly warn you about it. But it doesn't cause
any negative effect in the Metron UI because (again) moment.js is not a
required dependency of Pikaday.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron

[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js

GitHub user ruffle1986 opened a pull request:

https://github.com/apache/metron/pull/1219

METRON-1796: [UI] Migrate off moment.js

## Contributor Comments

[DISCUSS] thread on the dev mailing list:
https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E

Original ASF Jira ticket: https://issues.apache.org/jira/browse/METRON-1796

**Motivation**

**Changes included**

- Every code used moment.js is replaced with a built-in function or the
appropriate function from date-fns.

I highly recommend you to go through all the commits I've made one by one.
They're straightforward and easier to understand which parts of the app are
affected and in what way.

**Testing the bundle file size**

Here's the output of `npm run build` on the `master` branch:
![screen shot 2018-10-03 at 13 22
29](https://user-images.githubusercontent.com/2196208/46411822-08c66980-c71d-11e8-964b-6884c69b9d28.png)

And here's the output on this branch:
![screen shot 2018-10-03 at 13 28
08](https://user-images.githubusercontent.com/2196208/46411889-2d224600-c71d-11e8-9ad9-fb0fbd58807a.png)

As you can see, by removing moment.js, we could reduce the bundle size to
1.65 MB which is **15.6%** comparing to the bundle with moment.js. Not so
significant but still.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636945#comment-16636945
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1215
  
Thanks again for the review, @nickwallen !


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1215
  
Thanks again for the review, @nickwallen !


---

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636819#comment-16636819
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1215
  
+1 This is really good fix @anandsubbu .  Thanks for the easily grokable 
docs and clean code.


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1215
  
+1 This is really good fix @anandsubbu .  Thanks for the easily grokable 
docs and clean code.


---

[jira] [Created] (METRON-1803) Integrate Cypress tests with Travis

2018-10-03 Thread Tibor Meller (JIRA)

Tibor Meller created METRON-1803:


 Summary: Integrate Cypress tests with Travis
 Key: METRON-1803
 URL: https://issues.apache.org/jira/browse/METRON-1803
 Project: Metron
  Issue Type: Improvement
Reporter: Tibor Meller
Assignee: Tibor Meller






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation

[
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636695#comment-16636695
]

ASF GitHub Bot commented on METRON-1798:

GitHub user anandsubbu reopened a pull request:

https://github.com/apache/metron/pull/1215

METRON-1798: Add mpack support for parser aggregation

## Contributor Comments
This pull request allows users to submit an aggregated parser topology.

## Testing Steps
1. Spin up full-dev.
2. Stop the "Metron Parsers" service so that existing parser topologies are
killed/stopped.
3. Go to Ambari -> Metron -> Configs -> Parsers
4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example)
5. Save changes and restart the Metron Parsers service.
6. Go to Storm UI, and verify that the aggregated topologies viz.
"bro__snort" and "yaf" are started.

## Testing Done
### 1. Full Dev
* Set the Metron Parsers value as "bro,snort.yaf" and restarted the
service.
* A single aggregated topology is seen to be started:

![image](https://user-images.githubusercontent.com/20395490/46216724-dbb13a00-c35d-11e8-9651-b39d0766208f.png)

* Verified stop 'Metron Parsers' service to check that the aggregated
topology is stopped properly
* Verified restart 'Metron Parsers' service to check that the aggregated
topology is restarted properly

### 2. Multi-node setup
* Set the Metron Parsers value as:
"bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort
* The appropriate aggregated and single topologies are started:

![image](https://user-images.githubusercontent.com/20395490/46217047-84f83000-c35e-11e8-87e3-994b72d06cf3.png)

* Changed the parser list as: bro,snort,yaf
* Restarted the parsers service to validate that the three parser
topologies are started individually.

![image](https://user-images.githubusercontent.com/20395490/46223825-0dcc9700-c372-11e8-9027-ad6ad1577f5f.png)

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```

- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:

```
cd site-book
mvn site
```

Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1798

Alternatively you can review and

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636692#comment-16636692
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user anandsubbu closed the pull request at:

https://github.com/apache/metron/pull/1215


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636694#comment-16636694
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1215
  
Re-run travis


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1215
  
Re-run travis


---

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

GitHub user anandsubbu reopened a pull request:

https://github.com/apache/metron/pull/1215

METRON-1798: Add mpack support for parser aggregation

## Contributor Comments
This pull request allows users to submit an aggregated parser topology.

## Testing Done
### 1. Full Dev
* Set the Metron Parsers value as "bro,snort.yaf" and restarted the
service.
* A single aggregated topology is seen to be started:

![image](https://user-images.githubusercontent.com/20395490/46216724-dbb13a00-c35d-11e8-9651-b39d0766208f.png)

### 2. Multi-node setup
* Set the Metron Parsers value as:
"bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort
* The appropriate aggregated and single topologies are started:

![image](https://user-images.githubusercontent.com/20395490/46217047-84f83000-c35e-11e8-87e3-994b72d06cf3.png)

* Changed the parser list as: bro,snort,yaf
* Restarted the parsers service to validate that the three parser
topologies are started individually.

![image](https://user-images.githubusercontent.com/20395490/46223825-0dcc9700-c372-11e8-9027-ad6ad1577f5f.png)

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

```
cd site-book
mvn site
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1798

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1215

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

Github user anandsubbu closed the pull request at:

https://github.com/apache/metron/pull/1215


---

[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation



[ 
https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636668#comment-16636668
 ] 

ASF GitHub Bot commented on METRON-1798:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/1215
  
Hi @nickwallen , thank you for the review.

> @nickwallen One thing I notice right off is that we did not add any 
documentation for the parsers fields in Ambari. Would it make sense to add a 
brief description to the pop-up text describing how a user can define parsers?

Sure, makes sense. I have also added a section to the Parser Chaining 
README as well with examples. Please have a look.

> @nickwallen: When switching parser topologies, some of the original 
parser topologies can fail to be shut down properly.
> I have found that the same issue occurs in master. Since that is the 
case, we could choose to tackle this as a separate ticket. It is your choice.

I created [METRON-1802](https://issues.apache.org/jira/browse/METRON-1802) 
so this can be fixed outside of this PR. 


> Add mpack support for parser aggregation
> 
>
> Key: METRON-1798
> URL: https://issues.apache.org/jira/browse/METRON-1798
> Project: Metron
>  Issue Type: Task
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
>
> Support spawning of storm topologies if a user specifies an aggregated parser 
> configuration at: 
> Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers"
>  
> For example, specifying the following:
> "bro,snort,yaf", "snort,yaf", yaf
> should spawn an aggregated topology for first two, and a regular topology for 
> the 'yaf'.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation