[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1207 > Cool, thanks for your opinion @nickwallen . I will wait for @MohanDV 's thoughts as well before merging. +1 Thanks @anandsubbu ---
[jira] [Commented] (METRON-1795) General Purpose Regex Parser
[
https://issues.apache.org/jira/browse/METRON-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639203#comment-16639203
]
ASF GitHub Bot commented on METRON-1795:
Github user jagdeepsingh2 commented on the issue:
https://github.com/apache/metron/pull/1214
Yeah, I performed a rebase yesterday as I had to pull the latest changes
from upstream. What is the best way out? Should I discard this PR and create a
fresh and clean PR?
> General Purpose Regex Parser
>
>
> Key: METRON-1795
> URL: https://issues.apache.org/jira/browse/METRON-1795
> Project: Metron
> Issue Type: New Feature
>Reporter: Jagdeep Singh
>Priority: Minor
>
> We have implemented a general purpose regex parser for Metron that we are
> interested in contributing back to the community.
>
> While the Metron Grok parser provides some regex based capability today, the
> intention of this general purpose regex parser is to:
> # Allow for more advanced parsing scenarios (specifically, dealing with
> multiple regex lines for devices that contain several log formats within them)
> # Give users and developers of Metron additional options for parsing
> # With the new parser chaining and regex routing feature available in
> Metron, this gives some additional flexibility to logically separate a flow
> by:
> # Regex routing to segregate logs at a device level and handle envelope
> unwrapping
> # This general purpose regex parser to parse an entire device type that
> contains multiple log formats within the single device (for example, RHEL
> logs)
> At the high-level control flow is like this:
> # Identify the record type if incoming raw message.
> # Find and apply the regular expression of corresponding record type to
> extract the fields (using named groups).
> # Apply the message header regex to extract the fields in the header part of
> the message (using named groups).
>
> The parser config uses the following structure:
>
> {code:java}
> "recordTypeRegex": "(?(?<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
> "messageHeaderRegex":
> "(?(?<=^<)\\d{1,4}(?=>)).*?(?(?<=>)[A-Za-z]{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?(?(?<=\\s).*?(?=\\s))",
> "fields": [
> {
> "recordType": "kernel",
> "regex": ".*(?(?<=\\]|\\w\\:).*?(?=$))"
> },
> {
> "recordType": "syslog",
> "regex":
> ".*(?(?<=PID\\s=\\s).*?(?=\\sLine)).*(?(?<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))(?.*?(?=\")).*(?(?<=\").*?(?=$))"
> }
> ]
> {code}
>
> Where:
> * *recordTypeRegex* is used to distinctly identify a record type. It inputs
> a valid regular expression and may also have named groups, which would be
> extracted into fields.
> * *messageHeaderRegex* is used to specify a regular expression to extract
> fields from a message part which is common across all the messages (i.e,
> syslog fields, standard headers)
> * *fields*: json list of objects containing recordType and regex. The
> expression that is evaluated is based on the output of the recordTypeRegex
> * Note: *recordTypeRegex* and *messageHeaderRegex* could be specified as
> lists also (as a JSON array), where the list will be evaluated in order until
> a matching regular expression is found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1214: METRON-1795 Initial commit for a general purpose regular...
Github user jagdeepsingh2 commented on the issue: https://github.com/apache/metron/pull/1214 Yeah, I performed a rebase yesterday as I had to pull the latest changes from upstream. What is the best way out? Should I discard this PR and create a fresh and clean PR? ---
[jira] [Commented] (METRON-1791) Add GUID to Messages Produced by Profiler
[ https://issues.apache.org/jira/browse/METRON-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638915#comment-16638915 ] ASF GitHub Bot commented on METRON-1791: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1210 > Add GUID to Messages Produced by Profiler > - > > Key: METRON-1791 > URL: https://issues.apache.org/jira/browse/METRON-1791 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > If a profile defines a triage result, when that profile is flushed the > Profiler will generate a message containing the measurement data so that it > can be used for Threat Triage. The message that is generated does not > contain a GUID. All messages in Metron should contain a GUID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron-bro-plugin-kafka pull request #9: METRON-1773 Bro plugin docs should ...
Github user asfgit closed the pull request at: https://github.com/apache/metron-bro-plugin-kafka/pull/9 ---
[GitHub] metron pull request #1210: METRON-1791 Add GUID to Messages Produced by Prof...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1210 ---
[jira] [Commented] (METRON-1773) Bro plugin docs should refer to Apache Metron project
[ https://issues.apache.org/jira/browse/METRON-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638917#comment-16638917 ] ASF GitHub Bot commented on METRON-1773: Github user asfgit closed the pull request at: https://github.com/apache/metron-bro-plugin-kafka/pull/9 > Bro plugin docs should refer to Apache Metron project > - > > Key: METRON-1773 > URL: https://issues.apache.org/jira/browse/METRON-1773 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1190: METRON-1771: Update REST endpoints to support eventually...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1190 +1 Looks great. Works great. I ran this up in Full Dev and tested manually through the Alerts UI. Looking forward to the follow-on UI changes. ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[ https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638907#comment-16638907 ] ASF GitHub Bot commented on METRON-1771: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1190 +1 Looks great. Works great. I ran this up in Full Dev and tested manually through the Alerts UI. Looking forward to the follow-on UI changes. > Update REST endpoints to support eventually consistent UI updates > - > > Key: METRON-1771 > URL: https://issues.apache.org/jira/browse/METRON-1771 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > > Currently the REST endpoints that perform document updates either return > true/false or nothing. This puts the responsibility of retrieving the > updated state of the object on the client in a separate call or > optimistically applying the changes and reverting when an update fails. This > can be problematic if a client attempts to get the current state immediately > after an update and the change isn't visible yet in the back end. > Ideally they should return the updated state of the object, eliminating the > need to look up the updated state in a separate call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638899#comment-16638899
]
ASF GitHub Bot commented on METRON-1761:
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1184
I prefer @mmiklavc's suggestion several comments back: return a List of
something other than a List. Could we use a wrapper that can
contain results and errors? The calling class could then determine the best
way to handle errors without having to fail the whole batch.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1184 I prefer @mmiklavc's suggestion several comments back: return a List of something other than a List. Could we use a wrapper that can contain results and errors? The calling class could then determine the best way to handle errors without having to fail the whole batch. ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638872#comment-16638872 ] ASF GitHub Bot commented on METRON-1681: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1213 @mmiklavc It's been like that for a long time. https://github.com/apache/metron/blob/6fb85e124d87581c37b79ea105570f4f85b59384/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java#L118. It's probably earlier than this, I didn't track down the actual commit, just checked a random one from awhile ago. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1213 @mmiklavc It's been like that for a long time. https://github.com/apache/metron/blob/6fb85e124d87581c37b79ea105570f4f85b59384/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java#L118. It's probably earlier than this, I didn't track down the actual commit, just checked a random one from awhile ago. ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638864#comment-16638864
]
ASF GitHub Bot commented on METRON-1771:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r222823946
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
Looks good to me. Latest commit includes this change.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r222823946
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
Looks good to me. Latest commit includes this change.
---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638850#comment-16638850 ] ASF GitHub Bot commented on METRON-1681: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1213 @merrimanr @justinleet @cestella - regarding the ignored tuple when no config for sensor - has that been longstanding or is that something intentionally added with parser aggregation and chaining? Just cautious of adding a regression. At the very least, if it is intentional a comment should be added explaining it. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638858#comment-16638858 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 As far as I know it's been that way for a really long time. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 As far as I know it's been that way for a really long time. ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1213 @merrimanr @justinleet @cestella - regarding the ignored tuple when no config for sensor - has that been longstanding or is that something intentionally added with parser aggregation and chaining? Just cautious of adding a regression. At the very least, if it is intentional a comment should be added explaining it. ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638844#comment-16638844 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 The latest commit removes the callbacks. Now the execute method returns a list of ParserResult objects. I had to add a MetronError field to the ParserResult class so that both successful messages and errors are passed back. I also noticed we are currently skipping a tuple if no configuration is found for that sensor. I think skipping the tuple is fine but we should still log an error message. I made a change in the ParserRunner class to throw an error on this condition. The parser bolt will catch it and handle it similar to a parser error by creating a MetronError object and sending it to the error topic. Does anyone think we should keep as it is now? > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1791) Add GUID to Messages Produced by Profiler
[ https://issues.apache.org/jira/browse/METRON-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638845#comment-16638845 ] ASF GitHub Bot commented on METRON-1791: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1210 lgtm, +1 by inspection. > Add GUID to Messages Produced by Profiler > - > > Key: METRON-1791 > URL: https://issues.apache.org/jira/browse/METRON-1791 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > If a profile defines a triage result, when that profile is flushed the > Profiler will generate a message containing the measurement data so that it > can be used for Threat Triage. The message that is generated does not > contain a GUID. All messages in Metron should contain a GUID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1210: METRON-1791 Add GUID to Messages Produced by Profiler
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1210 lgtm, +1 by inspection. ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 The latest commit removes the callbacks. Now the execute method returns a list of ParserResult objects. I had to add a MetronError field to the ParserResult class so that both successful messages and errors are passed back. I also noticed we are currently skipping a tuple if no configuration is found for that sensor. I think skipping the tuple is fine but we should still log an error message. I made a change in the ParserRunner class to throw an error on this condition. The parser bolt will catch it and handle it similar to a parser error by creating a MetronError object and sending it to the error topic. Does anyone think we should keep as it is now? ---
[jira] [Commented] (METRON-1806) Upgrade Maven Shade Plugin version
[ https://issues.apache.org/jira/browse/METRON-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638838#comment-16638838 ] ASF GitHub Bot commented on METRON-1806: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1224 METRON-1806: Upgrade Maven Shade Plugin version ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1806 Update the maven shade plugin version. There is a bug in the current version that causes issues when shading and relocating various libraries. One such example is log4j. A recent doc update for generating the Elasticsearch X-Pack client jar addressed this issue similarly - https://github.com/apache/metron/pull/1196 **Testing** Run up full dev and verify data flows through indexing as normal. The management and alerts UI's should function as normal. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - N/A Have you written or updated unit tests and or integration tests to verify your changes? - N/A If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - N/A Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron upgrade-shade-plugin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1224.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1224 commit a60c13ad202e783151d6b2313254daa44755ef01 Author: Michael Miklavcic Date: 2018-10-01T22:42:55Z Upgrade maven-shade-plugin version > Upgrade Maven Shade Plugin version > -- > > Key: METRON-1806 > URL: https://issues.apache.org/jira/browse/METRON-1806 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1224: METRON-1806: Upgrade Maven Shade Plugin version
GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1224 METRON-1806: Upgrade Maven Shade Plugin version ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1806 Update the maven shade plugin version. There is a bug in the current version that causes issues when shading and relocating various libraries. One such example is log4j. A recent doc update for generating the Elasticsearch X-Pack client jar addressed this issue similarly - https://github.com/apache/metron/pull/1196 **Testing** Run up full dev and verify data flows through indexing as normal. The management and alerts UI's should function as normal. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - N/A Have you written or updated unit tests and or integration tests to verify your changes? - N/A If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - N/A Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron upgrade-shade-plugin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1224.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1224 commit a60c13ad202e783151d6b2313254daa44755ef01 Author: Michael Miklavcic Date: 2018-10-01T22:42:55Z Upgrade maven-shade-plugin version ---
[jira] [Created] (METRON-1806) Upgrade Maven Shade Plugin version
Michael Miklavcic created METRON-1806: - Summary: Upgrade Maven Shade Plugin version Key: METRON-1806 URL: https://issues.apache.org/jira/browse/METRON-1806 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron-bro-plugin-kafka issue #8: METRON-1768: Adjust versioning of metron-b...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 My PRs to the bro stack were merged, but it won't be available until bro 2.7 is released. Currently they are beta testing 2.6. ---
[jira] [Commented] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638670#comment-16638670 ] ASF GitHub Bot commented on METRON-1768: Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 My PRs to the bro stack were merged, but it won't be available until bro 2.7 is released. Currently they are beta testing 2.6. > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > In order to comprehensively do this, it will require an upstream fix in bro. > I opened https://bro-tracker.atlassian.net/browse/BIT-1985 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638553#comment-16638553 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 I don't feel that strongly about either approach so I'll switch it to return a list and remove the callbacks. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 I don't feel that strongly about either approach so I'll switch it to return a list and remove the callbacks. ---
[GitHub] metron pull request #1223: METRON-1797: Add PCAP UI test implemented in Cypr...
Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1223 ---
[jira] [Commented] (METRON-1797) Add PCAP UI test implemented in Cypress.js
[ https://issues.apache.org/jira/browse/METRON-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638543#comment-16638543 ] ASF GitHub Bot commented on METRON-1797: Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1223 > Add PCAP UI test implemented in Cypress.js > -- > > Key: METRON-1797 > URL: https://issues.apache.org/jira/browse/METRON-1797 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > [DISCUSS] thread on the dev mailing list: > [https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E] > This PR adds Cypress.io to our project. It makes us able to run E2E tests, UI > integration tests. Also higher level UI tests with mocked services. > As part of this PR I also added tests to cover the following functionalities > of PCAP UI panel: > * checking running jobs on navigating to PCAP tab > * submitting PCAP job request > * requesting job status > * process status in percentage > * getting PCAP JSON > * rendering PCAP table > * showing PCAP details > * navigating across pages > * downloading PDML > * cancelli PCAP query job > * showing filter validation messages > * showing date validation messages > Please consider while reviewing that these tests are not meant to cover all > the details of the functionalities above. The aim here is to extend the test > coverage build up by a large number of Unit Tests with higher level UI tests. > h2. Integration folder > That contains the actual Cypress tests. The folder name comes from the > Cypress naming convention, however, no real integration tests implemented > here yet (at least no other than the integration of UI components to a whole > application). We could change the name of the folder anytime we like to, but > only one folder applicable. > h2. Fixture folder > The tests are running on mocked data. This makes us able to integrate them > with Travis without the need to having PCAP related services available from > our CI. > Each file in the Fixture folder represents a response to a particular > request. This makes it fairly easy to create and maintain. > h1. How to run > To run Cypress tests locally please do the following: > # pull this branch to your workspace > # move to metron-interface/metron-alerts folder > # run: npm ci > # run: scripts/start-server-for-e2e.sh > # run: node_modules/cypress/bin/cypress run > or run: node_modules/cypress/bin/cypress open > if you like to use the dashboard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638537#comment-16638537
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r222750671
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
> The question is what should happen when this unexpected condition does
happen? Should it blow up and throw and exception or return a document if at
least one DAO returns a valid one.
Ah, OK. I'm groking it now. We want it to return null; at least based on
the javadocs defined in `RetrieveLatestDao.getLatest` and other places as null
means nothing was found. So functionally, we don't want it to throw that
exception.
But IMO it might benefit from some clarity. I would suggest changing
`getDocument` to `getLatestDocument` with some commentary and clarity on what
it means if there is no Document or exception and what happens if there is an
error in one of the underlying indices.
Feel free to tweak to your liking or disregard, but this is what I would do.
```
/**
* Returns the most recent {@link Document} from a list of {@link
DocumentContainer}s.
*
* @param documentContainers A list of containers; each retrieved from a
separate index.
* @return The latest {@link Document} found.
* @throws IOException If any of the {@link DocumentContainer}s contain
an exception.
*/
private Document getLatestDocument(List
documentContainers) throws IOException {
Document latestDocument = null;
List error = new ArrayList<>();
for(DocumentContainer dc : documentContainers) {
if(dc.getException().isPresent()) {
// collect each exception; multiple can occur, one in each index
Throwable e = dc.getException().get();
error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
} else if(dc.getDocument().isPresent()) {
Document d = dc.getDocument().get();
// is this the latest document so far?
if(latestDocument == null || latestDocument.getTimestamp() <
d.getTimestamp()) {
latestDocument = d;
}
} else {
// no document was found in the index
latestDocument = null;
}
}
if(error.size() > 0) {
// report all of the errors encountered
throw new IOException(Joiner.on("\n").join(error));
}
return latestDocument;
}
```
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r222750671
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
> The question is what should happen when this unexpected condition does
happen? Should it blow up and throw and exception or return a document if at
least one DAO returns a valid one.
Ah, OK. I'm groking it now. We want it to return null; at least based on
the javadocs defined in `RetrieveLatestDao.getLatest` and other places as null
means nothing was found. So functionally, we don't want it to throw that
exception.
But IMO it might benefit from some clarity. I would suggest changing
`getDocument` to `getLatestDocument` with some commentary and clarity on what
it means if there is no Document or exception and what happens if there is an
error in one of the underlying indices.
Feel free to tweak to your liking or disregard, but this is what I would do.
```
/**
* Returns the most recent {@link Document} from a list of {@link
DocumentContainer}s.
*
* @param documentContainers A list of containers; each retrieved from a
separate index.
* @return The latest {@link Document} found.
* @throws IOException If any of the {@link DocumentContainer}s contain
an exception.
*/
private Document getLatestDocument(List
documentContainers) throws IOException {
Document latestDocument = null;
List error = new ArrayList<>();
for(DocumentContainer dc : documentContainers) {
if(dc.getException().isPresent()) {
// collect each exception; multiple can occur, one in each index
Throwable e = dc.getException().get();
error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
} else if(dc.getDocument().isPresent()) {
Document d = dc.getDocument().get();
// is this the latest document so far?
if(latestDocument == null || latestDocument.getTimestamp() <
d.getTimestamp()) {
latestDocument = d;
}
} else {
// no document was found in the index
latestDocument = null;
}
}
if(error.size() > 0) {
// report all of the errors encountered
throw new IOException(Joiner.on("\n").join(error));
}
return latestDocument;
}
```
---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638492#comment-16638492 ] ASF GitHub Bot commented on METRON-1681: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1213 @merrimanr - I think I viewed the choice of ParserRunner implementation as a strategy of sorts. I guess I don't care either way. @nickwallen's suggestion of making that an interface, that I see you've implemented, should work well there also. There's definitely some impedance mismatch between what's in parsers vs enrichment, so there won't be a direct mapping of concepts. @nickwallen - I've been going back and forth about which approach to take in handling success/errors. I think I agree with you and prefer keeping the runner interface simpler by simply returning a `List`. The fact that we have to jump through hoops a bit for callbacks because of needing reference to the Tuple in onSuccess suggests to me that it's probably better to manage the results in a returned list. The validation on the runner is also simpler in this case as well, ie no need to validate you've set your callbacks. Having onSucess/Error methods in the runner interface feels a little fishy to me. I also think that it makes some sense to decouple the parsing phase from writing altogether, though I recognize that callbacks are not opinionated about the "what" that happens in success/error would be. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Cool, thanks for your opinion @nickwallen . I will wait for @MohanDV 's thoughts as well before merging. ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638469#comment-16638469 ] ASF GitHub Bot commented on METRON-1695: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1207 > @anandsubbu With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both) I agree. Since we have pcap properties affecting both the topology and REST on the same tab, we have to restart both services. That could be changed, but I think it is good enough for now. Not a big deal in my opinion. > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1207 > @anandsubbu With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both) I agree. Since we have pcap properties affecting both the topology and REST on the same tab, we have to restart both services. That could be changed, but I think it is good enough for now. Not a big deal in my opinion. ---
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r222736631
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -242,169 +226,81 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
}
}
- protected void initializeStellar() {
-Context.Builder builder = new Context.Builder()
-
.with(Context.Capabilities.ZOOKEEPER_CLIENT, () -> client)
-.with(Context.Capabilities.GLOBAL_CONFIG,
() -> getConfigurations().getGlobalConfig())
-.with(Context.Capabilities.STELLAR_CONFIG,
() -> getConfigurations().getGlobalConfig())
-;
-if(cache != null) {
- builder = builder.with(Context.Capabilities.CACHE, () -> cache);
-}
-this.stellarContext = builder.build();
-StellarFunctions.initialize(stellarContext);
- }
-
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
if (TupleUtils.isTick(tuple)) {
- try {
-for (Entry entry :
sensorToComponentMap.entrySet()) {
- entry.getValue().getWriter().flush(getConfigurations(),
messageGetStrategy);
-}
- } catch (Exception e) {
-throw new RuntimeException(
-"This should have been caught in the writerHandler. If you
see this, file a JIRA", e);
- } finally {
-collector.ack(tuple);
- }
+ handleTickTuple(tuple);
return;
}
-
+numWritten = 0;
byte[] originalMessage = (byte[]) messageGetStrategy.get(tuple);
+String topic =
tuple.getStringByField(FieldsConfiguration.TOPIC.getFieldName());
+String sensorType = topicToSensorMap.get(topic);
try {
- SensorParserConfig sensorParserConfig;
- MessageParser parser;
- String sensor;
- Map metadata;
- if (sensorToComponentMap.size() == 1) {
-// There's only one parser, so grab info directly
-Entry sensorParser =
sensorToComponentMap.entrySet().iterator()
-.next();
-sensor = sensorParser.getKey();
-parser = sensorParser.getValue().getMessageParser();
-sensorParserConfig = getSensorParserConfig(sensor);
- } else {
-// There's multiple parsers, so pull the topic from the Tuple and
look up the sensor
-String topic =
tuple.getStringByField(FieldsConfiguration.TOPIC.getFieldName());
-sensor = topicToSensorMap.get(topic);
-parser = sensorToComponentMap.get(sensor).getMessageParser();
-sensorParserConfig = getSensorParserConfig(sensor);
- }
+ ParserConfigurations parserConfigurations = getConfigurations();
+ SensorParserConfig sensorParserConfig =
parserConfigurations.getSensorParserConfig(sensorType);
+ RawMessage rawMessage = RawMessageUtil.INSTANCE.getRawMessage(
sensorParserConfig.getRawMessageStrategy()
+ , tuple
+ , originalMessage
+ , sensorParserConfig.getReadMetadata()
+ , sensorParserConfig.getRawMessageStrategyConfig()
+ );
+ parserRunner.setOnSuccess(parserResult -> onSuccess(parserResult,
tuple));
--- End diff --
Ah, that's right. This answers my earlier comment about setsuccess/error.
---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638456#comment-16638456
]
ASF GitHub Bot commented on METRON-1681:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r222736631
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -242,169 +226,81 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
}
}
- protected void initializeStellar() {
-Context.Builder builder = new Context.Builder()
-
.with(Context.Capabilities.ZOOKEEPER_CLIENT, () -> client)
-.with(Context.Capabilities.GLOBAL_CONFIG,
() -> getConfigurations().getGlobalConfig())
-.with(Context.Capabilities.STELLAR_CONFIG,
() -> getConfigurations().getGlobalConfig())
-;
-if(cache != null) {
- builder = builder.with(Context.Capabilities.CACHE, () -> cache);
-}
-this.stellarContext = builder.build();
-StellarFunctions.initialize(stellarContext);
- }
-
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
if (TupleUtils.isTick(tuple)) {
- try {
-for (Entry entry :
sensorToComponentMap.entrySet()) {
- entry.getValue().getWriter().flush(getConfigurations(),
messageGetStrategy);
-}
- } catch (Exception e) {
-throw new RuntimeException(
-"This should have been caught in the writerHandler. If you
see this, file a JIRA", e);
- } finally {
-collector.ack(tuple);
- }
+ handleTickTuple(tuple);
return;
}
-
+numWritten = 0;
byte[] originalMessage = (byte[]) messageGetStrategy.get(tuple);
+String topic =
tuple.getStringByField(FieldsConfiguration.TOPIC.getFieldName());
+String sensorType = topicToSensorMap.get(topic);
try {
- SensorParserConfig sensorParserConfig;
- MessageParser parser;
- String sensor;
- Map metadata;
- if (sensorToComponentMap.size() == 1) {
-// There's only one parser, so grab info directly
-Entry sensorParser =
sensorToComponentMap.entrySet().iterator()
-.next();
-sensor = sensorParser.getKey();
-parser = sensorParser.getValue().getMessageParser();
-sensorParserConfig = getSensorParserConfig(sensor);
- } else {
-// There's multiple parsers, so pull the topic from the Tuple and
look up the sensor
-String topic =
tuple.getStringByField(FieldsConfiguration.TOPIC.getFieldName());
-sensor = topicToSensorMap.get(topic);
-parser = sensorToComponentMap.get(sensor).getMessageParser();
-sensorParserConfig = getSensorParserConfig(sensor);
- }
+ ParserConfigurations parserConfigurations = getConfigurations();
+ SensorParserConfig sensorParserConfig =
parserConfigurations.getSensorParserConfig(sensorType);
+ RawMessage rawMessage = RawMessageUtil.INSTANCE.getRawMessage(
sensorParserConfig.getRawMessageStrategy()
+ , tuple
+ , originalMessage
+ , sensorParserConfig.getReadMetadata()
+ , sensorParserConfig.getRawMessageStrategyConfig()
+ );
+ parserRunner.setOnSuccess(parserResult -> onSuccess(parserResult,
tuple));
--- End diff --
Ah, that's right. This answers my earlier comment about setsuccess/error.
> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
> Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are
> concerns about the ParserBolt needed some refactoring. The discussion didn't
> hold the PR up, but it was generally agreed that we should decouple some of
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[
https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638455#comment-16638455
]
ASF GitHub Bot commented on METRON-1681:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r222735970
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -185,37 +199,12 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-
-// Build the Stellar cache
-Map cacheConfig = new HashMap<>();
-for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
- String sensor = entry.getKey();
- SensorParserConfig config = getSensorParserConfig(sensor);
-
- if (config != null) {
-cacheConfig.putAll(config.getCacheConfig());
- }
-}
-cache = CachingStellarProcessor.createCache(cacheConfig);
+this.parserRunner.setOnError(this::onError);
--- End diff --
Seems like success/error functions should go in init() if we went this
route. I think it's best to choose 1 approach or the other when managing
initialization and state validation for an object.
> Decouple the ParserBolt from the Parse execution logic
> --
>
> Key: METRON-1681
> URL: https://issues.apache.org/jira/browse/METRON-1681
> Project: Metron
> Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Major
>
> Per discussion on https://github.com/apache/metron/pull/1099, there are
> concerns about the ParserBolt needed some refactoring. The discussion didn't
> hold the PR up, but it was generally agreed that we should decouple some of
> the initialization and execution logic.
> This also aids us in integrating with other systems such as NiFi or Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1213#discussion_r222735970
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -185,37 +199,12 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
-
-// Build the Stellar cache
-Map cacheConfig = new HashMap<>();
-for (Map.Entry entry:
sensorToComponentMap.entrySet()) {
- String sensor = entry.getKey();
- SensorParserConfig config = getSensorParserConfig(sensor);
-
- if (config != null) {
-cacheConfig.putAll(config.getCacheConfig());
- }
-}
-cache = CachingStellarProcessor.createCache(cacheConfig);
+this.parserRunner.setOnError(this::onError);
--- End diff --
Seems like success/error functions should go in init() if we went this
route. I think it's best to choose 1 approach or the other when managing
initialization and state validation for an object.
---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638437#comment-16638437
]
ASF GitHub Bot commented on METRON-1761:
Github user lvets commented on the issue:
https://github.com/apache/metron/pull/1184
So I talked @ottobackwards into initially adding this feature The reason
I asked for this improvement is that have a bunch of log files which contain a
relevant entry per line. I currently have to rely on 3rdparty tools to split
these logfiles into messages (where 1 log line == 1 message) before sending
them to Metron. I was hoping to just ingest the log and Metron would take care
of this.
To me, relying on 3rdparty tools for what I assume to be a normal use case
for a SIEM seems a bit strange.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638425#comment-16638425 ] ASF GitHub Bot commented on METRON-1695: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1207 +1 Thanks @anandsubbu ! > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1795) General Purpose Regex Parser
[
https://issues.apache.org/jira/browse/METRON-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638440#comment-16638440
]
ASF GitHub Bot commented on METRON-1795:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1214
@jagdeepsingh2 - can you review your recent commits? It looks like there's
a bad merge somewhere considering the jump to 6k+ lines in the diff.
> General Purpose Regex Parser
>
>
> Key: METRON-1795
> URL: https://issues.apache.org/jira/browse/METRON-1795
> Project: Metron
> Issue Type: New Feature
>Reporter: Jagdeep Singh
>Priority: Minor
>
> We have implemented a general purpose regex parser for Metron that we are
> interested in contributing back to the community.
>
> While the Metron Grok parser provides some regex based capability today, the
> intention of this general purpose regex parser is to:
> # Allow for more advanced parsing scenarios (specifically, dealing with
> multiple regex lines for devices that contain several log formats within them)
> # Give users and developers of Metron additional options for parsing
> # With the new parser chaining and regex routing feature available in
> Metron, this gives some additional flexibility to logically separate a flow
> by:
> # Regex routing to segregate logs at a device level and handle envelope
> unwrapping
> # This general purpose regex parser to parse an entire device type that
> contains multiple log formats within the single device (for example, RHEL
> logs)
> At the high-level control flow is like this:
> # Identify the record type if incoming raw message.
> # Find and apply the regular expression of corresponding record type to
> extract the fields (using named groups).
> # Apply the message header regex to extract the fields in the header part of
> the message (using named groups).
>
> The parser config uses the following structure:
>
> {code:java}
> "recordTypeRegex": "(?(?<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
> "messageHeaderRegex":
> "(?(?<=^<)\\d{1,4}(?=>)).*?(?(?<=>)[A-Za-z]{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?(?(?<=\\s).*?(?=\\s))",
> "fields": [
> {
> "recordType": "kernel",
> "regex": ".*(?(?<=\\]|\\w\\:).*?(?=$))"
> },
> {
> "recordType": "syslog",
> "regex":
> ".*(?(?<=PID\\s=\\s).*?(?=\\sLine)).*(?(?<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))(?.*?(?=\")).*(?(?<=\").*?(?=$))"
> }
> ]
> {code}
>
> Where:
> * *recordTypeRegex* is used to distinctly identify a record type. It inputs
> a valid regular expression and may also have named groups, which would be
> extracted into fields.
> * *messageHeaderRegex* is used to specify a regular expression to extract
> fields from a message part which is common across all the messages (i.e,
> syslog fields, standard headers)
> * *fields*: json list of objects containing recordType and regex. The
> expression that is evaluated is based on the output of the recordTypeRegex
> * Note: *recordTypeRegex* and *messageHeaderRegex* could be specified as
> lists also (as a JSON array), where the list will be evaluated in order until
> a matching regular expression is found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1214: METRON-1795 Initial commit for a general purpose regular...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1214 @jagdeepsingh2 - can you review your recent commits? It looks like there's a bad merge somewhere considering the jump to 6k+ lines in the diff. ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user lvets commented on the issue: https://github.com/apache/metron/pull/1184 So I talked @ottobackwards into initially adding this feature ð The reason I asked for this improvement is that have a bunch of log files which contain a relevant entry per line. I currently have to rely on 3rdparty tools to split these logfiles into messages (where 1 log line == 1 message) before sending them to Metron. I was hoping to just ingest the log and Metron would take care of this. To me, relying on 3rdparty tools for what I assume to be a normal use case for a SIEM seems a bit strange. ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638436#comment-16638436 ] ASF GitHub Bot commented on METRON-1695: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks much @nickwallen for giving it another spin! Hey @MohanDV , let me know if my below explanation makes sense. @nickwallen , would you have any thoughts on this? > @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. > > In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): > > * pdml.script.path > * base.path > * base.interim.result.path > * final.output.path > * page.size > * yarn.queue > * finalizer.threadpool.size > > For this reason, I did not move these properties out of the rest_application.yml. > > > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology > > With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config... > >  > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks much @nickwallen for giving it another spin! Hey @MohanDV , let me know if my below explanation makes sense. @nickwallen , would you have any thoughts on this? > @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. > > In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): > > * pdml.script.path > * base.path > * base.interim.result.path > * final.output.path > * page.size > * yarn.queue > * finalizer.threadpool.size > > For this reason, I did not move these properties out of the rest_application.yml. > > > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology > > With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config... > >  ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1207 +1 Thanks @anandsubbu ! ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638298#comment-16638298
]
ASF GitHub Bot commented on METRON-1761:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1184
@ottobackwards Yeah, understood about the message vs raw bytes. It seems
like there are 2 options here - use validate or modify the message parser
interface to return a List of something other than a `List`. If you
were to go the validate route, how would you modify and return the messages
that failed to parse so that validate can handle them? I'd expect them to be
indexed to the error topic in the same style as the single message parsers.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638406#comment-16638406
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
Closing this pr. I will create a jira for api improvement
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638407#comment-16638407
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards closed the pull request at:
https://github.com/apache/metron/pull/1184
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638419#comment-16638419
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
re-opening for input
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638394#comment-16638394
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
Let me give it a shot, I'll document the semantics of the failure mode and
we can look again
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 re-opening for input ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638416#comment-16638416
]
ASF GitHub Bot commented on METRON-1761:
GitHub user ottobackwards reopened a pull request:
https://github.com/apache/metron/pull/1184
METRON-1761, allow application of grok statement multiple times
This PR adds support for incoming messages to grok parsers that have
multiple log lines.
Instead of having to split the logs before sending to metron/kafka, you
could just send the logs in batches.
# todo testing
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [-] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron grok-split
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1184.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1184
commit 498313a8c12bcc15c8a179f5a97afff1d673d0b2
Author: Otto Fowler
Date: 2018-09-04T10:50:38Z
METRON-1761, allow application of grok statement multiple times
commit 05b734cf18be120cbabea554a0d84bb0164193c9
Author: Otto Fowler
Date: 2018-09-28T14:17:41Z
Merge remote-tracking branch 'apache/master' into grok-split
commit c2b3bb88d2a06e5cde39fd90a87f92207906eac4
Author: Otto Fowler
Date: 2018-09-28T15:40:37Z
per review, do not require derivation
commit 91764c924a7a85e4f8146105857234ad7b046c72
Author: Otto Fowler
Date: 2018-10-02T13:12:56Z
Merge remote-tracking branch 'apache/master' into grok-split
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
GitHub user ottobackwards reopened a pull request: https://github.com/apache/metron/pull/1184 METRON-1761, allow application of grok statement multiple times This PR adds support for incoming messages to grok parsers that have multiple log lines. Instead of having to split the logs before sending to metron/kafka, you could just send the logs in batches. # todo testing ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [-] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [-] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron grok-split Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 498313a8c12bcc15c8a179f5a97afff1d673d0b2 Author: Otto Fowler Date: 2018-09-04T10:50:38Z METRON-1761, allow application of grok statement multiple times commit 05b734cf18be120cbabea554a0d84bb0164193c9 Author: Otto Fowler Date: 2018-09-28T14:17:41Z Merge remote-tracking branch 'apache/master' into grok-split commit c2b3bb88d2a06e5cde39fd90a87f92207906eac4 Author: Otto Fowler Date: 2018-09-28T15:40:37Z per review, do not require derivation commit 91764c924a7a85e4f8146105857234ad7b046c72 Author: Otto Fowler Date: 2018-10-02T13:12:56Z Merge remote-tracking branch 'apache/master' into grok-split ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638413#comment-16638413 ] ASF GitHub Bot commented on METRON-1695: Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222726135 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -119,8 +119,8 @@ metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab'] # Pcap -metron_pcap_topology = 'pcap' -pcap_input_topic = 'pcap' +metron_pcap_topology = config['configurations']['metron-pcap-env']['spout_kafka_topic_pcap'] --- End diff -- Done, changed. > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222726135 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -119,8 +119,8 @@ metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab'] # Pcap -metron_pcap_topology = 'pcap' -pcap_input_topic = 'pcap' +metron_pcap_topology = config['configurations']['metron-pcap-env']['spout_kafka_topic_pcap'] --- End diff -- Done, changed. ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/1184 ---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 Closing this pr. I will create a jira for api improvement ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638404#comment-16638404
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
actually, I'm just going to close this. Once I step back from "how could I
do this" to look at the big picture, it doesn't seem like a good idea. This is
not a field issue, more of an improvement, so there is no need to hammer
something in.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 actually, I'm just going to close this. Once I step back from "how could I do this" to look at the big picture, it doesn't seem like a good idea. This is not a field issue, more of an improvement, so there is no need to hammer something in. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638397#comment-16638397
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
If it turns out that this is just such a duck tape job, we can always close
the PR and open a jira for the new api
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 If it turns out that this is just such a duck tape job, we can always close the PR and open a jira for the new api ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638393#comment-16638393
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
It think the API should be improved ideally, but that is in the future
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 Let me give it a shot, I'll document the semantics of the failure mode and we can look again ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638388#comment-16638388
]
ASF GitHub Bot commented on METRON-1761:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1184
@ottobackwards That sounds reasonable to me. One thing I would definitely
add to the documentation for this is some comment about how it changes
processing semantics and might require additional tuning. Since we now have
batch sizing specified for the writers to Kafka, there may be a little bit of
tweaking there, but I'm just not sure.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1184 @ottobackwards That sounds reasonable to me. One thing I would definitely add to the documentation for this is some comment about how it changes processing semantics and might require additional tuning. Since we now have batch sizing specified for the writers to Kafka, there may be a little bit of tweaking there, but I'm just not sure. ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638387#comment-16638387 ] ASF GitHub Bot commented on METRON-1695: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222720563 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -119,8 +119,8 @@ metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab'] # Pcap -metron_pcap_topology = 'pcap' -pcap_input_topic = 'pcap' +metron_pcap_topology = config['configurations']['metron-pcap-env']['spout_kafka_topic_pcap'] --- End diff -- Quick fix needed. ``` metron_pcap_topology = 'pcap' ``` > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222720563 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -119,8 +119,8 @@ metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab'] # Pcap -metron_pcap_topology = 'pcap' -pcap_input_topic = 'pcap' +metron_pcap_topology = config['configurations']['metron-pcap-env']['spout_kafka_topic_pcap'] --- End diff -- Quick fix needed. ``` metron_pcap_topology = 'pcap' ``` ---
[jira] [Commented] (METRON-1785) Automate deployment of packet capture for development environment
[ https://issues.apache.org/jira/browse/METRON-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638376#comment-16638376 ] ASF GitHub Bot commented on METRON-1785: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1205 Yes, you are right @MohanDV. Somehow the default tags are interacting badly with the tags that are passed in. I'll try to figure out what's going on. > Automate deployment of packet capture for development environment > - > > Key: METRON-1785 > URL: https://issues.apache.org/jira/browse/METRON-1785 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > I added instructions for deploying all of the components required to generate > and capture network packets in the development environment. I also fixed > some issues I ran into along the way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1791) Add GUID to Messages Produced by Profiler
[ https://issues.apache.org/jira/browse/METRON-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638373#comment-16638373 ] ASF GitHub Bot commented on METRON-1791: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1210 Looks good to me. +1 by inspection. > Add GUID to Messages Produced by Profiler > - > > Key: METRON-1791 > URL: https://issues.apache.org/jira/browse/METRON-1791 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > If a profile defines a triage result, when that profile is flushed the > Profiler will generate a message containing the measurement data so that it > can be used for Threat Triage. The message that is generated does not > contain a GUID. All messages in Metron should contain a GUID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1205: METRON-1785 Automate deployment of packet capture for de...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1205 Yes, you are right @MohanDV. Somehow the default tags are interacting badly with the tags that are passed in. I'll try to figure out what's going on. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638337#comment-16638337
]
ASF GitHub Bot commented on METRON-1761:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1184
So the idea would be that the JSONObject returned for the failed line (
that would be passed to handle error ) would be a new object that had the raw
line, the exception type, the exception message in it.
That would all get passed through to the Error stuff. We could put
anything we wanted in there.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 So the idea would be that the JSONObject returned for the failed line ( that would be passed to handle error ) would be a new object that had the raw line, the exception type, the exception message in it. That would all get passed through to the Error stuff. We could put anything we wanted in there. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638308#comment-16638308
]
ASF GitHub Bot commented on METRON-1761:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1184
Looking at the code in parserbolt, I think the only thing missing with the
validate route is the original parser exception. Here's how errors are handled
when validation error occurs:
```
ErrorUtils.handleError(collector, error);
```
here's how they're handled when an exception occurs during parsing:
```
} catch (Throwable ex) {
handleError(originalMessage, tuple, ex, collector);
}
```
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1184
Looking at the code in parserbolt, I think the only thing missing with the
validate route is the original parser exception. Here's how errors are handled
when validation error occurs:
```
ErrorUtils.handleError(collector, error);
```
here's how they're handled when an exception occurs during parsing:
```
} catch (Throwable ex) {
handleError(originalMessage, tuple, ex, collector);
}
```
---
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1184 @ottobackwards Yeah, understood about the message vs raw bytes. It seems like there are 2 options here - use validate or modify the message parser interface to return a List of something other than a `List`. If you were to go the validate route, how would you modify and return the messages that failed to parse so that validate can handle them? I'd expect them to be indexed to the error topic in the same style as the single message parsers. ---
[jira] [Updated] (METRON-1803) Integrate Cypress tests with Travis
[ https://issues.apache.org/jira/browse/METRON-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tibor Meller updated METRON-1803: - Description: [DISCUSS] thread on the dev mailing list: [https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E] > Integrate Cypress tests with Travis > --- > > Key: METRON-1803 > URL: https://issues.apache.org/jira/browse/METRON-1803 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > [DISCUSS] thread on the dev mailing list: > [https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1797) Add PCAP UI test implemented in Cypress.js
[ https://issues.apache.org/jira/browse/METRON-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tibor Meller updated METRON-1797: - Description: [DISCUSS] thread on the dev mailing list: [https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E] This PR adds Cypress.io to our project. It makes us able to run E2E tests, UI integration tests. Also higher level UI tests with mocked services. As part of this PR I also added tests to cover the following functionalities of PCAP UI panel: * checking running jobs on navigating to PCAP tab * submitting PCAP job request * requesting job status * process status in percentage * getting PCAP JSON * rendering PCAP table * showing PCAP details * navigating across pages * downloading PDML * cancelli PCAP query job * showing filter validation messages * showing date validation messages Please consider while reviewing that these tests are not meant to cover all the details of the functionalities above. The aim here is to extend the test coverage build up by a large number of Unit Tests with higher level UI tests. h2. Integration folder That contains the actual Cypress tests. The folder name comes from the Cypress naming convention, however, no real integration tests implemented here yet (at least no other than the integration of UI components to a whole application). We could change the name of the folder anytime we like to, but only one folder applicable. h2. Fixture folder The tests are running on mocked data. This makes us able to integrate them with Travis without the need to having PCAP related services available from our CI. Each file in the Fixture folder represents a response to a particular request. This makes it fairly easy to create and maintain. h1. How to run To run Cypress tests locally please do the following: # pull this branch to your workspace # move to metron-interface/metron-alerts folder # run: npm ci # run: scripts/start-server-for-e2e.sh # run: node_modules/cypress/bin/cypress run or run: node_modules/cypress/bin/cypress open if you like to use the dashboard. was:https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E > Add PCAP UI test implemented in Cypress.js > -- > > Key: METRON-1797 > URL: https://issues.apache.org/jira/browse/METRON-1797 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > [DISCUSS] thread on the dev mailing list: > [https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E] > This PR adds Cypress.io to our project. It makes us able to run E2E tests, UI > integration tests. Also higher level UI tests with mocked services. > As part of this PR I also added tests to cover the following functionalities > of PCAP UI panel: > * checking running jobs on navigating to PCAP tab > * submitting PCAP job request > * requesting job status > * process status in percentage > * getting PCAP JSON > * rendering PCAP table > * showing PCAP details > * navigating across pages > * downloading PDML > * cancelli PCAP query job > * showing filter validation messages > * showing date validation messages > Please consider while reviewing that these tests are not meant to cover all > the details of the functionalities above. The aim here is to extend the test > coverage build up by a large number of Unit Tests with higher level UI tests. > h2. Integration folder > That contains the actual Cypress tests. The folder name comes from the > Cypress naming convention, however, no real integration tests implemented > here yet (at least no other than the integration of UI components to a whole > application). We could change the name of the folder anytime we like to, but > only one folder applicable. > h2. Fixture folder > The tests are running on mocked data. This makes us able to integrate them > with Travis without the need to having PCAP related services available from > our CI. > Each file in the Fixture folder represents a response to a particular > request. This makes it fairly easy to create and maintain. > h1. How to run > To run Cypress tests locally please do the following: > # pull this branch to your workspace > # move to metron-interface/metron-alerts folder > # run: npm ci > # run: scripts/start-server-for-e2e.sh > # run: node_modules/cypress/bin/cypress run > or run: node_modules/cypress/bin/cypress open > if you like to use the dashboard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1797) Add PCAP UI test implemented in Cypress.js
[ https://issues.apache.org/jira/browse/METRON-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638237#comment-16638237 ] ASF GitHub Bot commented on METRON-1797: GitHub user tiborm opened a pull request: https://github.com/apache/metron/pull/1223 METRON-1797: Add PCAP UI test implemented in Cypress.js ## Contributor Comments [DISCUSS] thread on the dev mailing list: https://lists.apache.org/thread.html/b6a0272c7809c05e8b7aff20171720e8ec76f8a0e9481169c37a4a4a@%3Cdev.metron.apache.org%3E This PR adds Cypress.io to our project. It makes us able to run E2E tests, UI integration tests. Also higher level UI tests with mocked services. As part of this PR I also added tests to cover the following functionalities of PCAP UI panel: - checking running jobs on navigating to PCAP tab - submitting PCAP job request - requesting job status - process status in percentage - getting PCAP JSON - rendering PCAP table - showing PCAP details - navigating across pages - downloading PDML - cancelli PCAP query job - showing filter validation messages - showing date validation messages Please consider while reviewing that these tests are not meant to cover all the details of the functionalities above. The aim here is to extend the test coverage build up by a large number of Unit Tests with higher level UI tests. ## Integration folder That contains the actual Cypress tests. The folder name comes from the Cypress naming convention, however, no real integration tests implemented here yet (at least no other than the integration of UI components to a whole application). We could change the name of the folder anytime we like to, but only one folder applicable. ## Fixture folder The tests are running on mocked data. This makes us able to integrate them with Travis without the need to having PCAP related services available from our CI. Each file in the Fixture folder represents a response to a particular request. This makes it fairly easy to create and maintain. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[
https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638065#comment-16638065
]
ASF GitHub Bot commented on METRON-1749:
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1217#discussion_r222622136
--- Diff: metron-interface/metron-config/package.json ---
@@ -7,10 +7,11 @@
},
"angular-cli": {},
"scripts": {
-"build": "./node_modules/angular-cli/bin/ng build -prod",
+"build": "./node_modules/@angular/cli/bin/ng build --prod",
--- End diff --
It can also be written as `ng build --prod`
> Update Angular to latest release in Management UI
> -
>
> Key: METRON-1749
> URL: https://issues.apache.org/jira/browse/METRON-1749
> Project: Metron
> Issue Type: Improvement
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Major
>
> Currently, the Management UI is on Angular v2. Not only has there been many
> updates and improvements to Angular, that release is no longer supported by
> the Angular team. This means critical fixes and security patches are no
> longer being done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1217: METRON-1749: Update Angular to latest release in ...
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1217#discussion_r222622136
--- Diff: metron-interface/metron-config/package.json ---
@@ -7,10 +7,11 @@
},
"angular-cli": {},
"scripts": {
-"build": "./node_modules/angular-cli/bin/ng build -prod",
+"build": "./node_modules/@angular/cli/bin/ng build --prod",
--- End diff --
It can also be written as `ng build --prod`
---
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[ https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638064#comment-16638064 ] ASF GitHub Bot commented on METRON-1749: Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1217#discussion_r222619789 --- Diff: metron-interface/metron-config/package.json --- @@ -10,7 +10,7 @@ "build": "./node_modules/angular-cli/bin/ng build -prod", --- End diff -- it can also be written as `ng build -prod` > Update Angular to latest release in Management UI > - > > Key: METRON-1749 > URL: https://issues.apache.org/jira/browse/METRON-1749 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > Currently, the Management UI is on Angular v2. Not only has there been many > updates and improvements to Angular, that release is no longer supported by > the Angular team. This means critical fixes and security patches are no > longer being done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1217: METRON-1749: Update Angular to latest release in ...
Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1217#discussion_r222619789 --- Diff: metron-interface/metron-config/package.json --- @@ -10,7 +10,7 @@ "build": "./node_modules/angular-cli/bin/ng build -prod", --- End diff -- it can also be written as `ng build -prod` ---
[jira] [Commented] (METRON-1802) When switching parser topologies, some of the original parser topologies can fail to be shut down properly
[ https://issues.apache.org/jira/browse/METRON-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638049#comment-16638049 ] Tuncay commented on METRON-1802: Hello, profiler topology also has the same problem.. it was worked when after was tried more than three times. > When switching parser topologies, some of the original parser topologies can > fail to be shut down properly > -- > > Key: METRON-1802 > URL: https://issues.apache.org/jira/browse/METRON-1802 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Priority: Major > > This defect was reported by [~nickwallen] when testing PR[ > #1215|[https://github.com/apache/metron/pull/1215].] > Steps to reproduce: > # On full-dev, start with default parsers; [bro,snort] > # Change Metron Parsers setting to say ["bro"] under Services -> Metron -> > Configs -> 'Parsers' tab -> 'Metron Parsers' field. Save configuration and > restart required services to apply the config change. > # Observe that the original "snort" topology is never shutdown. One would > expect this to be shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1798: -- Fix Version/s: 0.6.1 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.6.1 > > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
