Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Girish N
Hi, I am trying to parse the syslog I am getting below exceptions. Kindly help to resolve the issue. Thanks Sample Syslog- Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend. 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log SYS_DELIMITED <%{NUMBER:queue_id}>+%{SYSLO

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Otto Fowler
Change the grokPath to match below and try. { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "log", "parserConfig": { "grokPath": "/patterns/log", "patternLabel": "SYS_DELIMITED" }} This path is not an absolute path. On September 6, 2017 at 05:16:19, Girish N (giri.na

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Girish N
Thanks for your response Otto Fowler, I tried with the below config. Still the same exception. Kindly let me know if anything else has to be changed. { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "log", "parserConfig": { "grokPath": "/patterns/log", "patternLabel": "

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Anand Subramanian
Hey Girish, Can you validate using http://grokconstructor.appspot.com/do/match that you can parse the sample message using the pattern? I tried with your example and it did not work. -Anand On 9/6/17, 4:06 PM, "Girish N" wrote: >Thanks for your response Otto Fowler, > >I tried with the bel

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Otto Fowler
Have you tested that grok against that message? On September 6, 2017 at 06:36:24, Girish N (giri.narasimha.mur...@gmail.com) wrote: Thanks for your response Otto Fowler, I tried with the below config. Still the same exception. Kindly let me know if anything else has to be changed. { "parserCl

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Girish N
Hello, Grok pattern was wrong, changed it as below and tested and got a match. Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend. Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )? %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg} Grok pattern 2 - %{SYSLOGL

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread ed d
Does anyone know of a solid repository of Grok patterns that match standard traffic patterns, like Cisco (non-ASA), JunOS, McAfee (all flavors), syslog, CheckPoint, etc? Maybe a repository sectioned into tech, like Firewalls, Endpoints, Network, VPN, and other security tools? Is there a place

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Otto Fowler
http://grokconstructor.appspot.com/do/match#result mentions a lot of patterns. Maybe open a jira? On September 6, 2017 at 07:08:13, ed d (ragdel...@hotmail.com) wrote: Does anyone know of a solid repository of Grok patterns that match standard traffic patterns, like Cisco (non-ASA), JunOS, McAf

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread ed d
Well, that looks like a solid repository to me. No jira, will bookmark it and move on. :-) Thanks Otto. Sent from my Verizon 4G LTE Droid On Sep 6, 2017 7:42 AM, Otto Fowler wrote: http://grokconstructor.appspot.com/do/match#result mentions a lot of patterns. Maybe open a jira? On September

Re: Exceptions in Parser, enrichments and indexing topology

2017-09-06 Thread Otto Fowler
https://issues.apache.org/jira/browse/METRON-1157 Feel free to review and add your comments. It is important that ideas from the community get captured in jira, that is the strength of the community effort and the apache/open source system. It is what will help us to innovate. We should be so lu

Re: [GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread Otto Fowler
Bump on this. I would like to get this in before I take master again. On September 2, 2017 at 23:28:45, ottobackwards (g...@git.apache.org) wrote: GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/731 METRON-1146 Refactor for ParserExtensionConfig and Rest

[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...

2017-09-06 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/727#discussion_r137266659 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java --- @@ -450,4 +454,40 @@ public Object a

[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/727#discussion_r137270218 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java --- @@ -450,4 +454,40 @@ public Obj

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137273563 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -91,

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/729 Thanks for the submission, this looks good. Regarding this also living in 777, I think this is worth bringing in on its own, given the nature of the fix. ---

[GitHub] metron issue #730: METRON-1151: Grok patterns in HDFS should be owned by met...

2017-09-06 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/730 +1 by inspection ---

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137284827 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -91,1

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/729 The fix in 777 removes the need for a temp path/ temp file for this. If this resolves in the mean time that is great, but I'm not sure that it will replace what is in the feature branch. I'll

[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...

2017-09-06 Thread justinleet
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/734 METRON-1158: Build backend for grouping alerts into meta alerts ## Contributor Comments The motivation for this is also in the ticket, but the idea is that we want to be able to group alerts

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/729 I'm guessing it probably doesn't replace the changes in the feature branch, especially if it remove the need for temp path. I'm looking at this as the fix until that branch comes in, honestly.

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137305664 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137305828 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/729 I'm going to throw a quick review at this. I do not want to hold it back, but this is the place to ask questions. I want to understand these things for comparison ---

[GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/731#discussion_r137309525 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/ExtensionServiceImpl.java --- @@ -528,20 +527,26 @@ private void

[GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/731#discussion_r137308842 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/ParserExtensionController.java --- @@ -90,8 +91,8 @@ @ApiOp

[GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/731#discussion_r137316439 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/ParserExtensionController.java --- @@ -90,8 +91,8 @@ @Ap

[GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/731#discussion_r137316957 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/ExtensionServiceImpl.java --- @@ -528,20 +527,26 @@ private v

[GitHub] metron pull request #731: METRON-1146 Refactor for ParserExtensionConfig and...

2017-09-06 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/731#discussion_r137320162 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/ParserExtensionController.java --- @@ -90,8 +91,8 @@ @ApiOp

[GitHub] metron issue #731: METRON-1146 Refactor for ParserExtensionConfig and Rest g...

2017-09-06 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/731 +1 Thanks, Otto ---

[GitHub] metron issue #731: METRON-1146 Refactor for ParserExtensionConfig and Rest g...

2017-09-06 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/731 Hi @ottobackwards , I guess you wanted to name the PR title as METRON-1136 instead of METRON-1146? ---

[GitHub] metron issue #731: METRON-1136 Refactor for ParserExtensionConfig and Rest g...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/731 I just got done doing that! Luckily the scripts that I use have you verify everything, and I caught the different message :) ---

Metron master installation

2017-09-06 Thread Laurens Vets
Hi list, In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari. Any idea what might be going on or where I can start troubleshooting th

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137353301 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -90,1

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/729 Please see https://issues.apache.org/jira/browse/METRON-1150 for a description of why this is needed. Let me know if that isn't clear enough or you need more info. ---

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137354784 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -90,1

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137359517 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

2017-09-06 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/729#discussion_r137362276 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml --- @@ -90,1

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/729 +1 by inspection, @merrimanr I would run this up in full dev one more time after your last change to make sure nothing wacky goes on with ambari and ambari format(). ---

[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...

2017-09-06 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/734 ## Create Meta Alert The first thing to do is to build and deploy full dev with this PR. Afterwards, we're going to set up a bit of base data. Retrieve the current list of indi

[DISCUSS] metron-config build failure on Centos 7

2017-09-06 Thread Ian Abreu
Hello all, After spending a few days on it now, I'm stuck. I can't figure out how to build metron-config on Centos 7. I've gone down the rabbit hole so far as to believe that it's a problem with npm and dependencies for whatever reason not installing properly, but I'm posting here in the hope

[GitHub] metron pull request #735: METRON-1160 Blueprint configuration validation fai...

2017-09-06 Thread nickwallen
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/735 METRON-1160 Blueprint configuration validation failed: Missing required properties Deployment to AWS using `metron-deployment/amazon-ec2` fails with the following error message. This PR addresse

Re: [DISCUSS] Metron release 0.4.1

2017-09-06 Thread Nick Allen
We won't be able to perform an AWS deployment without this PR. In the past, we have used the AWS deployment as RC validation criteria, so I thought I would mention it. https://github.com/apache/metron/pull/735 On Tue, Sep 5, 2017 at 1:01 PM Matt Foley wrote: > Great, working on it! > > > > F

[GitHub] metron issue #735: METRON-1160 Blueprint configuration validation failed: Mi...

2017-09-06 Thread mattf-horton
Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/735 +1 by inspection, trusting your manual test. Please commit asap. Thanks! ---

Re: [DISCUSS] metron-config build failure on Centos 7

2017-09-06 Thread Nick Allen
What version of Metron are you running? This error seems a bit different, but do you have the patch for this issue that was fixed a while back? https://github.com/apache/metron/pull/691 https://issues.apache.org/jira/browse/METRON-1104 On Wed, Sep 6, 2017 at 3:41 PM Ian Abreu wrote: > Hello al

Re: [DISCUSS] metron-config build failure on Centos 7

2017-09-06 Thread Nick Allen
Actually, ignore my previous advice. That was completely wrong. I think you are running an old version of Node. Per our docs, you need 6.9+ [1]. It appears you are running 6.2. [1] https://github.com/apache/metron/tree/master/metron-interface/metron-config#prerequisites On Wed, Sep 6, 2017

Re: Question about SourceHandler and HdfsWriter

2017-09-06 Thread Otto Fowler
Bump - any thoughts? this seems to be cropping up for folks. On September 5, 2017 at 07:01:09, Otto Fowler (ottobackwa...@gmail.com) wrote: Note: I don’t think my pseudo code is actually the best way to resolve this On September 4, 2017 at 21:37:29, Otto Fowler (ottobackwa...@gmail.com) wro

Ambari Metrics Collector failing...

2017-09-06 Thread Laurens Vets
In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any idea what might be going on or where I can start troubleshooting t

Re: Ambari Metrics Collector failing...

2017-09-06 Thread zeo...@gmail.com
I'm seeing the same issue right now as well on my fresh bare metal install of HDP (no Metron yet), haven't dug into it further to troubleshoot. Jon On Wed, Sep 6, 2017, 18:22 Laurens Vets wrote: > In preparation of 0.4.1-rc, I'm trying to install the current github > master and I'm running into

Re: Ambari Metrics Collector failing...

2017-09-06 Thread Otto Fowler
https://issues.apache.org/jira/browse/ZOOKEEPER-1582? On September 6, 2017 at 19:46:03, zeo...@gmail.com (zeo...@gmail.com) wrote: org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)

[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...

2017-09-06 Thread nickwallen
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/736 METRON-1162 Apply Live Messages to the Profile Debugger I want to be able to use `PROFILER_APPLY` with live messages from a Metron cluster. For example, I would like to be able to grab 10 message

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

2017-09-06 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/729 question - can the rest service delete these temp files? From full_dev? I am seeing an issue where rest cannot delete files it creates because the permissions don't match the parent dir

Re: [DISCUSS] Metron release 0.4.1

2017-09-06 Thread Matt Foley
I’ve got a blocker bug, https://issues.apache.org/jira/browse/METRON-1163 , RAT failures for metron-interface/metron-alerts.  A comment in the jira suggests a way to address it, but someone familiar with the code should look at it. Raghu, would you be able to take a look? Thanks, --Matt

[GitHub] metron pull request #700: METRON-1077: Use column meta data end point in ale...

2017-09-06 Thread iraghumitra
Github user iraghumitra commented on a diff in the pull request: https://github.com/apache/metron/pull/700#discussion_r137454065 --- Diff: metron-interface/metron-alerts/src/app/utils/metron-rest-api-utils.ts --- @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Fou

[GitHub] metron pull request #700: METRON-1077: Use column meta data end point in ale...

2017-09-06 Thread iraghumitra
Github user iraghumitra commented on a diff in the pull request: https://github.com/apache/metron/pull/700#discussion_r137454070 --- Diff: metron-interface/metron-alerts/src/app/service/alert.service.ts --- @@ -39,21 +41,31 @@ export class AlertService { private

[GitHub] metron pull request #700: METRON-1077: Use column meta data end point in ale...

2017-09-06 Thread iraghumitra
Github user iraghumitra commented on a diff in the pull request: https://github.com/apache/metron/pull/700#discussion_r137454061 --- Diff: metron-interface/metron-alerts/src/app/service/cluster-metadata.service.ts --- @@ -16,14 +16,20 @@ * limitations under the License.