[jira] [Created] (METRON-1754) Add users to role mapping for ldap based Metron REST

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1754:
---

 Summary: Add users to role mapping for ldap based Metron REST
 Key: METRON-1754
 URL: https://issues.apache.org/jira/browse/METRON-1754
 Project: Metron
  Issue Type: Sub-task
Reporter: Simon Elliston Ball


We need a means of mapping users to roles in Spring Security for Metron REST. 
This should be done via the new LDAP group lookups, and be similar to the way 
Hadoop groups are mapped for consistency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594599#comment-16594599
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198753
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/java/org/apache/metron/ui/AlertsApplication.java
 ---
@@ -0,0 +1,31 @@
+/**
--- End diff --

Done


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594601#comment-16594601
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198883
  
--- Diff: metron-interface/metron-alerts-host/pom.xml ---
@@ -0,0 +1,143 @@
+
--- End diff --

Done, and agreed, it should be a formal style guide addition, and part of 
the formatting configs on the contributor wiki really.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198883
  
--- Diff: metron-interface/metron-alerts-host/pom.xml ---
@@ -0,0 +1,143 @@
+
--- End diff --

Done, and agreed, it should be a formal style guide addition, and part of 
the formatting configs on the contributor wiki really.


---

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198753
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/java/org/apache/metron/ui/AlertsApplication.java
 ---
@@ -0,0 +1,31 @@
+/**
--- End diff --

Done


---

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198659
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Done, I moved the spring versions up to interface.


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594597#comment-16594597
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213198659
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Done, I moved the spring versions up to interface.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594426#comment-16594426
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@nickwallen I committed the readme. Please feel free to correct it if 
something is wrong. 


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

[liuy-tnz]

Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@nickwallen I committed the readme. Please feel free to correct it if 
something is wrong. 


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594252#comment-16594252
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213124491
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Makes sense. I guess I'm half thinking about the idea of 3rd party 
extensions, but we can at least be consistent within the official bits and let 
people override if they absolutely for some strange reason have to definitely 
use something different. 


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594251#comment-16594251
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213124377
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

Right, my mistake. We should change it and carry through. Would it make 
sense to do that as part of the follow on PR for the mpack work, since that 
will be where the bulk of the changes for this would be?


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213124491
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Makes sense. I guess I'm half thinking about the idea of 3rd party 
extensions, but we can at least be consistent within the official bits and let 
people override if they absolutely for some strange reason have to definitely 
use something different. 


---

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213124377
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

Right, my mistake. We should change it and carry through. Would it make 
sense to do that as part of the follow on PR for the mpack work, since that 
will be where the bulk of the changes for this would be?


---

[jira] [Commented] (METRON-1714) Create RPM Packaging for the Batch Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594247#comment-16594247
 ] 

ASF GitHub Bot commented on METRON-1714:


Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1163


> Create RPM Packaging for the Batch Profiler
> ---
>
> Key: METRON-1714
> URL: https://issues.apache.org/jira/browse/METRON-1714
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create RPM packaging for the Batch Profiler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1714) Create RPM Packaging for the Batch Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594246#comment-16594246
 ] 

ASF GitHub Bot commented on METRON-1714:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
Many thanks for the review. This PR has been merged into the feature branch.


> Create RPM Packaging for the Batch Profiler
> ---
>
> Key: METRON-1714
> URL: https://issues.apache.org/jira/browse/METRON-1714
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create RPM packaging for the Batch Profiler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1163: METRON-1714 Create RPM Packaging for the Batch Profiler ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
Many thanks for the review. This PR has been merged into the feature branch.


---

[GitHub] metron pull request #1163: METRON-1714 Create RPM Packaging for the Batch Pr...

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1163


---

[jira] [Commented] (METRON-1714) Create RPM Packaging for the Batch Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594238#comment-16594238
 ] 

ASF GitHub Bot commented on METRON-1714:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1163
  
I tested this in full dev and everything is as expected.  The README is 
clear to me too.  +1


> Create RPM Packaging for the Batch Profiler
> ---
>
> Key: METRON-1714
> URL: https://issues.apache.org/jira/browse/METRON-1714
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create RPM packaging for the Batch Profiler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1163: METRON-1714 Create RPM Packaging for the Batch Profiler ...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1163
  
I tested this in full dev and everything is as expected.  The README is 
clear to me too.  +1


---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594223#comment-16594223
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@JonZeolla, this is fine. I understand


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

[liuy-tnz]

Github user liuy-tnz commented on the issue:

https://github.com/apache/metron/pull/1171
  
@JonZeolla, this is fine. I understand


---

[jira] [Commented] (METRON-1752) Prevent package.lock from changing during build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594177#comment-16594177
 ] 

ASF GitHub Bot commented on METRON-1752:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1177


> Prevent package.lock from changing during build
> ---
>
> Key: METRON-1752
> URL: https://issues.apache.org/jira/browse/METRON-1752
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Priority: Minor
>
> As referenced in this mail list discussion, the package-lock.json file in 
> metron-alerts updates whenever we run a maven build: 
> https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E
> The fix for this was originally included in [PR #1096, which upgrades the 
> Angular version used in the alerts 
> ui|https://github.com/apache/metron/pull/1096], but as mentioned in the 
> comments, it would be best to fix this bug in a separate issue in order to 
> simplify its inclusion in the next release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1177: METRON-1752: Prevent package.lock from changing d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1177


---

[jira] [Commented] (METRON-1752) Prevent package.lock from changing during build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594172#comment-16594172
 ] 

ASF GitHub Bot commented on METRON-1752:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1177
  
I tested this and was able to build metron-alerts and metron-config without 
the package-lock.json changing.  +1


> Prevent package.lock from changing during build
> ---
>
> Key: METRON-1752
> URL: https://issues.apache.org/jira/browse/METRON-1752
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Priority: Minor
>
> As referenced in this mail list discussion, the package-lock.json file in 
> metron-alerts updates whenever we run a maven build: 
> https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E
> The fix for this was originally included in [PR #1096, which upgrades the 
> Angular version used in the alerts 
> ui|https://github.com/apache/metron/pull/1096], but as mentioned in the 
> comments, it would be best to fix this bug in a separate issue in order to 
> simplify its inclusion in the next release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1177: METRON-1752: Prevent package.lock from changing during b...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1177
  
I tested this and was able to build metron-alerts and metron-config without 
the package-lock.json changing.  +1


---

[jira] [Commented] (METRON-1714) Create RPM Packaging for the Batch Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594165#comment-16594165
 ] 

ASF GitHub Bot commented on METRON-1714:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
I added a README for the Spark Profiler in this PR.


> Create RPM Packaging for the Batch Profiler
> ---
>
> Key: METRON-1714
> URL: https://issues.apache.org/jira/browse/METRON-1714
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create RPM packaging for the Batch Profiler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1163: METRON-1714 Create RPM Packaging for the Batch Profiler ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
I added a README for the Spark Profiler in this PR.


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594162#comment-16594162
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/
  
Here is what I think is pending on this PR:
- Address the minor comments I've made inline
- Update the metron-rest and/or other relevant READMEs with the new 
authentication approach
- Create a follow on Jira to implement a simple LDAP group to role mapping 
so that we're not introducing a regression
- Create a follow on Jira to implement roles properly (would that be 
outside of this feature branch?)
- Add architecture documentation either in this PR or create a follow on 
Jira for it
- Create a follow on Jira to convert metron-rest to use the testing 
infrastructure in metron-security


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594160#comment-16594160
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213093853
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Sure if you think certain versions should be configured separately.  At the 
very least I think we want the spring boot version to be consistent.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1111: METRON-1665 Host UIs on Spring Boot and add SSO authenti...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/
  
Here is what I think is pending on this PR:
- Address the minor comments I've made inline
- Update the metron-rest and/or other relevant READMEs with the new 
authentication approach
- Create a follow on Jira to implement a simple LDAP group to role mapping 
so that we're not introducing a regression
- Create a follow on Jira to implement roles properly (would that be 
outside of this feature branch?)
- Add architecture documentation either in this PR or create a follow on 
Jira for it
- Create a follow on Jira to convert metron-rest to use the testing 
infrastructure in metron-security


---

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213093853
  
--- Diff: metron-interface/metron-ui-host/pom.xml ---
@@ -0,0 +1,142 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+metron-ui-host
+jar
+
+Metron Generic UI Host
+Spring Server to host config ui
+
+
+org.apache.metron
+metron-interface
+0.5.1
+
+
+
--- End diff --

Sure if you think certain versions should be configured separately.  At the 
very least I think we want the spring boot version to be consistent.


---

[jira] [Commented] (METRON-1665) Move hosting of Alerts and Config UIs from Nodejs to Spring Boot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594158#comment-16594158
 ] 

ASF GitHub Bot commented on METRON-1665:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213093319
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

This environment variable is sourced from /etc/default/metron which is 
bound to the spring options setting in the REST tab in Ambari.  I tested it in 
full dev and the option is indeed added to alerts and config UI start commands.


> Move hosting of Alerts and Config UIs from Nodejs to Spring Boot
> 
>
> Key: METRON-1665
> URL: https://issues.apache.org/jira/browse/METRON-1665
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
>Priority: Major
>
> The current UIs are served up by very lightweight nodejs applications, which 
> serve the static bundle files produced by the angular build process, and 
> proxies the rest api.
> The proposal is to use a spring boot application, allowing us to harmonise 
> the security implementation across the UI static servers and the REST layer, 
> and to provide a routing platform for later microservices.
> The UIs currently proxy to the REST API to avoid CORS issues, this will be 
> achieved with Zuul.
> Spring Security will also be extended to use a Knox SSO authenticator. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1111: METRON-1665 Host UIs on Spring Boot and add SSO a...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/#discussion_r213093319
  
--- Diff: 
metron-interface/metron-alerts-host/src/main/scripts/metron-alerts.sh ---
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${METRON_SSL_PASSWORD}" ]; then
+echo "METRON_SSL_PASSWORD unset."
+fi
+
+METRON_VERSION=${project.version}
+METRON_HOME="${METRON_HOME:-/usr/metron/${METRON_VERSION}}"
+METRON_SYSCONFIG="${METRON_SYSCONFIG:-/etc/default/metron}"
+
+echo "METRON_VERSION=${METRON_VERSION}"
+echo "METRON_HOME=${METRON_HOME}"
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+
+if [ -f "$METRON_SYSCONFIG" ]; then
+echo "METRON_SYSCONFIG=${METRON_SYSCONFIG}"
+set -a
+. "$METRON_SYSCONFIG"
+fi
+
+echo "METRON_SPRING_PROFILES_ACTIVE=${METRON_SPRING_PROFILES_ACTIVE}"
+
+METRON_CONFIG_LOCATION=" 
--spring.config.location=classpath:/application.yml,$METRON_HOME/config/alerts_ui.yml"
+echo "METRON_CONFIG_LOCATION=${METRON_CONFIG_LOCATION}"
+METRON_SPRING_OPTIONS+=${METRON_CONFIG_LOCATION}
--- End diff --

This environment variable is sourced from /etc/default/metron which is 
bound to the spring options setting in the REST tab in Ambari.  I tested it in 
full dev and the option is indeed added to alerts and config UI start commands.


---

[jira] [Commented] (METRON-1708) Run the Batch Profiler in Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594026#comment-16594026
 ] 

ASF GitHub Bot commented on METRON-1708:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1161
  
This has been merged.


> Run the Batch Profiler in Spark
> ---
>
> Key: METRON-1708
> URL: https://issues.apache.org/jira/browse/METRON-1708
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create tooling to allow the user to run the Batch Profiler from the command 
> line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1708) Run the Batch Profiler in Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594027#comment-16594027
 ] 

ASF GitHub Bot commented on METRON-1708:


Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1161


> Run the Batch Profiler in Spark
> ---
>
> Key: METRON-1708
> URL: https://issues.apache.org/jira/browse/METRON-1708
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create tooling to allow the user to run the Batch Profiler from the command 
> line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1161: METRON-1708 Run the Batch Profiler in Spark [Feat...

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1161


---

[jira] [Commented] (METRON-1708) Run the Batch Profiler in Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594025#comment-16594025
 ] 

ASF GitHub Bot commented on METRON-1708:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1161
  
Thanks @merrimanr !  I updated the testing steps in the PR description to 
account for what you found.  

I will add a README for the Spark Profiler in #1163 .  That will allow it 
to include some installation steps.


> Run the Batch Profiler in Spark
> ---
>
> Key: METRON-1708
> URL: https://issues.apache.org/jira/browse/METRON-1708
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create tooling to allow the user to run the Batch Profiler from the command 
> line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1161: METRON-1708 Run the Batch Profiler in Spark [Feature Bra...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1161
  
This has been merged.


---

[GitHub] metron issue #1161: METRON-1708 Run the Batch Profiler in Spark [Feature Bra...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1161
  
Thanks @merrimanr !  I updated the testing steps in the PR description to 
account for what you found.  

I will add a README for the Spark Profiler in #1163 .  That will allow it 
to include some installation steps.


---

[GitHub] metron issue #1163: METRON-1714 Create RPM Packaging for the Batch Profiler ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
FYI - I need to create a README for the Spark Profiler project.  I'll add 
that to this PR before it goes in.


---

[jira] [Commented] (METRON-1714) Create RPM Packaging for the Batch Profiler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594004#comment-16594004
 ] 

ASF GitHub Bot commented on METRON-1714:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1163
  
FYI - I need to create a README for the Spark Profiler project.  I'll add 
that to this PR before it goes in.


> Create RPM Packaging for the Batch Profiler
> ---
>
> Key: METRON-1714
> URL: https://issues.apache.org/jira/browse/METRON-1714
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create RPM packaging for the Batch Profiler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1708) Run the Batch Profiler in Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594003#comment-16594003
 ] 

ASF GitHub Bot commented on METRON-1708:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1161
  
I had to change the period settings in `batch-profiler.properties` to match 
what's in zookeeper.  Everything worked as expected after that.  Assuming the 
README will be updated in a future PR.  +1


> Run the Batch Profiler in Spark
> ---
>
> Key: METRON-1708
> URL: https://issues.apache.org/jira/browse/METRON-1708
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Create tooling to allow the user to run the Batch Profiler from the command 
> line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1161: METRON-1708 Run the Batch Profiler in Spark [Feature Bra...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/1161
  
I had to change the period settings in `batch-profiler.properties` to match 
what's in zookeeper.  Everything worked as expected after that.  Assuming the 
README will be updated in a future PR.  +1


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593994#comment-16593994
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213051917
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

yeah


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213051917
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

yeah


---

[jira] [Commented] (METRON-1752) Prevent package.lock from changing during build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593990#comment-16593990
 ] 

ASF GitHub Bot commented on METRON-1752:


GitHub user sardell opened a pull request:

https://github.com/apache/metron/pull/1177

METRON-1752: Prevent package.lock from changing during build

## Contributor Comments
This PR updates the build script to use `npm ci` instead of `npm install`. 
The former command installs exactly what is in a project's package-lock.json 
file, whereas the later will reference package.json and update the lock file. 
Besides causing confusion with an unexpected file change, `npm install` had the 
potential to install different package versions of dependencies in different 
environments depending on when the install was last done. 

### Changes Included
* Update pom.xml files in both the alerts and management ui to use `npm ci` 
instead of `npm install`.
* Add a package-lock.json file to the management ui. There were no new 
dependencies added, just a locked version of dependencies that already existed 
in the project. This was done because the new command relies on reading the 
lock file to install exact versions of dependencies.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sardell/metron METRON-1752

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1177


commit bf03de93441efd4afa24f351b470140cb9058b6b
Author: Shane Ardell 
Date:   2018-08-27T17:18:56Z

update build to use npm ci

commit 6fd23f4f092a2ba02a912f4a71cd57abbf4d7d14
Author: Shane Ardell 
Date:   2018-08-27T17:19:28Z

add lock file to management ui




> Prevent package.lock from changing during build
> ---
>
> Key: METRON-1752
> URL: https://issues.apache.org/jira/browse/METRON-1752
> P

[GitHub] metron pull request #1177: METRON-1752: Prevent package.lock from changing d...

[sardell]

GitHub user sardell opened a pull request:

https://github.com/apache/metron/pull/1177

METRON-1752: Prevent package.lock from changing during build

## Contributor Comments
This PR updates the build script to use `npm ci` instead of `npm install`. 
The former command installs exactly what is in a project's package-lock.json 
file, whereas the later will reference package.json and update the lock file. 
Besides causing confusion with an unexpected file change, `npm install` had the 
potential to install different package versions of dependencies in different 
environments depending on when the install was last done. 

### Changes Included
* Update pom.xml files in both the alerts and management ui to use `npm ci` 
instead of `npm install`.
* Add a package-lock.json file to the management ui. There were no new 
dependencies added, just a locked version of dependencies that already existed 
in the project. This was done because the new command relies on reading the 
lock file to install exact versions of dependencies.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sardell/metron METRON-1752

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1177


commit bf03de93441efd4afa24f351b470140cb9058b6b
Author: Shane Ardell 
Date:   2018-08-27T17:18:56Z

update build to use npm ci

commit 6fd23f4f092a2ba02a912f4a71cd57abbf4d7d14
Author: Shane Ardell 
Date:   2018-08-27T17:19:28Z

add lock file to management ui




---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593974#comment-16593974
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048634
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Is it worth moving the SyslogParserBuilder to the configure() method and 
just storing it off, rather than recreating every time?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048634
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Is it worth moving the SyslogParserBuilder to the configure() method and 
just storing it off, rather than recreating every time?


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593970#comment-16593970
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048275
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Created https://issues.apache.org/jira/browse/METRON-1753


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048275
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Created https://issues.apache.org/jira/browse/METRON-1753


---

[jira] [Created] (METRON-1753) Allow MessageParsers to be have configs be updateable

[Justin Leet (JIRA)]

Justin Leet created METRON-1753:
---

 Summary: Allow MessageParsers to be have configs be updateable
 Key: METRON-1753
 URL: https://issues.apache.org/jira/browse/METRON-1753
 Project: Metron
  Issue Type: Improvement
Reporter: Justin Leet


This came up in a review 
https://github.com/apache/metron/pull/1175#discussion_r212985523

ParserBolt doesn't implement the `reloadCallback` that the enrichment bolts do. 
This means they can't update on config changes the same way.

I expect adding this capability is something to the effect of:
* Add update method to MessageParser
* Implement reloadCallback to update the parser configs and make sure anything 
else in the bolt gets updated as necessary.
* Tests and docs
* Update any parsers that use configs to handle this appropriately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593935#comment-16593935
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1171
  
> @liuy-tnz: I would love to add README. Please advise me where the README 
goes? Should it be part of the main readme or stay with its Java code here?

It probably makes sense to have one specific to Panos at 
`metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/README.md`.




> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1171
  
> @liuy-tnz: I would love to add README. Please advise me where the README 
goes? Should it be part of the main readme or stay with its Java code here?

It probably makes sense to have one specific to Panos at 
`metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/README.md`.




---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593921#comment-16593921
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213039514
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I don't think I want to touch all the parsers for this PR , there might be 
more than one parser follow on depending on how many have configurations.  Can 
you create  a jira or shall I?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213039514
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I don't think I want to touch all the parsers for this PR , there might be 
more than one parser follow on depending on how many have configurations.  Can 
you create  a jira or shall I?


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593874#comment-16593874
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213027440
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I'm okay with a follow-on (particularly if it's a lot of work, or if it's 
risky).  I'd just make sure the ticket includes updating any parsers to use the 
functionality.

It's probably not too hard to add (and I may be being incredibly blasé 
about something here).  I'd expect it to be:

* Add update method to `MessageParser`
* Implement `reloadCallback` to update the parser configs and make sure 
anything else in the bolt gets updated as necessary.
* Tests and docs


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213027440
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I'm okay with a follow-on (particularly if it's a lot of work, or if it's 
risky).  I'd just make sure the ticket includes updating any parsers to use the 
functionality.

It's probably not too hard to add (and I may be being incredibly blasé 
about something here).  I'd expect it to be:

* Add update method to `MessageParser`
* Implement `reloadCallback` to update the parser configs and make sure 
anything else in the bolt gets updated as necessary.
* Tests and docs


---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593863#comment-16593863
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1171
  
Sorry, I cannot @liuy-tnz 


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1171
  
Sorry, I cannot @liuy-tnz 


---

[jira] [Commented] (METRON-1476) Update angular

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593842#comment-16593842
 ] 

ASF GitHub Bot commented on METRON-1476:


Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1096
  
@justinleet I completely agree. This PR is pretty big, so submitting the 
`npm ci` fix in a separate PR would mitigate our risk if trying to include it 
in the next release. I created a separate ASF issue 
[here](https://issues.apache.org/jira/browse/METRON-1752). I'll try to submit a 
PR for that today.


> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1096: METRON-1476: Update angular

[sardell]

Github user sardell commented on the issue:

https://github.com/apache/metron/pull/1096
  
@justinleet I completely agree. This PR is pretty big, so submitting the 
`npm ci` fix in a separate PR would mitigate our risk if trying to include it 
in the next release. I created a separate ASF issue 
[here](https://issues.apache.org/jira/browse/METRON-1752). I'll try to submit a 
PR for that today.


---

[jira] [Created] (METRON-1752) Prevent package.lock from changing during build

[Shane Ardell (JIRA)]

Shane Ardell created METRON-1752:


 Summary: Prevent package.lock from changing during build
 Key: METRON-1752
 URL: https://issues.apache.org/jira/browse/METRON-1752
 Project: Metron
  Issue Type: Bug
Reporter: Shane Ardell


As referenced in this mail list discussion, the package-lock.json file in 
metron-alerts updates whenever we run a maven build: 
https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E

The fix for this was originally included in [PR #1096, which upgrades the 
Angular version used in the alerts 
ui|https://github.com/apache/metron/pull/1096], but as mentioned in the 
comments, it would be best to fix this bug in a separate issue in order to 
simplify its inclusion in the next release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593830#comment-16593830
 ] 

ASF GitHub Bot commented on METRON-1748:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1174
  
BTW - This should be ready to go.  I fixed the sporadic test failures.


> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output 
> of the Storm Profiler Integration Test.  This is better validation that 
> things are working end-to-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1174: METRON-1748 Improve Storm Profiler Integration Test

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1174
  
BTW - This should be ready to go.  I fixed the sporadic test failures.


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593826#comment-16593826
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213016887
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

You ok with that as part of this pr?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213016887
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

You ok with that as part of this pr?


---

[jira] [Commented] (METRON-1476) Update angular

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593799#comment-16593799
 ] 

ASF GitHub Bot commented on METRON-1476:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
Could we migrate the `npm ci` change from [this 
discussion](https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E)
 out to a separate PR?

I personally would like to see that small change go into the upcoming 0.6.0 
release so that everything is deterministic, but pulling in a large upgrade 
like this might be a little riskier (although let me know if you disagree).


> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1096: METRON-1476: Update angular

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
Could we migrate the `npm ci` change from [this 
discussion](https://lists.apache.org/thread.html/d0da3647f2955b4257c3eb0d89235779aed64a58097b416a18de6cd9@%3Cdev.metron.apache.org%3E)
 out to a separate PR?

I personally would like to see that small change go into the upcoming 0.6.0 
release so that everything is deterministic, but pulling in a large upgrade 
like this might be a little riskier (although let me know if you disagree).


---

[jira] [Commented] (METRON-1476) Update angular

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593763#comment-16593763
 ] 

ASF GitHub Bot commented on METRON-1476:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
@sardell Can you deconflict this?


> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1096: METRON-1476: Update angular

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
@sardell Can you deconflict this?


---

[jira] [Commented] (METRON-1743) CEF testPaloAltoCEF test using a confusing variable name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593725#comment-16593725
 ] 

ASF GitHub Bot commented on METRON-1743:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1173
  
+1 by inspection. Thanks!


> CEF testPaloAltoCEF test using a confusing variable name
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> A confusing test URL 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1173: METRON-1743: CEF test confusing URL

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1173
  
+1 by inspection. Thanks!


---

[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593699#comment-16593699
 ] 

ASF GitHub Bot commented on METRON-1751:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1176

METRON-1751 Storm Profiler dies when consuming null message

The Storm Profiler dies if a null message is consumed from the input Kafka 
topic.  The Profiler should skip past any null messages and continue processing.

## Testing

1. Create a "hello-world" profile using the Storm Profiler as outlined in 
the README.

1. Push a null message on to the "indexing" Kafka topic.

1. Make sure the Storm topology does not die and that a profile measurement 
is flushed.

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1751

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1176


commit 5123f28ea76ac47d97772d2d4bf0c193aa156697
Author: Nick Allen 
Date:   2018-08-27T13:56:28Z

METRON-1751 Storm Profiler dies when consuming null message




> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is 
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Async loop died! java.lang.RuntimeException: 
> java.lang.NullPointerException at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at 
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: 
> java.lang.NullPointerException at java.lang.String.(String.java:491) 
> ~[?:1.8.0_181] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
>  ~[stormjar.jar:?] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
>  ~[stormjar.jar:?] at 
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
>  ~[storm-core-

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593700#comment-16593700
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r212985523
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

This is half comment, half question.

It seems odd to recreate the SyslogParserBuilder every `parser`. I dug in a 
bit, and I expected (personally expected, not "Metron itself expects x 
condition") this process to be somewhat similar to the enrichment where the 
bolt implements the `reloadCallback` method (e.g. 
[UnifiedEnrichmentBolt](https://github.com/apache/metron/blob/1d95b8316a18097747be116a0276c56b894fb79c/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L229))
 which would then delegate updating the config of the underlying pieces.

E.g. here I would expect the `SyslogParser` to be created a priori and then 
when the parser config gets updated `reloadCallback` would be called and this 
would be updated (and in this case recreated).

Looking into it a bit further, it looks like ParserBolt has the appropriate 
method passed down to it, but chooses not to implement it.  I suspect it's 
because nothing the underlying parsers don't update configs, although I didn't 
check all of them.

Would it be reasonable to get that setup in the ParserBolt and then handle 
the `SyslogParser` object that way, rather than recreating it every time?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r212985523
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

This is half comment, half question.

It seems odd to recreate the SyslogParserBuilder every `parser`. I dug in a 
bit, and I expected (personally expected, not "Metron itself expects x 
condition") this process to be somewhat similar to the enrichment where the 
bolt implements the `reloadCallback` method (e.g. 
[UnifiedEnrichmentBolt](https://github.com/apache/metron/blob/1d95b8316a18097747be116a0276c56b894fb79c/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L229))
 which would then delegate updating the config of the underlying pieces.

E.g. here I would expect the `SyslogParser` to be created a priori and then 
when the parser config gets updated `reloadCallback` would be called and this 
would be updated (and in this case recreated).

Looking into it a bit further, it looks like ParserBolt has the appropriate 
method passed down to it, but chooses not to implement it.  I suspect it's 
because nothing the underlying parsers don't update configs, although I didn't 
check all of them.

Would it be reasonable to get that setup in the ParserBolt and then handle 
the `SyslogParser` object that way, rather than recreating it every time?


---

[GitHub] metron pull request #1176: METRON-1751 Storm Profiler dies when consuming nu...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1176

METRON-1751 Storm Profiler dies when consuming null message

The Storm Profiler dies if a null message is consumed from the input Kafka 
topic.  The Profiler should skip past any null messages and continue processing.

## Testing

1. Create a "hello-world" profile using the Storm Profiler as outlined in 
the README.

1. Push a null message on to the "indexing" Kafka topic.

1. Make sure the Storm topology does not die and that a profile measurement 
is flushed.

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1751

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1176


commit 5123f28ea76ac47d97772d2d4bf0c193aa156697
Author: Nick Allen 
Date:   2018-08-27T13:56:28Z

METRON-1751 Storm Profiler dies when consuming null message




---

[jira] [Updated] (METRON-1751) Storm Profiler dies when consuming null message

[Nick Allen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1751:
---
Summary: Storm Profiler dies when consuming null message  (was: Profiler 
dies if we push a null message to the configured profiler input topic )

> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is 
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Async loop died! java.lang.RuntimeException: 
> java.lang.NullPointerException at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at 
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: 
> java.lang.NullPointerException at java.lang.String.(String.java:491) 
> ~[?:1.8.0_181] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
>  ~[stormjar.jar:?] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
>  ~[stormjar.jar:?] at 
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even  if you start up the profiler again and run it 
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at 
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at 
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1751) Profiler dies if we push a null message to the configured profiler input topic

[Mohan (JIRA)]

Mohan created METRON-1751:
-

 Summary: Profiler dies if we push a null message to the configured 
profiler input topic 
 Key: METRON-1751
 URL: https://issues.apache.org/jira/browse/METRON-1751
 Project: Metron
  Issue Type: Bug
Reporter: Mohan


When You publish a null message to the profiler input kafka topic which is 
'indexing' in my case I see the below exception messages on the worker log
{code:java}
2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7] [ERROR] 
Async loop died! java.lang.RuntimeException: java.lang.NullPointerException at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at 
java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: 
java.lang.NullPointerException at java.lang.String.(String.java:491) 
~[?:1.8.0_181] at 
org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
 ~[stormjar.jar:?] at 
org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
 ~[stormjar.jar:?] at 
org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
{code}

also the profiler dies , even  if you start up the profiler again and run it 
through the same topic, it does always die.

{code:java}
2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7] [ERROR] 
Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at 
org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763) 
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at 
org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
 [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494) 
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1724) Date/time validation missing in PCAP query

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593619#comment-16593619
 ] 

ASF GitHub Bot commented on METRON-1724:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1172


> Date/time validation missing in PCAP query
> --
>
> Key: METRON-1724
> URL: https://issues.apache.org/jira/browse/METRON-1724
> Project: Metron
>  Issue Type: Bug
>Reporter: Tibor Meller
>Priority: Major
>
> Validation formula should be the following: 
>  From < To < current date/time
>  
> Validation messages:
> Selected date range is invalid. The "To" date must be later than the "From" 
> date and the "To" date cannot be in the future.
> Source IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Source port is invalid. Port number must be within the range of 0-65535.
> Destination IP address format is invalid. Use valid v4IP format, for example, 
> [192.168.0.1|http://192.168.0.1/].
> Destination port is invalid. Port number must be within the range of 0-65535.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1172: METRON-1724: Date/time validation missing in PCAP...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1172


---