[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424096#comment-16424096
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/972


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417472#comment-16417472
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/972
  
+1 by inspection, thanks for the contribution


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417421#comment-16417421
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/972
  
Alright, I think I addressed the issues @ottobackwards 


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412014#comment-16412014
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/972#discussion_r176855080
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -45,6 +45,33 @@ There are two general types types of parsers:
   * `ERROR` : Throw an error when a multidimensional map is encountered
 * `jsonpQuery` : A [JSON Path](#json_path) query string. If present, 
the result of the JSON Path query should be a list of messages. This is useful 
if you have a JSON document which contains a list or array of messages embedded 
in it, and you do not have another means of splitting the message.
 * A field called `timestamp` is expected to exist and, if it does not, 
then current time is inserted.  
+
+## Parser Error Routing
+
--- End diff --

It feels like it may be confusing.  The title is `Parser Error Routing`, 
but the description is `invalid`.  Then we talk about `invalid` again, but end 
with `error` messages.

We may need to be more clear about errors v. invalids, and how they both 
can be routed to the error queue.  Or not use the word invalid but instead us a 
more specific error, like saying there are parser errors and validation errors.



> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412013#comment-16412013
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/972#discussion_r176854208
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -45,6 +45,33 @@ There are two general types types of parsers:
   * `ERROR` : Throw an error when a multidimensional map is encountered
 * `jsonpQuery` : A [JSON Path](#json_path) query string. If present, 
the result of the JSON Path query should be a list of messages. This is useful 
if you have a JSON document which contains a list or array of messages embedded 
in it, and you do not have another means of splitting the message.
 * A field called `timestamp` is expected to exist and, if it does not, 
then current time is inserted.  
+
+## Parser Error Routing
+
+Currently, we have a few mechanisms for either deferring processing of
+messages or marking messages as invalid.
+
+### Invalid
+
--- End diff --

I would says 'there are two reasons a message will be marked as invalid'
and * Failing global blah
* Failing the parser's..


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412012#comment-16412012
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/972#discussion_r176855220
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -45,6 +45,33 @@ There are two general types types of parsers:
   * `ERROR` : Throw an error when a multidimensional map is encountered
 * `jsonpQuery` : A [JSON Path](#json_path) query string. If present, 
the result of the JSON Path query should be a list of messages. This is useful 
if you have a JSON document which contains a list or array of messages embedded 
in it, and you do not have another means of splitting the message.
 * A field called `timestamp` is expected to exist and, if it does not, 
then current time is inserted.  
+
+## Parser Error Routing
+
+Currently, we have a few mechanisms for either deferring processing of
+messages or marking messages as invalid.
+
+### Invalid
+
+There are two ways for a message to get marked as invalid:
+* Fail [global validation](../metron-common#validation-framework)
+* Fail the parser's validate function (generally that means to not have a 
`timestamp` field or a `original_string` field.
+
+Those messages which are marked as invalid are sent to the error queue
+with an indication that they are invalid in the error message.
+
--- End diff --

Maybe filtering shouldn't be mixed in with errors?


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411929#comment-16411929
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/972
  
Yes, sorry, commit pending for that...


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411928#comment-16411928
 ] 

ASF GitHub Bot commented on METRON-1501:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/972
  
This seems like it should have some documentation to go with it


> Parser messages that fail to validate are dropped silently
> --
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to 
> be passed through:
>  * Messages that fail global validation
>  * Messages that fail the parser's validation function
>  * Messages that are filtered out
>  
> Currently in the case of messages that fail global validation, we send them 
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop 
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>  
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types 
> of messages where we want to pass along only one type), it makes sense to 
> drop them silently.  However, in the case of the messages that fail the 
> parser's validation function, we should like to know that and treat those 
> similarly to those messages that fail global validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1501) Parser messages that fail to validate are dropped silently

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411903#comment-16411903
 ] 

ASF GitHub Bot commented on METRON-1501:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/972

METRON-1501: Parser messages that fail to validate are dropped silently

## Contributor Comments
Currently we have two concepts of dealing with messages that are not going 
to be passed through:

* Messages that fail global validation
* Messages that fail the parser's validation function
* Messages that are filtered out
Currently 
* in the case of messages that fail global validation, we send them to the 
error queue.
* in the case of messages that fail the parser's validation function, we 
drop them silently.
* in the case of messages that are filtered out, we drop them silently as 
well.

Given the use-cases behind filtering (e.g. a kafka topic with multiple 
types of messages where we want to pass along only one type), it makes sense to 
drop them silently.  However, in the case of the messages that fail the 
parser's validation function, we should like to know that and treat those 
similarly to those messages that fail global validation.

In order to test this, create a grok parser without a timestamp configured 
(this will make it fail the grok parser's validation) and pass along a message. 
 Ensure that it makes it to the error queue rather than being dropped silently.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron grok_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/972.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #972


commit dab5d23b4f0c5eda4e8f714f0696adedec2fc633
Author: cstella 
Date:   2018-03-20T20:32:54Z

Fixing the grok test and making MessageParser.validate(any()) == true imply 
that we handle it like an invalid message