[
https://issues.apache.org/jira/browse/METRON-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412014#comment-16412014
]
ASF GitHub Bot commented on METRON-1501:
----------------------------------------
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/972#discussion_r176855080
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -45,6 +45,33 @@ There are two general types types of parsers:
* `ERROR` : Throw an error when a multidimensional map is encountered
* `jsonpQuery` : A [JSON Path](#json_path) query string. If present,
the result of the JSON Path query should be a list of messages. This is useful
if you have a JSON document which contains a list or array of messages embedded
in it, and you do not have another means of splitting the message.
* A field called `timestamp` is expected to exist and, if it does not,
then current time is inserted.
+
+## Parser Error Routing
+
--- End diff --
It feels like it may be confusing. The title is `Parser Error Routing`,
but the description is `invalid`. Then we talk about `invalid` again, but end
with `error` messages.
We may need to be more clear about errors v. invalids, and how they both
can be routed to the error queue. Or not use the word invalid but instead us a
more specific error, like saying there are parser errors and validation errors.
> Parser messages that fail to validate are dropped silently
> ----------------------------------------------------------
>
> Key: METRON-1501
> URL: https://issues.apache.org/jira/browse/METRON-1501
> Project: Metron
> Issue Type: Bug
> Reporter: Casey Stella
> Priority: Major
>
> Currently we have two concepts of dealing with messages that are not going to
> be passed through:
> * Messages that fail global validation
> * Messages that fail the parser's validation function
> * Messages that are filtered out
>
> Currently in the case of messages that fail global validation, we send them
> to the error queue.
> In the case of messages that fail the parser's validation function, we drop
> them silently.
> In the case of messages that are filtered out, we drop them silently as well.
>
> Given the use-cases behind filtering (e.g. a kafka topic with multiple types
> of messages where we want to pass along only one type), it makes sense to
> drop them silently. However, in the case of the messages that fail the
> parser's validation function, we should like to know that and treat those
> similarly to those messages that fail global validation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
