[
https://issues.apache.org/jira/browse/METRON-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623327#comment-15623327
]
ASF GitHub Bot commented on METRON-249:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/333
@ottobackwards Thanks so much for the review.
Yeah, it's a general question about stellar functions about when to error
versus when to tolerate. We have traditionally tried to be as permissive as
possible about errors that are associated with missing data. I think you can
make an argument for malformed URLs to be a proper error, but it has
historically not been mostly because it was considered a semantic error. The
original concern was you might want to transform fields which were
inconsistently populated or populated with default values some of the time. In
that case, if the field is malformed then the URL that it produced should be
`null`. You still have the original field and can make decisions downstream in
enrichment or threat triage.
Contrasting that would be syntactic or systemic errors, like passing in a
number when you intended a string. These would indicate a high likelihood of
misconfiguration, so should be errors. This was the original thinking behind
when stellar functions error and when they tolerate variable input.
I think it is high time to provide an `EXCEPTION` function so you can
choose to throw an exception under certain circumstances if you want a more
conservative interpretation of error.
> Field Transformation functions fail to handle invalid user inputs
> ------------------------------------------------------------------
>
> Key: METRON-249
> URL: https://issues.apache.org/jira/browse/METRON-249
> Project: Metron
> Issue Type: Bug
> Reporter: Neha Sinha
> Priority: Minor
> Labels: platform
> Fix For: 0.2.1BETA
>
> Attachments: LogException.rtf
>
>
> Hi,
> The field transformation functions fail to handle invalid user input .On
> providing invalid inputs the parser throws exceptions and fails to create the
> required indices in elasticsearch.
> ==========================
> Steps to Reproduce
> ==========================
> Edit the squid.json file and provide the following definition to it:-(Note-we
> are giving an invalid input :-123 to the URL_TO_HOST function)
> -----------------------------------------------------------------------------------------------
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "squid",
> "parserConfig": {
> "grokPath": "/patterns/squid",
> "patternLabel": "SQUID_DELIMITED",
> "timestampField": "timestamp"
> },
> "fieldTransformations" : [
> {
> "transformation" : "MTL"
> ,"output" : [ "full_hostname", "domain_without_subdomains" ]
> ,"config" : {
> "full_hostname" : “URL_TO_HOST(123)"
> ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
> }
> }
> ]
> }
> ----------------------------------------------------------------------------------------------------
> Replay Squid events/logs and monitor the logs in storm for squid topology.
> Attached exception log would be seen and no indexes would be created
> respective to the logs.
> Expected Behaviour :-
> 1.The error should be more clean.
> 2.Since we cannot validate the inputs the invalid inputs should be ignored
> and the indices should get created anyway based on the Grok parser output
> Regards,
> Neha
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
