[
https://issues.apache.org/jira/browse/METRON-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283507#comment-16283507
]
ASF GitHub Bot commented on METRON-1341:
----------------------------------------
Github user simonellistonball commented on the issue:
https://github.com/apache/metron/pull/861
A typical case might be something like the CEF parser. You could
potentially kick out a lot of fields you really don't care about, which at
scale can produce huge amounts of ES and HDFS storage (in addition to the
original_string representation. The goal for this is to focus on just
outputting fields which match active use cases in the rest of the flow to
control data storage costs and data clarity.
This also allows you to map an explicit data model and provide some
governability to the data model.
> Projection FieldTransformation
> ------------------------------
>
> Key: METRON-1341
> URL: https://issues.apache.org/jira/browse/METRON-1341
> Project: Metron
> Issue Type: Improvement
> Affects Versions: 0.4.2
> Reporter: Simon Elliston Ball
> Assignee: Simon Elliston Ball
> Labels: newbie
>
> It would be useful to have a projection transformation for Parsers which
> could use configured to limit the fields output from the parser.
> The configuration would look like this:
> {code:java}
> {
> "fieldTransformations": [
> {
> "transformation": "STELLAR",
> "config": [
> "ipSrc = TRIM(raw_ip_src)"
> "ip_src_addr := ipSrc"
> ]
> },
> {
> "transformation": "SELECT",
> "output" : [ "ip_src_addr", "ip_dst_addr", "message"]
> }
> ]
> }
> {code}
> This would lead to only the fields in the output definition of the SELECT
> transformation being put into the outbound message.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
