DISCUSS: Quick change to parser config

[Simon Elliston Ball]

I’m looking at the way parser config works, and transformation of field from 
their native names in, for example the ASA or CEF parsers, into a standard data 
model. 

At the moment I would do something like this: 

assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might 
have:

{
  "fieldTransformations": [
    {
      "transformation": "STELLAR",
      "output": ["ip_src_addr", "ip_dst_addr", "message"],
      "config": {
        "ip_src_addr": "ipSrc",
        "ip_dest_addr": "ipDst"
      }
    }
  ]
}

which leave me with the field set: 
[ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr]

unless I go with:-

{
  "fieldTransformations": [
    {
      "transformation": "STELLAR",
      "output": ["ip_src_addr", "ip_dst_addr", "message"],
      "config": {
        "ip_src_addr": "ipSrc",
        "ip_dest_addr": "ipDst",
        "pointlessExtraStuff": null,
        "ipSrc": null,
        "ipDst": null
      }
    }
  ]
}

which seems a little over verbose. 

Do you think it would be valuable to add a switch of some sort on the 
transformation to make it “complete”, i.e. to only preserve fields which are 
explicitly set. 

To my mind, this breaks a principal of mutability, but gives us much much 
cleaner mapping of data. 

I would propose something like:

{
  "fieldTransformations": [
    {
      "transformation": "STELLAR",
      "complete": true,
      "output": ["ip_src_addr", "ip_dst_addr", "message"],
      "config": {
        "ip_src_addr": "ipSrc",
        "ip_dest_addr": "ipDst"
      }
    }
  ]
}

which would give me the set ["ip_src_addr", "ip_dst_addr", "message”] 
effectively making the nulling in my previous example implicit. 

Thoughts? 

Also, in the second scenario, if ‘output' were to be empty would we assume that 
the output field set should be ["ip_src_addr", “ip_dst_addr”]? 

Simon