Before we construct a super generic solution, can we get an analysis of all
the places in the UI where we're hard-coding fields? It seems like pulling
the field from the global config is the strategy that we've gone with that
could be expanded upon in https://github.com/apache/metron/pull/1010
(though didn't quite get the semantic correct as it required
https://github.com/apache/metron/pull/1038). Is there a reason why we
wouldn't create a PR to refer to all of the hard-coded fields in the same
way?
I guess my perspective is that this seems like a problem contained to the
UI accessing a small number of hard-coded fields and expansion of those
fields seem pretty contained. If so, I'd suggest we continue with the
pattern we already have. If you want to expand it, you might consider
taking advantage of the fact that the global config can use maps and doing
something like:
{
...
"fieldNameTransformations" : {
"source:type" : "source.type",
"threat:triage:reason" : "threat.triage.reason"
}
}
Whereby in the UI when accessing a hard-coded field, it will look up the
field in the fieldNameTransformations map from global config. If it exists
in the map, then it'll use the translated field. If it does not, then it
will use the field it passed in (e.g. source:type). That would allow us to
add new translations easily, but it may be overkill if we're talking about
3 fields.
Another question, is there an easy way to bulk change field names in ES
across many indices? Could we normalize on .'s and not do this translation
at all? I think in order to do that, we'd need instructions on how to
transition at least selected fields (i.e. those hard coded fields in the
UI).
Casey
On Mon, Jun 4, 2018 at 4:55 PM Ryan Merriman <merrim...@gmail.com> wrote:
> We've been dealing with a reoccurring challenge in Metron. It is common
> for various fields to contain '.' characters for the purpose of making them
> more readable, namespacing, etc. At one point we only supported
> Elasticsearch 2.3 which did not allow dots and forced us to use ':'
> instead. This limitation does not exist in later versions of Elasticsearch
> or Solr.
>
> Now we're in a situation where we need to allow a user to use either one
> because they may still be using ES 2.3 or have data with ':' characters in
> field names. We've attempted to make this configurable in a couple
> different PRs:
>
> https://github.com/apache/metron/pull/1022
> https://github.com/apache/metron/pull/1010
> https://github.com/apache/metron/pull/1038
>
> The approaches taken in these are not consistent and fall short in
> different ways. The first (METRON-1569 Allow user to change field name
> conversion when indexing) only applies to indexing and not querying. The
> others only apply to a single field which does not scale well. Now we have
> an issue with another field in
> https://issues.apache.org/jira/browse/METRON-1600. Rather than continuing
> with a patchwork of different fixes I want to attempt to design a
> system-wide solution.
>
> My first thought is to expand https://github.com/apache/metron/pull/1022
> to
> apply globally. However this is not trivial and would require significant
> changes. It would also make https://github.com/apache/metron/pull/1010
> obsolete and we might end up having to revert all of it.
>
> Does anyone have any ideas or opinions? I am still researching solutions
> but would love some guidance from the community.
>
