Having 2 different patterns for configuring field name transformations on read vs write is confusing to me. I agree with both of you that normalizing on '.' and not having to do the translation at all would be ideal. Like you both suggested, we would need some utility or script to convert preexisting data to match this format. There could also be some adjustments a user would need to make in the UI but I feel like we could document around that. Are there any objections to doing it this way?
On Mon, Jun 4, 2018 at 4:30 PM, Laurens Vets <laur...@daemon.be> wrote: > ES 2.x support officially ended 4 months ago ( > https://www.elastic.co/support/eol), so why still support ':' at all? :) > Additionally, 2.x isn't even supported at all on the last 2 Ubuntu LTS > releases (16.04 & 18.05). > > Therefor, move everything to use '.' and provide a conversion/upgrade > script to change '.' to ':'? > > > On 2018-06-04 13:55, Ryan Merriman wrote: > >> We've been dealing with a reoccurring challenge in Metron. It is common >> for various fields to contain '.' characters for the purpose of making >> them >> more readable, namespacing, etc. At one point we only supported >> Elasticsearch 2.3 which did not allow dots and forced us to use ':' >> instead. This limitation does not exist in later versions of >> Elasticsearch >> or Solr. >> >> Now we're in a situation where we need to allow a user to use either one >> because they may still be using ES 2.3 or have data with ':' characters in >> field names. We've attempted to make this configurable in a couple >> different PRs: >> >> https://github.com/apache/metron/pull/1022 >> https://github.com/apache/metron/pull/1010 >> https://github.com/apache/metron/pull/1038 >> >> The approaches taken in these are not consistent and fall short in >> different ways. The first (METRON-1569 Allow user to change field name >> conversion when indexing) only applies to indexing and not querying. The >> others only apply to a single field which does not scale well. Now we >> have >> an issue with another field in >> https://issues.apache.org/jira/browse/METRON-1600. Rather than >> continuing >> with a patchwork of different fixes I want to attempt to design a >> system-wide solution. >> >> My first thought is to expand https://github.com/apache/metron/pull/1022 >> to >> apply globally. However this is not trivial and would require significant >> changes. It would also make https://github.com/apache/metron/pull/1010 >> obsolete and we might end up having to revert all of it. >> >> Does anyone have any ideas or opinions? I am still researching solutions >> but would love some guidance from the community. >> >
