Re: [DISCUSS] Field conversions

2018-06-06 Thread Michael Miklavcic
Yeah Otto, pre-0.5.0 (0.4.2) would be ES 2.3 if users were not using master. ES upgrade is a big piece of this Apache release. Last release was 0.4.2 on Fri Dec 22 2017. I'm +1 on the idea of an example referencing the ES docs and keeping this as simple as possible. *

Re: [DISCUSS] Field conversions

2018-06-05 Thread Otto Fowler
Aren’t people who are on an old version of ES everyone pre 0.5.0? Like all the metron users? On June 5, 2018 at 12:31:30, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Yes, anything using elastic would need the field names changed. That said, people who are on such an old version

Re: [DISCUSS] Field conversions

2018-06-05 Thread Simon Elliston Ball
Yes, anything using elastic would need the field names changed. That said, people who are on such an old version (eol) will need to not the bullet with ES compatibility as some point. Simon > On 5 Jun 2018, at 17:17, Otto Fowler wrote: > > Are there consequences with Kibana as well?

Re: [DISCUSS] Field conversions

2018-06-05 Thread Otto Fowler
Are there consequences with Kibana as well? queries, visualizations, templates they may have? On June 5, 2018 at 12:03:44, Nick Allen (n...@nickallen.org) wrote: I just don't know if telling users to do a bulk upgrade of their indices is sufficient enough of an upgrade path. I would expect

Re: [DISCUSS] Field conversions

2018-06-05 Thread Nick Allen
I just don't know if telling users to do a bulk upgrade of their indices is sufficient enough of an upgrade path. I would expect some to have downstream processes dependent on those field names, which would also need to be updated. Although, we could tell users to do any field name conversions

Re: [DISCUSS] Field conversions

2018-06-05 Thread Casey Stella
Agreed, we should definitely have a clear picture about how to do that, maybe even a worked example in the use-cases that we can reference. I'm just saying we don't need to migrate ES docs into Metron, but rather reference them as much as we possibly can. On Tue, Jun 5, 2018 at 11:38 AM Otto

Re: [DISCUSS] Field conversions

2018-06-05 Thread Otto Fowler
It is still our user list and dev list that will have the burden of talking folks through that. On June 5, 2018 at 09:58:32, Casey Stella (ceste...@gmail.com) wrote: To be clear, I'm not even suggesting that we create any tooling here. I'd say just a reference to the ES docs and a call-out in

Re: [DISCUSS] Field conversions

2018-06-05 Thread Nick Allen
I am in favor of removing the `FieldNameConverter` abstraction as an end state. Although, I don't agree with Simon that we could have just done that directly without providing a backwards compatible solution as was done in #1022. There are too many touch points that rely on that conversion and

Re: [DISCUSS] Field conversions

2018-06-05 Thread Simon Elliston Ball
+1 to that. It's a simple problem to solve if you have it, and with a little docs help I imagine we'll be fine. On 5 June 2018 at 06:58, Casey Stella wrote: > To be clear, I'm not even suggesting that we create any tooling here. I'd > say just a reference to the ES docs and a call-out in

Re: [DISCUSS] Field conversions

2018-06-05 Thread Casey Stella
To be clear, I'm not even suggesting that we create any tooling here. I'd say just a reference to the ES docs and a call-out in Upgrading.md would suffice as long as we have some strong reason to believe it'll work. As far as I'm concerned, the sooner we're out of the business of transforming

Re: [DISCUSS] Field conversions

2018-06-05 Thread Justin Leet
ES does have some docs around how this gets handled in upgrades: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html Might be worth taking a look to see what conflicts we'd have going from 2.x to 5.x and figuring out where to go from there. On Tue, Jun 5, 2018 at 9:46

Re: [DISCUSS] Field conversions

2018-06-05 Thread Simon Elliston Ball
I guess in principal you could use https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-change-name to reindex with the new fields. It wouldn't be hard to script up a bit of python to help users out with that, or of course to leave that as an exercise to

Re: [DISCUSS] Field conversions

2018-06-05 Thread Casey Stella
+1 to that, Simon. Do we have a sense of if there are utilities provided by ES to do this kind of migration transformation easily? On Tue, Jun 5, 2018 at 9:37 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > I would definitely agree that the transformation should be removed. We

Re: [DISCUSS] Field conversions

2018-06-05 Thread Simon Elliston Ball
I would definitely agree that the transformation should be removed. We have now however added a complex generic solution in the backend, which is going to be noop for most people. This was done I believe for the sake of backward compatibility. I would argue however, that there is no need to

Re: [DISCUSS] Field conversions

2018-06-05 Thread Ryan Merriman
I agree completely. I will leave this thread open for a day or two to give others a chance to weigh in. If no one opposes, I will creates Jiras for removing field transformations and transforming existing data. On Tue, Jun 5, 2018 at 8:21 AM, Casey Stella wrote: > Well, on write it is a

Re: [DISCUSS] Field conversions

2018-06-05 Thread Casey Stella
Well, on write it is a transformation, on read it's a translation. This is to say that you're providing a mapping on read to translate field names given the index you're using. The other approach that I was considering last night is a field transformation REST call which translates field names

Re: [DISCUSS] Field conversions

2018-06-05 Thread Ryan Merriman
Having 2 different patterns for configuring field name transformations on read vs write is confusing to me. I agree with both of you that normalizing on '.' and not having to do the translation at all would be ideal. Like you both suggested, we would need some utility or script to convert

Re: [DISCUSS] Field conversions

2018-06-04 Thread Laurens Vets
ES 2.x support officially ended 4 months ago (https://www.elastic.co/support/eol), so why still support ':' at all? :) Additionally, 2.x isn't even supported at all on the last 2 Ubuntu LTS releases (16.04 & 18.05). Therefor, move everything to use '.' and provide a conversion/upgrade script

Re: [DISCUSS] Field conversions

2018-06-04 Thread Casey Stella
Before we construct a super generic solution, can we get an analysis of all the places in the UI where we're hard-coding fields? It seems like pulling the field from the global config is the strategy that we've gone with that could be expanded upon in https://github.com/apache/metron/pull/1010

[DISCUSS] Field conversions

2018-06-04 Thread Ryan Merriman
We've been dealing with a reoccurring challenge in Metron. It is common for various fields to contain '.' characters for the purpose of making them more readable, namespacing, etc. At one point we only supported Elasticsearch 2.3 which did not allow dots and forced us to use ':' instead. This