i am +1 on it then 26.01.2018, 15:56, "Michael Miklavcic" <michael.miklav...@gmail.com>: > Just checked on the length issue - we should be good - > https://github.com/elastic/elasticsearch/issues/8079 > > On Fri, Jan 26, 2018 at 3:37 PM, James Sirota <jsir...@apache.org> wrote: > >> Seems reasonable to me. The only thing is that it may make the index >> names too long. Not sure if that matters to ES or not >> >> 26.01.2018, 15:32, "Simon Elliston Ball" <si...@simonellistonball.com>: >> > +1 on this. The idea of a default broad matching template should also >> include an order entry to avoid conflicts with more specific templates, and >> we should then document the need for a higher order value in all per-source >> index templates. >> > >> > In terms of production migration, I think we may want to provide some >> detailed documentation in the upgrade guide on this, because there will be >> people with a lot of existing indices that will be difficult to handle. We >> may also need some tooling, but I expect docs would do the job. What do >> people think about migration? >> > >> > Simon >> > >> >> One other benefit of this revised approach - we can more effectively >> use >> >> index template patterns to specify our base set of Metron property >> types. >> >> Call me crazy, but I think we should be able to do something like: >> >> >> >> <metron_default_template> >> >> >> >> { >> >> *"template": "metron_*",* >> >> "mappings": { >> >> "metron_doc": { >> >> "dynamic_templates": [ >> >> { >> >> "geo_location_point": { >> >> "match": "enrichments:geo:*:location_point", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "geo_point" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_country": { >> >> "match": "enrichments:geo:*:country", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "keyword" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_city": { >> >> "match": "enrichments:geo:*:city", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "keyword" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_location_id": { >> >> "match": "enrichments:geo:*:locID", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "keyword" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_dma_code": { >> >> "match": "enrichments:geo:*:dmaCode", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "keyword" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_postal_code": { >> >> "match": "enrichments:geo:*:postalCode", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "keyword" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_latitude": { >> >> "match": "enrichments:geo:*:latitude", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "float" >> >> } >> >> } >> >> }, >> >> { >> >> "geo_longitude": { >> >> "match": "enrichments:geo:*:longitude", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "float" >> >> } >> >> } >> >> }, >> >> { >> >> "timestamps": { >> >> "match": "*:ts", >> >> "match_mapping_type": "*", >> >> "mapping": { >> >> "type": "date", >> >> "format": "epoch_millis" >> >> } >> >> } >> >> }, >> >> { >> >> "threat_triage_score": { >> >> "mapping": { >> >> "type": "float" >> >> }, >> >> "match": "threat:triage:*score", >> >> "match_mapping_type": "*" >> >> } >> >> }, >> >> { >> >> "threat_triage_reason": { >> >> "mapping": { >> >> "type": "text", >> >> "fielddata": "true" >> >> }, >> >> "match": "threat:triage:rules:*:reason", >> >> "match_mapping_type": "*" >> >> } >> >> }, >> >> { >> >> "threat_triage_name": { >> >> "mapping": { >> >> "type": "text", >> >> "fielddata": "true" >> >> }, >> >> "match": "threat:triage:rules:*:name", >> >> "match_mapping_type": "*" >> >> } >> >> } >> >> >> >> ]}} >> >> >> >> That means that for every new sensor we bring on board we can skip >> >> adding that boiler plate mapping config to every new template. >> >> >> >> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic < >> >> michael.miklav...@gmail.com> wrote: >> >> >> >>> I hear you Ali. I think this type of change would actually ease issues >> >>> with downtime because it offers an easy path to migrating existing >> indices. >> >>> I'd have to review the specifics in the ES docs again, but I believe >> you >> >>> could duplicate the old indexes and migrate them to "metron_" in >> advance of >> >>> the upgrade, and then consume new data to the new index pattern/name >> after >> >>> the upgrade. That should be pretty seamless, I think. I guess it >> depends on >> >>> how you're using ES. >> >>> >> >>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazem...@gmail.com> >> >>> wrote: >> >>> >> >>>> Hi All, >> >>>> >> >>>> I just wanted to say it would be great if we can be careful with >> these >> >>>> type >> >>>> of changes. From the development point of view, it is just a few >> lines of >> >>>> code which can provide multiple advantages, but for live large-scale >> >>>> Metron >> >>>> platforms, some of these changes might be really expensive to >> address with >> >>>> zero-downtime. >> >>>> >> >>>> Cheers, >> >>>> Ali >> >>>> >> >>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler < >> ottobackwa...@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> +1 >> >>>>> >> >>>>> On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org) >> wrote: >> >>>>> >> >>>>> +1 to a standard prefix for all Metron indices. I've had the same >> >>>> thought >> >>>>> myself and you laid out the advantages well. >> >>>>> >> >>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com> >> >>>> wrote: >> >>>>>> I agree with having a metron_ prefix for ES indexes, and the >> timing. >> >>>>>> >> >>>>>> Jon >> >>>>>> >> >>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic < >> >>>>>> michael.miklav...@gmail.com> wrote: >> >>>>>> >> >>>>>>> With the completion of https://github.com/apache/metron/pull/840 >> >>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the >> makings >> >>>> for >> >>>>> a >> >>>>>>> major release rev of Metron in the upcoming release (currently >> >>>> slotted >> >>>>> to >> >>>>>>> 0.4.3, I believe). Since there are non-backwards compatible >> changes >> >>>>>>> pertaining to ES indexing, it seems like a good opportunity to >> >>>> revisit >> >>>>>> our >> >>>>>>> index naming standards. >> >>>>>>> >> >>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes. >> >>>> There >> >>>>>> are >> >>>>>>> numerous reasons for doing so >> >>>>>>> >> >>>>>>> - removes the likelihood of index name collisions when we perform >> >>>>>>> operations on index wildcard names, e.g. "enrichment_*, >> indexing_*, >> >>>>>>> etc.". >> >>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES >> >>>>>>> environment for relatively low engineering cost. >> >>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to >> >>>>>> create a >> >>>>>>> special index pattern in order to accommodate multi-index pattern >> >>>>>>> matching >> >>>>>>> across all metron-specific indexes. Using metron_* would be much >> >>>>>> simpler >> >>>>>>> and less prone to error. >> >>>>>>> - easier for customers to debug and identify Metron-specific >> indexes >> >>>>>> and >> >>>>>>> associated data >> >>>>>>> >> >>>>>>> The reason for making these changes now is that we already have >> >>>>> breaking >> >>>>>>> changes with ES. Leveraging existing indexed data rather than >> >>>> deleting >> >>>>>>> indexes and starting from scractch already requires a >> >>>>>> re-indexing/migration >> >>>>>>> step, so there is no additional effort on the part of users if >> they >> >>>>>> choose >> >>>>>>> to attempt a migration. It further makes sense with our current >> work >> >>>>>>> towards upgrading Solr. >> >>>>>>> >> >>>>>>> We already have a battery of integration and manual tests after >> the >> >>>> ES >> >>>>>>> upgrade work that can be leveraged to validate the changes. >> >>>>>>> >> >>>>>>> Mike Miklavcic >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Jon >> >>>> >> >>>> -- >> >>>> A.Nazemian >> >> ------------------- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org
------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
