Hi All, I just wanted to say it would be great if we can be careful with these type of changes. From the development point of view, it is just a few lines of code which can provide multiple advantages, but for live large-scale Metron platforms, some of these changes might be really expensive to address with zero-downtime.
Cheers, Ali On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > +1 > > > On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org) wrote: > > +1 to a standard prefix for all Metron indices. I've had the same thought > myself and you laid out the advantages well. > > > > > > On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com> wrote: > > > I agree with having a metron_ prefix for ES indexes, and the timing. > > > > Jon > > > > On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > With the completion of https://github.com/apache/metron/pull/840 > > > (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for > a > > > major release rev of Metron in the upcoming release (currently slotted > to > > > 0.4.3, I believe). Since there are non-backwards compatible changes > > > pertaining to ES indexing, it seems like a good opportunity to revisit > > our > > > index naming standards. > > > > > > I propose we add a simple prefix "metron_" to all Metron indexes. There > > are > > > numerous reasons for doing so > > > > > > - removes the likelihood of index name collisions when we perform > > > operations on index wildcard names, e.g. "enrichment_*, indexing_*, > > > etc.". > > > - ie, this allows us to be more friendly in a multi-tenant ES > > > environment for relatively low engineering cost. > > > - simplifies the Kibana dashboard a bit. We currently needed to > > create a > > > special index pattern in order to accommodate multi-index pattern > > > matching > > > across all metron-specific indexes. Using metron_* would be much > > simpler > > > and less prone to error. > > > - easier for customers to debug and identify Metron-specific indexes > > and > > > associated data > > > > > > > > > The reason for making these changes now is that we already have > breaking > > > changes with ES. Leveraging existing indexed data rather than deleting > > > indexes and starting from scractch already requires a > > re-indexing/migration > > > step, so there is no additional effort on the part of users if they > > choose > > > to attempt a migration. It further makes sense with our current work > > > towards upgrading Solr. > > > > > > We already have a battery of integration and manual tests after the ES > > > upgrade work that can be leveraged to validate the changes. > > > > > > Mike Miklavcic > > > > > > > > > -- > > > > Jon > > > -- A.Nazemian
