Just checked on the length issue - we should be good -
https://github.com/elastic/elasticsearch/issues/8079
On Fri, Jan 26, 2018 at 3:37 PM, James Sirota <jsir...@apache.org> wrote:
> Seems reasonable to me. The only thing is that it may make the index
> names too long. Not sure if that matters to ES or not
>
> 26.01.2018, 15:32, "Simon Elliston Ball" <si...@simonellistonball.com>:
> > +1 on this. The idea of a default broad matching template should also
> include an order entry to avoid conflicts with more specific templates, and
> we should then document the need for a higher order value in all per-source
> index templates.
> >
> > In terms of production migration, I think we may want to provide some
> detailed documentation in the upgrade guide on this, because there will be
> people with a lot of existing indices that will be difficult to handle. We
> may also need some tooling, but I expect docs would do the job. What do
> people think about migration?
> >
> > Simon
> >
> >> One other benefit of this revised approach - we can more effectively
> use
> >> index template patterns to specify our base set of Metron property
> types.
> >> Call me crazy, but I think we should be able to do something like:
> >>
> >> <metron_default_template>
> >>
> >> {
> >> *"template": "metron_*",*
> >> "mappings": {
> >> "metron_doc": {
> >> "dynamic_templates": [
> >> {
> >> "geo_location_point": {
> >> "match": "enrichments:geo:*:location_point",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "geo_point"
> >> }
> >> }
> >> },
> >> {
> >> "geo_country": {
> >> "match": "enrichments:geo:*:country",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "keyword"
> >> }
> >> }
> >> },
> >> {
> >> "geo_city": {
> >> "match": "enrichments:geo:*:city",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "keyword"
> >> }
> >> }
> >> },
> >> {
> >> "geo_location_id": {
> >> "match": "enrichments:geo:*:locID",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "keyword"
> >> }
> >> }
> >> },
> >> {
> >> "geo_dma_code": {
> >> "match": "enrichments:geo:*:dmaCode",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "keyword"
> >> }
> >> }
> >> },
> >> {
> >> "geo_postal_code": {
> >> "match": "enrichments:geo:*:postalCode",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "keyword"
> >> }
> >> }
> >> },
> >> {
> >> "geo_latitude": {
> >> "match": "enrichments:geo:*:latitude",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "float"
> >> }
> >> }
> >> },
> >> {
> >> "geo_longitude": {
> >> "match": "enrichments:geo:*:longitude",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "float"
> >> }
> >> }
> >> },
> >> {
> >> "timestamps": {
> >> "match": "*:ts",
> >> "match_mapping_type": "*",
> >> "mapping": {
> >> "type": "date",
> >> "format": "epoch_millis"
> >> }
> >> }
> >> },
> >> {
> >> "threat_triage_score": {
> >> "mapping": {
> >> "type": "float"
> >> },
> >> "match": "threat:triage:*score",
> >> "match_mapping_type": "*"
> >> }
> >> },
> >> {
> >> "threat_triage_reason": {
> >> "mapping": {
> >> "type": "text",
> >> "fielddata": "true"
> >> },
> >> "match": "threat:triage:rules:*:reason",
> >> "match_mapping_type": "*"
> >> }
> >> },
> >> {
> >> "threat_triage_name": {
> >> "mapping": {
> >> "type": "text",
> >> "fielddata": "true"
> >> },
> >> "match": "threat:triage:rules:*:name",
> >> "match_mapping_type": "*"
> >> }
> >> }
> >>
> >> ]}}
> >>
> >> That means that for every new sensor we bring on board we can skip
> >> adding that boiler plate mapping config to every new template.
> >>
> >> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >>
> >>> I hear you Ali. I think this type of change would actually ease issues
> >>> with downtime because it offers an easy path to migrating existing
> indices.
> >>> I'd have to review the specifics in the ES docs again, but I believe
> you
> >>> could duplicate the old indexes and migrate them to "metron_" in
> advance of
> >>> the upgrade, and then consume new data to the new index pattern/name
> after
> >>> the upgrade. That should be pretty seamless, I think. I guess it
> depends on
> >>> how you're using ES.
> >>>
> >>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazem...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> I just wanted to say it would be great if we can be careful with
> these
> >>>> type
> >>>> of changes. From the development point of view, it is just a few
> lines of
> >>>> code which can provide multiple advantages, but for live large-scale
> >>>> Metron
> >>>> platforms, some of these changes might be really expensive to
> address with
> >>>> zero-downtime.
> >>>>
> >>>> Cheers,
> >>>> Ali
> >>>>
> >>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org)
> wrote:
> >>>>>
> >>>>> +1 to a standard prefix for all Metron indices. I've had the same
> >>>> thought
> >>>>> myself and you laid out the advantages well.
> >>>>>
> >>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com>
> >>>> wrote:
> >>>>>> I agree with having a metron_ prefix for ES indexes, and the
> timing.
> >>>>>>
> >>>>>> Jon
> >>>>>>
> >>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
> >>>>>> michael.miklav...@gmail.com> wrote:
> >>>>>>
> >>>>>>> With the completion of https://github.com/apache/metron/pull/840
> >>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the
> makings
> >>>> for
> >>>>> a
> >>>>>>> major release rev of Metron in the upcoming release (currently
> >>>> slotted
> >>>>> to
> >>>>>>> 0.4.3, I believe). Since there are non-backwards compatible
> changes
> >>>>>>> pertaining to ES indexing, it seems like a good opportunity to
> >>>> revisit
> >>>>>> our
> >>>>>>> index naming standards.
> >>>>>>>
> >>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes.
> >>>> There
> >>>>>> are
> >>>>>>> numerous reasons for doing so
> >>>>>>>
> >>>>>>> - removes the likelihood of index name collisions when we perform
> >>>>>>> operations on index wildcard names, e.g. "enrichment_*,
> indexing_*,
> >>>>>>> etc.".
> >>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES
> >>>>>>> environment for relatively low engineering cost.
> >>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to
> >>>>>> create a
> >>>>>>> special index pattern in order to accommodate multi-index pattern
> >>>>>>> matching
> >>>>>>> across all metron-specific indexes. Using metron_* would be much
> >>>>>> simpler
> >>>>>>> and less prone to error.
> >>>>>>> - easier for customers to debug and identify Metron-specific
> indexes
> >>>>>> and
> >>>>>>> associated data
> >>>>>>>
> >>>>>>> The reason for making these changes now is that we already have
> >>>>> breaking
> >>>>>>> changes with ES. Leveraging existing indexed data rather than
> >>>> deleting
> >>>>>>> indexes and starting from scractch already requires a
> >>>>>> re-indexing/migration
> >>>>>>> step, so there is no additional effort on the part of users if
> they
> >>>>>> choose
> >>>>>>> to attempt a migration. It further makes sense with our current
> work
> >>>>>>> towards upgrading Solr.
> >>>>>>>
> >>>>>>> We already have a battery of integration and manual tests after
> the
> >>>> ES
> >>>>>>> upgrade work that can be leveraged to validate the changes.
> >>>>>>>
> >>>>>>> Mike Miklavcic
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jon
> >>>>
> >>>> --
> >>>> A.Nazemian
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
