Seems reasonable to me. The only thing is that it may make the index names too
long. Not sure if that matters to ES or not
26.01.2018, 15:32, "Simon Elliston Ball" <si...@simonellistonball.com>:
> +1 on this. The idea of a default broad matching template should also include
> an order entry to avoid conflicts with more specific templates, and we should
> then document the need for a higher order value in all per-source index
> templates.
>
> In terms of production migration, I think we may want to provide some
> detailed documentation in the upgrade guide on this, because there will be
> people with a lot of existing indices that will be difficult to handle. We
> may also need some tooling, but I expect docs would do the job. What do
> people think about migration?
>
> Simon
>
>> One other benefit of this revised approach - we can more effectively use
>> index template patterns to specify our base set of Metron property types.
>> Call me crazy, but I think we should be able to do something like:
>>
>> <metron_default_template>
>>
>> {
>> *"template": "metron_*",*
>> "mappings": {
>> "metron_doc": {
>> "dynamic_templates": [
>> {
>> "geo_location_point": {
>> "match": "enrichments:geo:*:location_point",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "geo_point"
>> }
>> }
>> },
>> {
>> "geo_country": {
>> "match": "enrichments:geo:*:country",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "keyword"
>> }
>> }
>> },
>> {
>> "geo_city": {
>> "match": "enrichments:geo:*:city",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "keyword"
>> }
>> }
>> },
>> {
>> "geo_location_id": {
>> "match": "enrichments:geo:*:locID",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "keyword"
>> }
>> }
>> },
>> {
>> "geo_dma_code": {
>> "match": "enrichments:geo:*:dmaCode",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "keyword"
>> }
>> }
>> },
>> {
>> "geo_postal_code": {
>> "match": "enrichments:geo:*:postalCode",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "keyword"
>> }
>> }
>> },
>> {
>> "geo_latitude": {
>> "match": "enrichments:geo:*:latitude",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "float"
>> }
>> }
>> },
>> {
>> "geo_longitude": {
>> "match": "enrichments:geo:*:longitude",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "float"
>> }
>> }
>> },
>> {
>> "timestamps": {
>> "match": "*:ts",
>> "match_mapping_type": "*",
>> "mapping": {
>> "type": "date",
>> "format": "epoch_millis"
>> }
>> }
>> },
>> {
>> "threat_triage_score": {
>> "mapping": {
>> "type": "float"
>> },
>> "match": "threat:triage:*score",
>> "match_mapping_type": "*"
>> }
>> },
>> {
>> "threat_triage_reason": {
>> "mapping": {
>> "type": "text",
>> "fielddata": "true"
>> },
>> "match": "threat:triage:rules:*:reason",
>> "match_mapping_type": "*"
>> }
>> },
>> {
>> "threat_triage_name": {
>> "mapping": {
>> "type": "text",
>> "fielddata": "true"
>> },
>> "match": "threat:triage:rules:*:name",
>> "match_mapping_type": "*"
>> }
>> }
>>
>> ]}}
>>
>> That means that for every new sensor we bring on board we can skip
>> adding that boiler plate mapping config to every new template.
>>
>> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> I hear you Ali. I think this type of change would actually ease issues
>>> with downtime because it offers an easy path to migrating existing indices.
>>> I'd have to review the specifics in the ES docs again, but I believe you
>>> could duplicate the old indexes and migrate them to "metron_" in advance of
>>> the upgrade, and then consume new data to the new index pattern/name after
>>> the upgrade. That should be pretty seamless, I think. I guess it depends on
>>> how you're using ES.
>>>
>>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazem...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I just wanted to say it would be great if we can be careful with these
>>>> type
>>>> of changes. From the development point of view, it is just a few lines of
>>>> code which can provide multiple advantages, but for live large-scale
>>>> Metron
>>>> platforms, some of these changes might be really expensive to address with
>>>> zero-downtime.
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwa...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org) wrote:
>>>>>
>>>>> +1 to a standard prefix for all Metron indices. I've had the same
>>>> thought
>>>>> myself and you laid out the advantages well.
>>>>>
>>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com>
>>>> wrote:
>>>>>> I agree with having a metron_ prefix for ES indexes, and the timing.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> With the completion of https://github.com/apache/metron/pull/840
>>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
>>>> for
>>>>> a
>>>>>>> major release rev of Metron in the upcoming release (currently
>>>> slotted
>>>>> to
>>>>>>> 0.4.3, I believe). Since there are non-backwards compatible changes
>>>>>>> pertaining to ES indexing, it seems like a good opportunity to
>>>> revisit
>>>>>> our
>>>>>>> index naming standards.
>>>>>>>
>>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes.
>>>> There
>>>>>> are
>>>>>>> numerous reasons for doing so
>>>>>>>
>>>>>>> - removes the likelihood of index name collisions when we perform
>>>>>>> operations on index wildcard names, e.g. "enrichment_*, indexing_*,
>>>>>>> etc.".
>>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES
>>>>>>> environment for relatively low engineering cost.
>>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to
>>>>>> create a
>>>>>>> special index pattern in order to accommodate multi-index pattern
>>>>>>> matching
>>>>>>> across all metron-specific indexes. Using metron_* would be much
>>>>>> simpler
>>>>>>> and less prone to error.
>>>>>>> - easier for customers to debug and identify Metron-specific indexes
>>>>>> and
>>>>>>> associated data
>>>>>>>
>>>>>>> The reason for making these changes now is that we already have
>>>>> breaking
>>>>>>> changes with ES. Leveraging existing indexed data rather than
>>>> deleting
>>>>>>> indexes and starting from scractch already requires a
>>>>>> re-indexing/migration
>>>>>>> step, so there is no additional effort on the part of users if they
>>>>>> choose
>>>>>>> to attempt a migration. It further makes sense with our current work
>>>>>>> towards upgrading Solr.
>>>>>>>
>>>>>>> We already have a battery of integration and manual tests after the
>>>> ES
>>>>>>> upgrade work that can be leveraged to validate the changes.
>>>>>>>
>>>>>>> Mike Miklavcic
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jon
>>>>
>>>> --
>>>> A.Nazemian
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
