Hi Justin, I am adding the nested “alert" field requirement to the product documentation, and I had a question:
* If the user opts to use the default configuration, will they still need to add the nested “alert” field? When I’m finished with the documentation, I’ll send you a link to make sure what I’ve documented is correct. Thanks, Rita Rita McKissick ! Sr. Technical Writer [email protected] (mobile) 831-234-3676 On 9/29/17, 6:59 AM, "Justin Leet" <[email protected]> wrote: >As part of building a backend for meta-alerts ( >https://github.com/apache/metron/pull/734), there's an additional >requirement for the Elasticsearch templates for new sensors. Although >seemingly minor, this should be called out explicitly because of the wider >implications of leaving it out of ANY sensor. Specifically, this can >result in the UI and other queries not returning results, because >Elasticsearch throws an error. > >A nested "alert" field must be added in the form: > >"alert": { > "type": "nested" >}, > >This results from Elasticsearch 2.x requiring the type of searches that >meta alerts wrap to have the fields exist in all indices or the query fails. > >In Elasticsearch 5.x, there is a per query parameter that can be set to >avoid this: see >https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields >. > >The obvious short-term thing that needs to happen with this is improved >documentation. A ticket for documenting this (with some more specific >details and what the error looks like is at >https://issues.apache.org/jira/browse/METRON-1220). Where should that >documentation live? It seems like our documentation in general around this >type of stuff is a little lacking. Right now, I'm putting a prelim version >into the metron-indexing README, but either now or as a followon we should >have a more robust version that lays requirements for things like >templating. > >There are a couple options I see to address this more robustly. >1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A >beginning basis for this change exists in >https://github.com/apache/metron/pull/619. More works needs to happen >there to finalize it >2) Add in ZK hooks for when a new sensor is added. The DAOs could receive >word that a new sensor has been added in ZK and then build and submit the >modified template itself. This (or a variant) is probably something that >should happen anyway, in order to be more consistent with the other pieces >that monitor and act on ZK updates. >3) There may be some mitigating that can be done here, e.g. if a query >fails with the relevant error, rerun a different variant that may not hit >the meta alerts, but doesn't fail as extravagantly. > >Is there a preference on either where the new documentation lives? And is >there a preference on how we address this going forward? > >Justin
