I put up a preliminary PR at https://github.com/apache/metron/pull/780. As noted there, this should almost certainly be under a different heading, and possibly a different README, so feel free to chime in on that. Primary goal is to make sure the content makes sense and get adjustments as needed.
On Fri, Sep 29, 2017 at 10:10 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > We can also consider this when thinking about creating parsers with > archetypes that contain ‘default’ > elasticsearch templates. > > > > On September 29, 2017 at 10:00:03, Justin Leet (justinjl...@gmail.com) > wrote: > > As part of building a backend for meta-alerts ( > https://github.com/apache/metron/pull/734), there's an additional > requirement for the Elasticsearch templates for new sensors. Although > seemingly minor, this should be called out explicitly because of the wider > implications of leaving it out of ANY sensor. Specifically, this can > result in the UI and other queries not returning results, because > Elasticsearch throws an error. > > A nested "alert" field must be added in the form: > > "alert": { > "type": "nested" > }, > > This results from Elasticsearch 2.x requiring the type of searches that > meta alerts wrap to have the fields exist in all indices or the query > fails. > > In Elasticsearch 5.x, there is a per query parameter that can be set to > avoid this: see > https://www.elastic.co/guide/en/elasticsearch/reference/ > current/search-request-sort.html#_ignoring_unmapped_fields > . > > The obvious short-term thing that needs to happen with this is improved > documentation. A ticket for documenting this (with some more specific > details and what the error looks like is at > https://issues.apache.org/jira/browse/METRON-1220). Where should that > documentation live? It seems like our documentation in general around this > type of stuff is a little lacking. Right now, I'm putting a prelim version > into the metron-indexing README, but either now or as a followon we should > have a more robust version that lays requirements for things like > templating. > > There are a couple options I see to address this more robustly. > 1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A > beginning basis for this change exists in > https://github.com/apache/metron/pull/619. More works needs to happen > there to finalize it > 2) Add in ZK hooks for when a new sensor is added. The DAOs could receive > word that a new sensor has been added in ZK and then build and submit the > modified template itself. This (or a variant) is probably something that > should happen anyway, in order to be more consistent with the other pieces > that monitor and act on ZK updates. > 3) There may be some mitigating that can be done here, e.g. if a query > fails with the relevant error, rerun a different variant that may not hit > the meta alerts, but doesn't fail as extravagantly. > > Is there a preference on either where the new documentation lives? And is > there a preference on how we address this going forward? > > Justin > >
