Setup ES alerting //Re: Meeting minutes with Nielsen:
hi Ananthanarayanan, There are several ways to set up elasticsearch alerting. 1. alerts with x-pack x-pack is provided by Elastic Corporation. More information is availabe at https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html. x-pack requires license after evaluation period. The overall settings up are like the following. * install x-pack plugin on all the nodes in the cluster * set up channels like email server, slack server in the elasticsearch.yml file * set up alerts with desired query. * when the desired query meet its condition, alerts will be issued via channels. 2. free alternative ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. It's implemented in python. More information available at https://elastalert.readthedocs.io/en/latest/ The overall settings up are like the following. * install python packages elastalert and elasticsearch * set up config.yaml for the connections and authentications * set up the elasticsearch index for ElastAlert * set up alerts * run ElastAlert as a demon service or a python process * when alerts meet their conditions, alerts will be issued via channels. Thanks, William From: William GUO <guo...@outlook.com> on behalf of William Guo <gu...@apache.org> Sent: Friday, August 4, 2017 1:37:47 PM To: Ananthanarayanan Ms; Mara Preotescu Cc: dev@griffin.incubator.apache.org; Kunduru, Abishek Subject: Re: Meeting minutes with Nielsen: hi Ananthanarayanan, For profiling, we have developed some samples based on our measures. Will make samples available in our repo next week. Accuracy, we have test cases for it, but we will also make samples in our repo next week. Thanks, William From: Ananthanarayanan Ms <ananthanarayanan.ms...@nielsen.com> Sent: Friday, August 4, 2017 12:30:22 AM To: Mara Preotescu Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek Subject: Re: Meeting minutes with Nielsen: Hi William/Lionel, Could you please help us to understand on the profiling feature availability, we see the code traces on profiling, could you let us know if we can starting using it from some branch which could be used so that we could leverage griffin on two dimensions (accuracy & profiling). If its not available a tentative date so that we could decide upon the same. Regards, Ananthanarayanan.M.S On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> wrote: Thank you William. Ananth will follow up with a few more questions on the roadmap. Thanks, Mara On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org<mailto:gu...@apache.org>> wrote: Meeting minutes with Nielsen: * Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI. * Griffin provides RESTful API for backend. * Discuss griffin to support multiple source or target tables. * Discuss more supporting file types, such as parquet. * In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information. * Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly. * Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules. * Sample ratio field in config file is optional, in batch mode we don’t need to configure it. * In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script. * At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list * Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search * In griffin, you can’t modify the exist rules or measure at current. Thanks, William From: William GUO <guo...@outlook.com<mailto:guo...@outlook.com>> on behalf of William Guo <gu...@apache.org<mailto:gu...@apache.org>> Sent: Wednesday, August 2, 2017 10:02:15 AM To: Mara Preotescu Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek Subject: Re: Griffin support & roadmap hi mara, Are you join? Thanks, William From: Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> Sent: Monday, July 31, 2017 11:22:00 PM To: William Guo Cc: dev@griffin.incubator.apache.org<mailto
Re: Meeting minutes with Nielsen:
hi Ananthanarayanan, For profiling, we have developed some samples based on our measures. Will make samples available in our repo next week. Accuracy, we have test cases for it, but we will also make samples in our repo next week. Thanks, William From: Ananthanarayanan Ms <ananthanarayanan.ms...@nielsen.com> Sent: Friday, August 4, 2017 12:30:22 AM To: Mara Preotescu Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek Subject: Re: Meeting minutes with Nielsen: Hi William/Lionel, Could you please help us to understand on the profiling feature availability, we see the code traces on profiling, could you let us know if we can starting using it from some branch which could be used so that we could leverage griffin on two dimensions (accuracy & profiling). If its not available a tentative date so that we could decide upon the same. Regards, Ananthanarayanan.M.S On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> wrote: Thank you William. Ananth will follow up with a few more questions on the roadmap. Thanks, Mara On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org<mailto:gu...@apache.org>> wrote: Meeting minutes with Nielsen: * Discuss griffin to support filters for metastore tables or navigation assistance for table selection on UI. * Griffin provides RESTful API for backend. * Discuss griffin to support multiple source or target tables. * Discuss more supporting file types, such as parquet. * In griffin, the partition field is optional, it just helps to provide the specific part of data, it will get all the data of a table without any partition information. * Config json file provides the parameters for griffin measure calculation, you can also submit a spark job with it directly. * Currently, griffin can only reuse measure, not rule. We’ll discuss about this, if we need to support reusing rules. * Sample ratio field in config file is optional, in batch mode we don’t need to configure it. * In griffin, mapping of columns are limited, discuss to support advanced features like joining between tables , or advanced sql script. * At current, the rule parser doesn’t support customized rules, griffin has the plan to support this. //TODO document it and send it to dev list * Griffin doesn’t support metrics alert function, it posts all the metrics to elasticsearch, es supports such feature. //TODO, write a solution for it based on elastic search * In griffin, you can’t modify the exist rules or measure at current. Thanks, William From: William GUO <guo...@outlook.com<mailto:guo...@outlook.com>> on behalf of William Guo <gu...@apache.org<mailto:gu...@apache.org>> Sent: Wednesday, August 2, 2017 10:02:15 AM To: Mara Preotescu Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek Subject: Re: Griffin support & roadmap hi mara, Are you join? Thanks, William From: Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> Sent: Monday, July 31, 2017 11:22:00 PM To: William Guo Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek Subject: Re: Griffin support & roadmap Hi William, Would 10:00 am CST (Beijing) work for you on Wednesday 08/02? Thanks, Mara On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>> wrote: hi Mara, We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together. China day time is fine for me. Thanks, William From: Mara Preotescu <mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>>> Sent: Monday, July 31, 2017 10:54:25 AM To: William Guo Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>> Subject: Re: Griffin support & roadmap Hi William, Either Wednesday or Thursday will work for us. Any better time working for you? What time zone are you in? I am in US ET time, a colleague of mine who I would like to join our discussion is in India, Chennai. Thanks, Mara On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>> wrote: hi Mara, Sure, We could schedule a meeting to discuss background, requirements, status and milestone. We sh
