Hi William, Was going through the ES watcher post our call and since we are using ES 2.4.1 as of now and will go over this x-pack alerting and let know.
Thanks again and kindly do let us know once profiling is available for us to use. Regards, Ananthanarayanan.M.S On Mon, Aug 7, 2017 at 2:33 PM, William Guo <[email protected]> wrote: > > hi Ananthanarayanan, > > > There are several ways to set up elasticsearch alerting. > > 1. alerts with x-pack > > x-pack is provided by Elastic Corporation. More information is availabe at > https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html. > x-pack requires license after evaluation period. The overall settings up > are like the following. > > > - install x-pack plugin on all the nodes in the cluster > - set up channels like email server, slack server in the > elasticsearch.yml file > - set up alerts with desired query. > - when the desired query meet its condition, alerts will be issued via > channels. > > > > > 2. free alternative > > ElastAlert is a simple framework for alerting on anomalies, spikes, or > other patterns of interest from data in Elasticsearch. It's implemented in > python. More information available at https://elastalert. > readthedocs.io/en/latest/ > > The overall settings up are like the following. > > - install python packages elastalert and elasticsearch > - set up config.yaml for the connections and authentications > - set up the elasticsearch index for ElastAlert > - set up alerts > - run ElastAlert as a demon service or a python process > - when alerts meet their conditions, alerts will be issued via > channels. > > > Thanks, > > William > > > > ------------------------------ > *From:* William GUO <[email protected]> on behalf of William Guo < > [email protected]> > *Sent:* Friday, August 4, 2017 1:37:47 PM > *To:* Ananthanarayanan Ms; Mara Preotescu > *Cc:* [email protected]; Kunduru, Abishek > *Subject:* Re: Meeting minutes with Nielsen: > > hi Ananthanarayanan, > > > For profiling, we have developed some samples based on our measures. Will > make samples available in our repo next week. > > > Accuracy, we have test cases for it, but we will also make samples in our > repo next week. > > > Thanks, > > William > > ________________________________ > From: Ananthanarayanan Ms <[email protected]> > Sent: Friday, August 4, 2017 12:30:22 AM > To: Mara Preotescu > Cc: William Guo; [email protected]; Kunduru, Abishek > Subject: Re: Meeting minutes with Nielsen: > > Hi William/Lionel, > Could you please help us to understand on the profiling feature > availability, we see the code traces on profiling, could you let us know if > we can starting using it from some branch which could be used so that we > could leverage griffin on two dimensions (accuracy & profiling). If its not > available a tentative date so that we could decide upon the same. > > > Regards, > Ananthanarayanan.M.S > > On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu < > [email protected]<mailto:[email protected]>> wrote: > Thank you William. > > Ananth will follow up with a few more questions on the roadmap. > > Thanks, > Mara > > On Wed, Aug 2, 2017 at 9:20 AM, William Guo <[email protected]<mailto:guoyp > @apache.org>> wrote: > > > Meeting minutes with Nielsen: > > > * Discuss griffin to support filters for metastore tables or > navigation assistance for table selection on UI. > * Griffin provides RESTful API for backend. > * Discuss griffin to support multiple source or target tables. > * Discuss more supporting file types, such as parquet. > * In griffin, the partition field is optional, it just helps to > provide the specific part of data, it will get all the data of a table > without any partition information. > * Config json file provides the parameters for griffin measure > calculation, you can also submit a spark job with it directly. > * Currently, griffin can only reuse measure, not rule. We’ll discuss > about this, if we need to support reusing rules. > * Sample ratio field in config file is optional, in batch mode we > don’t need to configure it. > * In griffin, mapping of columns are limited, discuss to support > advanced features like joining between tables , or advanced sql script. > * At current, the rule parser doesn’t support customized rules, > griffin has the plan to support this. //TODO document it and send it to dev > list > * Griffin doesn’t support metrics alert function, it posts all the > metrics to elasticsearch, es supports such feature. //TODO, write a > solution for it based on elastic search > * In griffin, you can’t modify the exist rules or measure at current. > > > > Thanks, > William > > > ________________________________ > From: William GUO <[email protected]<mailto:[email protected]>> on > behalf of William Guo <[email protected]<mailto:[email protected]>> > Sent: Wednesday, August 2, 2017 10:02:15 AM > To: Mara Preotescu > Cc: [email protected]<mailto:dev@griffin. > incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek > Subject: Re: Griffin support & roadmap > > hi mara, > > > Are you join? > > > Thanks, > > William > > ________________________________ > From: Mara Preotescu <[email protected]<mailto: > [email protected]>> > Sent: Monday, July 31, 2017 11:22:00 PM > To: William Guo > Cc: [email protected]<mailto:dev@griffin. > incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek > Subject: Re: Griffin support & roadmap > > Hi William, > > Would 10:00 am CST (Beijing) work for you on Wednesday 08/02? > > Thanks, > Mara > > On Sun, Jul 30, 2017 at 10:59 PM, William Guo <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > > hi Mara, > > > We are in China, it is hard to arrange a meeting for US, CHINA, INDIA > together. > > > China day time is fine for me. > > > Thanks, > > William > > ________________________________ > From: Mara Preotescu <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto:mara. > [email protected]>>> > Sent: Monday, July 31, 2017 10:54:25 AM > > To: William Guo > Cc: Lv, Alex; Guo, William; [email protected]<mailto: > [email protected]><mailto:[email protected] > <mailto:[email protected]>> > Subject: Re: Griffin support & roadmap > > Hi William, > > Either Wednesday or Thursday will work for us. Any better time working > for you? What time zone are you in? I am in US ET time, a colleague of > mine who I would like to join our discussion is in India, Chennai. > > Thanks, > Mara > > On Sun, Jul 30, 2017 at 7:34 PM, William Guo <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > > hi Mara, > > > Sure, We could schedule a meeting to discuss background, requirements, > status and milestone. > > > We should be fine in Wednesday or Thursday, what is your proposal? > > > > Thanks, > > William > > ________________________________ > From: Mara Preotescu <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto:mara. > [email protected]>>> > Sent: Friday, July 28, 2017 7:57:45 PM > To: William Guo > Cc: Lv, Alex; Guo, William; [email protected]<mailto: > [email protected]><mailto:[email protected] > <mailto:[email protected]>> > Subject: Re: Griffin support & roadmap > > HI Alex, William, > > THANK YOU so much for your responses. Thank you for the links. And, I > hope you don't mind if I'll take up your offer to contact you if needed. > We are considering, here at Nielsen, using Griffin for our new Data Quality > framework ... we know the project is still in the incubator but we would > like give it a try and even contributing, if needed. We already install > it and ran a few tests. > > If your time permits I would like scheduling a quick call so we could > understand the current status and, most importantly if the roadmap stays as > in the published documents. > > Thanks again, > Mara > > > On Fri, Jul 28, 2017 at 4:21 AM, William Guo <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > > hi Mara, > > Few links might help, you can contact us by [email protected]. > org<mailto:[email protected]><mailto:d > [email protected]<mailto:[email protected]>> > or my personal account [email protected]<mailto:guoyp@ > apache.org><mailto:[email protected] > <[email protected]%3E%3Cmailto:[email protected]>> > > > GitHub : https://github.com/apache/incubator-griffin<https:// > github.com/eBay/griffin<https://github.com/apache/incubator- > griffin%3Chttps://github.com/eBay/griffin > <https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin%3Chttps://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin> > >> > Website : https://griffin.incubator.apache.org<https://griffin. > incubator.apache.org/> > Contact: mailto://[email protected]< > mailto:[email protected]> > Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN > Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/ > Griffin > > Thanks, William > > ________________________________ > From: Lv, Alex <[email protected]<mailto:[email protected]><mailto: > [email protected]<mailto:[email protected]>>> > Sent: Friday, July 28, 2017 9:10:09 AM > To: Mara Preotescu; Guo, William; [email protected]<mailto:guoyp@ > apache.org><mailto:[email protected] > <[email protected]%3E%3Cmailto:[email protected]>> > Cc: [email protected]<mailto:dev@griffin. > incubator.apache.org><mailto:[email protected]<mailto: > [email protected]>> > Subject: RE: Griffin support & roadmap > > <<Move Amber to BCC>> > Hi Mara, > > Glad to hear from you, you may discuss the details with William. > Thx. > > Best regards, > Alex Lv > > From: Mara Preotescu [mailto:[email protected]<mailto:mara. > [email protected]<mailto:[email protected]% > 3Cmailto:[email protected] > <[email protected]%3Cmailto:[email protected]%3Cmailto:[email protected]%3Cmailto:[email protected]> > >>] > Sent: 2017年7月28日 6:14 > To: Lv, Alex <[email protected]<mailto:[email protected]><mailto:lzhixing > @ebay.com<mailto:[email protected]>>>; Vaidya, Amber <[email protected] > <mailto:[email protected]><mailto:[email protected]<mailto:amvaidya@ > ebay.com>>> > Subject: Griffin support & roadmap > > Hello Alex, Amber, > > I am writing you trying to reach the support for Griffin, both support > e-mails for the product returned as invalid addresses ( > [email protected]<mailto:su > [email protected]><mailto:subscribe- > [email protected]<mailto:subscribe-dev@ > griffin.incubator.apache.org>>, [email protected] > <mailto:[email protected]><mailto:ebay-griffin-devs@ > googlegroups.com<mailto:[email protected]>>). > > Could you please let me know who should we contact to discuss about > Griffin's roadmap? > > We are looking, here at Nielsen, to use the Griffin framework for our DQ > processes. As of today we learned, and tested, the only dimension > available, Accuracy. Would you be able to share the roadmap for any other > DQ dimensions availability? > > We are looking as well to add a few custom validations - does the tool > offer any APIs that can be used for this purpose? > > Any information you could provide would be very, very helpful. > > > Thank you in advance for your help and time. > Mara Preotescu > VP Technology, DevOps Nielsen > > > > > >
